Docker Qcow2 Large

Dec 05, 2020 The following script delete and re-create a new shrinked Docker.qcow2 file preserving the images passed as arguments. When running a large database.

• Docker Qcow2 Large Image
• Docker Qcow2 Large Cap

Yesterday morning I was innocently minding my own business, downloading some files, when I noticed that nearly all the disk space on my 256GB work laptop had been consumed. This seemed rather unlikely to me, given that I'd only had the laptop for about a year and I didn't store anything other than code and work documents on it. Text files just don't take up that much space most of the time. So I decided to make a purchase I'd considered for a while, bought Daisy Disk, and began investigating.

• Jan 29, 2018 It has something to do with the change of Docker's disk format from QCOW2 to RAW. I had to revert to Docker 17.09.1-ce-mac42 (21090) to get rid of the issue. Seems to be connected with issue #2542.
• Docker Desktop is an application for MacOS and Windows machines for the building and sharing of containerized applications and microservices. Docker Desktop delivers the speed, choice and security you need for designing and delivering containerized applications on your desktop.
• OpenStack software includes native and upstream support for virtualization over HDR 200 gigabit InfiniBand network, enabling customers to build high-performance OpenStack-based HPC and AI cloud services taking advantage of cloud multi-tenancy and security capabilities while running their RDMA/IPoIB workloads over the most enhanced interconnect infrastructure, leveraging.
• Type: qemu The Qemu Packer builder is able to create KVM virtual machine images. The builder builds a virtual machine by creating a new virtual machine from scratch, booting it, installing an OS, rebooting the machine with the boot media as the virtual hard drive, provisioning software within the OS, then shutting it down.

The first thing I noticed was a huge amount of the disk space (about half) was taken up by the ~/Library/Containers folder. That folder contained my email history and also data on my Docker containers. Docker functions as a lightweight VM, and essentially holds copies of virtualized operating systems and file systems inside each docker container and image, so it made sense to me that it could be taking up a lot of space, though >120GB still seemed wrong for my paltry 4 containers. So my first step was to delete all of the containers and images on my laptop. That cleared about 20GB of space but still left my drive looking like this:

At this point, I was annoyed. 103.5 GB was taken up by some folder called Docker.qcow2 and Daisy Disk wouldn't show me what was inside. So I drilled in on the file system and immediately found out that Docker.qcow2 was not a directory like I'd assumed due to size, but a single 100+GB file. At that point I decided to poll my teammates to see if I was the only one dealing with this:

Wii u customer service. So between the 5 of us, we each had 'cow files' taking up between 23 and 103GB of disk space. Some Googling revealed a github thread that showed this is a so far unsolved issue with Docker For Mac. Summary: qcow2 files are a format for saving disk images. Docker For Mac's implementation works well for writing and updating images, but doesn't automatically free up disk space when a container or image is deleted. So as you use and delete containers over time this file gradually grows. I have a habit of deleting and recreating containers when I'm trying to debug something, which explains why I had a much larger file than others. There isn't currently a true fix for this issue, but you can delete the file. You'll lose all your containers and images, but when you recreate them the file will be small again. There also is supposed to be some automatic limiting of the file. The current cap is 64GB, but there is work on making it configurable. Unfortunately the current cap is not respected if you had previously used docker-machine to control docker on your machine, which is how my file exceeded the cap. I'm also unclear on what happens when the file hits the cap. I get the impression that things stop working and you need to delete everything anyway (you just avoid running out of disk space on the host machine).

Fortunately it was no problem for me to delete everything, and so I was able to clear things out, at which point my disk looked a lot happier:

Lessons Learned

I had 3 takeaways from this interesting adventure.

Docker Qcow2 Large Image

1. If you're using Docker For Mac, keep an eye on your disk space. If you're able to occasionally delete and recreate your containers without data loss, consider occasionally doing that and deleting the cow file. If you can't do that, be careful how many containers you add and delete, and make sure you manage your disk space well.
2. Daisy Disk is awesome and highly recommended. It's an example of a rare breed: the beautiful system utility. The visualizations it shows are both pretty and useful; it made diagnosing this issue a breeze.
3. One more thing I learned from my coworkers slack yesterday: ls takes an -h argument that shows file sizes in KBs/MBs/GBs instead of all in bytes. This is super helpful when examining large files. Compare the 2 lists of files from my Downloads folder in the image below. The normal form is very nice for comparing 2 files side by side and seeing which one is bigger, but the second form is much more helpful when you want to get an idea of exactly how big something is, or communicate it to others. Most of us don't think about file sizes in terms of bytes anymore.

Docker Qcow2 Large Cap