Thursday, June 4, 2015

Docker Gotchas

I've watched as the memes have gravitated to Docker like bees to nectar or flies to poop. I guess that's a half full half empty thing. As I've said many times in the past you gotta know your stack. And the more I use Docker and the more they advance the project the more I realize that it's just too early for adoption... unless you have a person or three on the inside.

By example my first CoreOS+docker server is continuously running out of disk space in part because btmp is filling up (which I've read is probably an attack). My second CoreOS+Docker server shows an explosion of files in the Docker overlay folder. I'm not certain why I need so many folders and if any of them are zombies or not.

Another article made mention of my filesystem iNodes. While my filesystem is at %5 used and iNodes at 15%; which is not an unhealthy ratio, in my opinion, but since this is a single Docker container running on a 200GB HDD it suggests that getting any sort of density is going to be unpredictable and painful.

I may need to investigate NixOS containers.

UPDATE:  found this article echoing my problem.

UPDATE: I made some discoveries that are not actually documented anywhere.  The first truth that I realized is that proper VMs are probably still a better proposition than containers of any kind. CoreOS toolbox and it's analogs make sense but not for actually running a production service. This is yet another reason why the Phusion containers are a bad choice.

Next, you simply do not need intermediate layers. The first reason is that many times the builder falsely uses an old container. For example I have a RUN go get some_go_package and docker does not rebuild it. SO I have to but my go get commands after my ADD commands in order to insure they are executed.

Now that I'm deleting all of my images, intermediate layers, I'm getting ready to rebuild my containers with the --rm and --force-rm options. I do not want any intermediate files. The issue is that docker build is consuming too much disk space and any hope of reasonable density is impossible.

delete all of your contianers
docker rm $(docker ps -a -q)
delete all of your images
docker rmi $(docker images -q)
delete eveything
docker rmi $(docker images -q)
** this last one takes a VERY long time. My 331 overlays has been deleting for nearly 30 minutes and white it was only 2.5GB it made up 15% of the inodes meaning there were a lot of files.

UPDATE:  One more thing to keep in mind as my system is still deleting those intermediate layers... GCE (google compute engine) limits the iops for my host VM and so bulk deletes like this are going to take a while regardless whether it's an SSD or HDD.

UPDATE: In hindsight this sort of cleanup (a) would have been faster to rebuild the host (b) if there had been more than one container on this system I would never been able to clean it up because there are no tags. The also means that the build and deploy system should probably be different machines.

UPDATE: compressing a container [link] (devbox and devboxrun are my image and container names)
ID=$(docker run -d devbox /bin/bash)
docker export $ID | docker import - devboxrun
UPDATE: backup compressed snapshots
ID=$(docker run -d image-name /bin/bash)
(docker export $ID | gzip -c > image.tgz)
gzip -dc image.tgz | docker import - flat-image-name 
UPDATE: I'm testing my last clean build configuration (bash functions)
function cleanbox {
        docker rm $(docker ps -q -a --filter=image=builddevbox:latest)
        docker rmi builddevbox
        docker rmi golang:cross
}

function buildbox {
        docker build --force-rm --rm -t builddevbox .
        ID=$(docker run -d builddevbox /bin/bash)
        docker export $ID | docker import - devbox
        cleanbox
}

No comments:

Post a Comment

another bad day for open source

One of the hallmarks of a good open source project is just how complicated it is to install, configure and maintain. Happily gitlab and the ...