docker fills up /data, prevents code from running

jaredjohansen

There is one other thing I did that is noteworthy. I reflashed the base image with 3.3 and installed the voxl-suite. In this process, I selected the option that left /data intact.

Eric Katzfey

Can you show the output of # df -i /data?

jaredjohansen

Here you are:

yocto:/$ df -i /data
Filesystem     Inodes  IUsed IFree IUse% Mounted on
/dev/sda9      977280 977280     0  100% /data

jaredjohansen

Here is some more info. Looks like docker is using all the inodes:

yocto:/data$ sudo find . -xdev -type f | cut -d "/" -f 2 | sort | uniq -c | sort -n              
      1 adb_devid
      1 db
      1 dhcpcd-wlan0.info
      1 dnsmasq.conf
      1 dnsmasq_d.leases
      1 l2tp_cfg.xml
      1 linkgraph.db
      1 mobileap_cfg.xml
      1 modalai
      1 network
      1 repositories-overlay
      2 usb
      2 web_root
      4 persist
      7 iproute2
     20 misc
     53 containers
    189 graph
3265231 overlay

Eric Katzfey

System image 3.3.0 increases the number of inodes on /data to 3M. However, that may require that /data is completely wiped when you install it.

jaredjohansen

I believe I already had system image 3.3 installed on my VOXL. (The number of inodes dedicated to overlay is around the 3M mark.) Earlier, when I said I reflashed the base image with 3.3, it is was an effort to rule out the system image being corrupt.

Is there some way to clean up all the inodes being used by docker via the command line? (The docker-daemon still dies on bootup/restart, so I can't use docker commands.)

jaredjohansen

I found a partial solution. I deleted some unused, empty directories in /data (e.g., audio). This allowed me to create four free inodes.

I was able to run systemctl restart docker-daemon and it worked for just a few seconds before crashing. In that time, I was able to run docker ps -a and see the list of containers. I tried to docker start <my_container> but the docker-daemon had already crashed.

I went into /data/containers and deleted the hello world container.

I re-ran systemctl restart docker-daemon and was able to run docker rm <container_name> for some of the unused containers. Doing this a few times freed up ~200 inodes. The docker-daemon was able to run without crashing.

At that point, I was able to start <my_container>, and enter it. I was able to push my code to my git repo. This is what I set out to do.

Now that I have my data saved, I could nuke the /data/overlay directory and reclaim most of my inodes. I'd prefer if there was a better way to keep the /data/overlay directory clean (perhaps as a part of regular maintenance). This would be preferable to nuking the entire direcotry from time to time as it gets filled up. If there is a good way modalAI knows how to do this, please share!

(And thanks for the pointer about the inodes. I didn't consider that that was what was happening.)

jaredjohansen

@Eric-Katzfey, let me ask a follow-up question.

Currently, the docker version that is on the VOXL is v1.9. This version doesn't support commands like docker system prune or docker prune images.

Does ModalAI plan to update the docker version used on VOXL?

Is it possible to upgrade it myself? Or are there reasons why ModalAI is still using v1.9? (A google search told me the latest version of docker is 20.10.)

Eric Katzfey

We use Yocto to build the system image. It is a fairly old version (Jethro) and that really limits what we can do as far as upgrades. When you try to upgrade a single component you usually have to upgrade it's dependencies, and then the dependencies of those dependencies, etc. So upgrading the Docker version is likely a very large task and not something we are planning to do any time soon.

jaredjohansen

Good to know -- thanks, Eric!