docker fills up /data, prevents code from running
-
There is one other thing I did that is noteworthy. I reflashed the base image with 3.3 and installed the voxl-suite. In this process, I selected the option that left /data intact.
-
Can you show the output of
# df -i /data
? -
Here you are:
yocto:/$ df -i /data Filesystem Inodes IUsed IFree IUse% Mounted on /dev/sda9 977280 977280 0 100% /data
-
Here is some more info. Looks like docker is using all the inodes:
yocto:/data$ sudo find . -xdev -type f | cut -d "/" -f 2 | sort | uniq -c | sort -n 1 adb_devid 1 db 1 dhcpcd-wlan0.info 1 dnsmasq.conf 1 dnsmasq_d.leases 1 l2tp_cfg.xml 1 linkgraph.db 1 mobileap_cfg.xml 1 modalai 1 network 1 repositories-overlay 2 usb 2 web_root 4 persist 7 iproute2 20 misc 53 containers 189 graph 3265231 overlay
-
System image 3.3.0 increases the number of inodes on /data to 3M. However, that may require that /data is completely wiped when you install it.
-
I believe I already had system image 3.3 installed on my VOXL. (The number of inodes dedicated to
overlay
is around the 3M mark.) Earlier, when I said I reflashed the base image with 3.3, it is was an effort to rule out the system image being corrupt.Is there some way to clean up all the inodes being used by docker via the command line? (The docker-daemon still dies on bootup/restart, so I can't use docker commands.)
-
I found a partial solution. I deleted some unused, empty directories in
/data
(e.g.,audio
). This allowed me to create four free inodes.I was able to run
systemctl restart docker-daemon
and it worked for just a few seconds before crashing. In that time, I was able to rundocker ps -a
and see the list of containers. I tried todocker start <my_container>
but the docker-daemon had already crashed.I went into
/data/containers
and deleted thehello world
container.I re-ran
systemctl restart docker-daemon
and was able to rundocker rm <container_name>
for some of the unused containers. Doing this a few times freed up ~200 inodes. The docker-daemon was able to run without crashing.At that point, I was able to start
<my_container>
, and enter it. I was able to push my code to my git repo. This is what I set out to do.Now that I have my data saved, I could nuke the /
data/overlay
directory and reclaim most of my inodes. I'd prefer if there was a better way to keep the/data/overlay
directory clean (perhaps as a part of regular maintenance). This would be preferable to nuking the entire direcotry from time to time as it gets filled up. If there is a good way modalAI knows how to do this, please share!(And thanks for the pointer about the inodes. I didn't consider that that was what was happening.)
-
@Eric-Katzfey, let me ask a follow-up question.
Currently, the docker version that is on the VOXL is v1.9. This version doesn't support commands like
docker system prune
ordocker prune images
.Does ModalAI plan to update the docker version used on VOXL?
Is it possible to upgrade it myself? Or are there reasons why ModalAI is still using v1.9? (A google search told me the latest version of docker is 20.10.)
-
We use Yocto to build the system image. It is a fairly old version (Jethro) and that really limits what we can do as far as upgrades. When you try to upgrade a single component you usually have to upgrade it's dependencies, and then the dependencies of those dependencies, etc. So upgrading the Docker version is likely a very large task and not something we are planning to do any time soon.
-
Good to know -- thanks, Eric!