Docker daemon fails to start on voxl

kiprock

@Chad-Sweet Thanks for the fast response!

yocto:~# voxl-configure-docker-support.sh
Stopping original docker service
Enabling our own services docker-start & docker-prepare
starting docker-start.service
loading hello-world docker image
Cannot connect to the Docker daemon. Is the docker daemon running on this host?
yocto:~#  systemctl status docker
● docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; disabled; vendor preset: enabled)
   Active: inactive (dead)
     Docs: http://docs.docker.com
May 13 15:46:33 apq8096 systemd[1]: Stopped Docker Application Container Engine.

yocto:~#

yocto:~#  systemctl status docker
● docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; disabled; vendor preset: enabled)
   Active: inactive (dead)
     Docs: http://docs.docker.com

May 13 15:46:33 apq8096 systemd[1]: Stopped Docker Application Container Engine.
yocto:~#  systemctl start docker
yocto:~#  systemctl status docker
● docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; disabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Thu 2021-05-13 15:48:03 UTC; 2s ago
     Docs: http://docs.docker.com
  Process: 3678 ExecStart=/usr/bin/docker daemon -H fd:// --raw-logs (code=exited, status=2)
 Main PID: 3678 (code=exited, status=2)

May 13 15:48:03 apq8096 systemd[1]: Started Docker Application Container Engine.
May 13 15:48:03 apq8096 docker[3678]: flag provided but not defined: --raw-logs
May 13 15:48:03 apq8096 docker[3678]: See '/usr/bin/docker daemon --help'.
May 13 15:48:03 apq8096 systemd[1]: docker.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
May 13 15:48:03 apq8096 systemd[1]: docker.service: Unit entered failed state.
May 13 15:48:03 apq8096 systemd[1]: docker.service: Failed with result 'exit-code'.
yocto:~# docker images
An error occurred trying to connect: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.21/images/json: read unix @->/var/run/docker.sock: read: connection reset by peer

Chad Sweet

Not really sure what state your system is in. This github issue shows it could be related to disk space or needs a reboot. Did you install other Dockers on the image? They are very large and maybe all of the space is consumed
https://github.com/docker/for-linux/issues/435

kiprock

Just looking at your script, I realized that modal has it's own docker services.. so perhaps this provides a little more info: I have created a few images from the kinetic image

yocto:~# systemctl disable docker.service
yocto:~# systemctl enable docker-start.service
yocto:~# systemctl enable docker-prepare.service
yocto:~# systemctl status docker-prepare.service
● docker-prepare.service
   Loaded: loaded (/etc/systemd/system/docker-prepare.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Thu 1970-01-01 00:00:24 UTC; 51 years 4 months ago
 Main PID: 2352 (code=exited, status=1/FAILURE)

Jan 01 00:00:20 apq8096 docker-prepare.sh[2352]: /usr/bin/docker-prepare.sh: line 17: /sys/fs/cgroup/cpuset/syst...ctory
Jan 01 00:00:21 apq8096 docker-prepare.sh[2352]: /usr/bin/docker-prepare.sh: line 17: /sys/fs/cgroup/cpuset/syst...ctory
Jan 01 00:00:22 apq8096 docker-prepare.sh[2352]: /usr/bin/docker-prepare.sh: line 17: /sys/fs/cgroup/cpuset/syst...ctory
Jan 01 00:00:23 apq8096 docker-prepare.sh[2352]: /usr/bin/docker-prepare.sh: line 17: /sys/fs/cgroup/cpuset/syst...ctory
Jan 01 00:00:24 apq8096 docker-prepare.sh[2352]: /usr/bin/docker-prepare.sh: line 17: /sys/fs/cgroup/cpuset/syst...ctory
Jan 01 00:00:24 apq8096 docker-prepare.sh[2352]: failed to see cpuset appear after 15 seconds
Jan 01 00:00:24 apq8096 systemd[1]: docker-prepare.service: Main process exited, code=exited, status=1/FAILURE
Jan 01 00:00:24 apq8096 systemd[1]: Failed to start docker-prepare.service.
Jan 01 00:00:24 apq8096 systemd[1]: docker-prepare.service: Unit entered failed state.
Jan 01 00:00:24 apq8096 systemd[1]: docker-prepare.service: Failed with result 'exit-code'.
Hint: Some lines were ellipsized, use -l to show in full.
yocto:~#
yocto:~#
yocto:~# systemctl status docker-start.service
● docker-start.service
   Loaded: loaded (/etc/systemd/system/docker-start.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Thu 2021-05-13 15:57:11 UTC; 3min 11s ago
 Main PID: 3873 (code=exited, status=1/FAILURE)

May 13 15:57:11 apq8096 systemd[1]: Started docker-start.service.
May 13 15:57:11 apq8096 docker[3873]: time="2021-05-13T15:57:11.453820000Z" level=info msg="API listen on /var/....sock"
May 13 15:57:11 apq8096 docker[3873]: time="2021-05-13T15:57:11.477567000Z" level=info msg="[graphdriver] using...lay\""
May 13 15:57:11 apq8096 docker[3873]: time="2021-05-13T15:57:11.527540000Z" level=info msg="Firewalld running: false"
May 13 15:57:11 apq8096 docker[3873]: time="2021-05-13T15:57:11.595538000Z" level=fatal msg="Error starting dae...oints"
May 13 15:57:11 apq8096 systemd[1]: docker-start.service: Main process exited, code=exited, status=1/FAILURE
May 13 15:57:11 apq8096 systemd[1]: docker-start.service: Unit entered failed state.
May 13 15:57:11 apq8096 systemd[1]: docker-start.service: Failed with result 'exit-code'.
Hint: Some lines were ellipsized, use -l to show in full.
yocto:~#
yocto:~# df
Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/root        2837860 1890496    930980  68% /
devtmpfs         1861956       0   1861956   0% /dev
tmpfs            1930660       0   1930660   0% /dev/shm
tmpfs            1930660   17136   1913524   1% /run
tmpfs            1930660       0   1930660   0% /sys/fs/cgroup
tmpfs            1930660       4   1930656   1% /var/volatile
/dev/sda2           2168      80      1964   4% /persist
/dev/sde12         12016    4184      7508  36% /dsp
/dev/sda3          58888      40     57568   1% /cache
/dev/sde11         97232   33616     63616  35% /firmware
/dev/sda9       15248568 7060916   8171268  47% /data
tmpfs             386132       0    386132   0% /run/user/0

kiprock

I had a battery go pretty low and the system crashed while running mavros in docker last night, after reboot, docker deamon is not responding and I cannot see any images.

Eric Katzfey

@kiprock I would reflash the system, setup your IP connection, configure Docker and just verify that it works as expected.

kiprock

just looking at the logs, the docker deamon fails here:

level=fatal msg="Error starting daemon: Error initializing network controller: could not delete the default bridge network: network bridge has active endpoints"

Most forums suggest removing the docker network folder: rm -r /var/lib/docker/network

I did this but still having issues. Just curious if the modalAI setup stores docker files somewhere else?

Eric Katzfey

@kiprock Everything is stored on /data. But again, I would reflash and start over just to verify Docker operation.

kiprock

rm /data/network/files/local-kv.db

did the trick. Thanks for providing that missing piece of data!

shlee853

@kiprock Hi I have a exactly same issue.

I flashed image and reset factory configuration on my VOXL, but, It has still issue.

Did you resolve this problem?

Thanks,

kiprock

@Eric-Katzfey if you cannot start docker images, you can try the command above to clear them. Good luck.

Eric Katzfey

Thanks, but I think you meant to ping @shlee853

shlee853

@kiprock thank you for your reply
I’ll try it

shlee853

@shlee853

Great!

I resolved this problem as you mentioned.

yocto:/$ rm /data/network/files/local-kv.db
yocto:/$ voxl-configure-docker-support.sh 
Stopping original docker service
Enabling our own services docker-start & docker-prepare
starting docker-start.service
loading hello-world docker image
successfully loaded hello-world
starting docker-prepare service
Created symlink from /etc/systemd/system/default.target.wants/docker-autorun.service to /etc/systemd/system/docker-autorun.service.

done configuring voxl-docker-support

LucasW

@kiprock Thank you for this, that also solved the issue in my case.

To be a little more cautious, I simply renamed the existing file, which forced the generation of a new one when running the configuration script voxl-configure-docker-support.sh:

voxl:/data/network/files$ mv local-kv.db local-kv_DISABLED.db

José Quintanilla

@modaltb I have a problem, after I write that line, the drone display a message that says: voxl:/$ voxl-configure-docker-suport.sh
bash: voxl-configure-docker-suport.sh: command not found
voxl:/$

tom

@José-Quintanilla Looks like you spelled it wrong

tom

@José-Quintanilla You can use [TAB] after you start writing a command to auto-fill

José Quintanilla

@tom I think it worked, thanks

José Quintanilla

@tom It wasn't that, because I need that especific line on my VOXL to work, so I can use properly the sdk

tom

@José-Quintanilla What is not working? Did you run voxl-configure-docker-support.sh?