Docker daemon fails to start on voxl

Alex123 · 9 Jan 2021, 13:38

RE: https://forum.modalai.com/topic/37/docker-daemon-fails-to-start-voxl
I'm also facing the same issue.

yocto:/# voxl-docker -i roskinetic-xenial:opencv
Cannot connect to the Docker daemon. Is the docker daemon running on this host?
yocto:/#
yocto:/#
yocto:/# docker images
Cannot connect to the Docker daemon. Is the docker daemon running on this host?
yocto:/#
yocto:/#
yocto:/# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/root 2.8G 2.2G 581M 79% /
devtmpfs 1.8G 0 1.8G 0% /dev
tmpfs 1.9G 0 1.9G 0% /dev/shm
tmpfs 1.9G 17M 1.9G 1% /run
tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup
tmpfs 1.9G 4.0K 1.9G 1% /var/volatile
/dev/mmcblk0p1 7.4G 204M 7.2G 3% /mnt/sdcard
/dev/sda2 2.2M 80K 2.0M 4% /persist
/dev/sde12 12M 4.1M 7.4M 36% /dsp
/dev/sde11 95M 33M 63M 35% /firmware
/dev/sda3 58M 40K 57M 1% /cache
/dev/sda9 15G 13G 2.4G 85% /data
yocto:/#

As you can see, /data partition has all the data and docker images.

What can be a solution for this?

modaltb · 9 Jan 2021, 14:34

What do you see after running voxl-configure-docker.sh ?

Should be something like:

yocto:/# voxl-configure-docker-support.sh
Stopping original docker service
Enabling our own services docker-start & docker-prepare
starting docker-start.service
loading hello-world docker image
successfully loaded hello-world
starting docker-prepare service

done configuring voxl-docker-support

Alex123 · 11 Jan 2021, 05:30

yocto:/# voxl-configure-docker-support.sh
Stopping original docker service
Enabling our own services docker-start & docker-prepare
starting docker-start.service
loading hello-world docker image
Cannot connect to the Docker daemon. Is the docker daemon running on this host?
yocto:/#

Alex123 · 11 Jan 2021, 06:35

yocto:/# systemctl status docker
● docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; disabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Mon 2021-01-11 06:31:53 UTC; 1min 53s ago
Docs: http://docs.docker.com
Main PID: 4675 (code=exited, status=2)

Jan 11 06:31:53 apq8096 systemd[1]: Started Docker Application Container Engine.
Jan 11 06:31:53 apq8096 docker[4675]: flag provided but not defined: --raw-logs
Jan 11 06:31:53 apq8096 docker[4675]: See '/usr/bin/docker daemon --help'.
Jan 11 06:31:53 apq8096 systemd[1]: docker.service: Main process exited, co...NT
Jan 11 06:31:53 apq8096 systemd[1]: docker.service: Unit entered failed state.
Jan 11 06:31:53 apq8096 systemd[1]: docker.service: Failed with result 'exi...'.
Jan 11 06:31:57 apq8096 systemd[1]: Stopped Docker Application Container Engine.
Jan 11 06:33:06 apq8096 systemd[1]: Stopped Docker Application Container Engine.
Hint: Some lines were ellipsized, use -l to show in full.

Alex123 · 11 Jan 2021, 07:30

This post is deleted!

Alex123 · 11 Jan 2021, 10:57

As I'm working on this issue, I noticed the following.

yocto:/# ifconfig
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:32 errors:0 dropped:0 overruns:0 frame:0
TX packets:32 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:2224 (2.1 KiB) TX bytes:2224 (2.1 KiB)

wlan0 Link encap:Ethernet HWaddr :::::
inet addr:192.168..** Bcast:192.168..*** Mask:...
inet6 addr: :::::**/** Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:467 errors:0 dropped:0 overruns:0 frame:0
TX packets:88 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:3000
RX bytes:40293 (39.3 KiB) TX bytes:10317 (10.0 KiB)

https://stackoverflow.com/questions/64028448/error-during-connect-get-http-2fvar2frun2fdocker-sock-v1-40-containers-jso

It's not showing any IP address for docker0.
How to configure the IP address that will listen to docker daemon on voxl-core board?

kiprock · 13 May 2021, 15:42

Hey @Alex123 were you able to resolve this? I'm having the same issue.

Chad Sweet · 13 May 2021, 15:49

Are you following the docs here?

The voxl-docker command is for developing in a Docker on a PC, it should not be run on VOXL.

The tutorial steps to run on target are:

yocto:/# voxl-configure-docker-support.sh
yocto:/# docker images
yocto:/# docker run hello-world

If you have run the PC scripts on VOXL, not really sure what would happen

kiprock · 13 May 2021, 15:54

@Chad-Sweet Thanks for the fast response!

yocto:~# voxl-configure-docker-support.sh
Stopping original docker service
Enabling our own services docker-start & docker-prepare
starting docker-start.service
loading hello-world docker image
Cannot connect to the Docker daemon. Is the docker daemon running on this host?
yocto:~#  systemctl status docker
● docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; disabled; vendor preset: enabled)
   Active: inactive (dead)
     Docs: http://docs.docker.com
May 13 15:46:33 apq8096 systemd[1]: Stopped Docker Application Container Engine.

yocto:~#

yocto:~#  systemctl status docker
● docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; disabled; vendor preset: enabled)
   Active: inactive (dead)
     Docs: http://docs.docker.com

May 13 15:46:33 apq8096 systemd[1]: Stopped Docker Application Container Engine.
yocto:~#  systemctl start docker
yocto:~#  systemctl status docker
● docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; disabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Thu 2021-05-13 15:48:03 UTC; 2s ago
     Docs: http://docs.docker.com
  Process: 3678 ExecStart=/usr/bin/docker daemon -H fd:// --raw-logs (code=exited, status=2)
 Main PID: 3678 (code=exited, status=2)

May 13 15:48:03 apq8096 systemd[1]: Started Docker Application Container Engine.
May 13 15:48:03 apq8096 docker[3678]: flag provided but not defined: --raw-logs
May 13 15:48:03 apq8096 docker[3678]: See '/usr/bin/docker daemon --help'.
May 13 15:48:03 apq8096 systemd[1]: docker.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
May 13 15:48:03 apq8096 systemd[1]: docker.service: Unit entered failed state.
May 13 15:48:03 apq8096 systemd[1]: docker.service: Failed with result 'exit-code'.
yocto:~# docker images
An error occurred trying to connect: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.21/images/json: read unix @->/var/run/docker.sock: read: connection reset by peer

Chad Sweet · 13 May 2021, 15:59

Not really sure what state your system is in. This github issue shows it could be related to disk space or needs a reboot. Did you install other Dockers on the image? They are very large and maybe all of the space is consumed
https://github.com/docker/for-linux/issues/435

kiprock · 13 May 2021, 16:04

Just looking at your script, I realized that modal has it's own docker services.. so perhaps this provides a little more info: I have created a few images from the kinetic image

yocto:~# systemctl disable docker.service
yocto:~# systemctl enable docker-start.service
yocto:~# systemctl enable docker-prepare.service
yocto:~# systemctl status docker-prepare.service
● docker-prepare.service
   Loaded: loaded (/etc/systemd/system/docker-prepare.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Thu 1970-01-01 00:00:24 UTC; 51 years 4 months ago
 Main PID: 2352 (code=exited, status=1/FAILURE)

Jan 01 00:00:20 apq8096 docker-prepare.sh[2352]: /usr/bin/docker-prepare.sh: line 17: /sys/fs/cgroup/cpuset/syst...ctory
Jan 01 00:00:21 apq8096 docker-prepare.sh[2352]: /usr/bin/docker-prepare.sh: line 17: /sys/fs/cgroup/cpuset/syst...ctory
Jan 01 00:00:22 apq8096 docker-prepare.sh[2352]: /usr/bin/docker-prepare.sh: line 17: /sys/fs/cgroup/cpuset/syst...ctory
Jan 01 00:00:23 apq8096 docker-prepare.sh[2352]: /usr/bin/docker-prepare.sh: line 17: /sys/fs/cgroup/cpuset/syst...ctory
Jan 01 00:00:24 apq8096 docker-prepare.sh[2352]: /usr/bin/docker-prepare.sh: line 17: /sys/fs/cgroup/cpuset/syst...ctory
Jan 01 00:00:24 apq8096 docker-prepare.sh[2352]: failed to see cpuset appear after 15 seconds
Jan 01 00:00:24 apq8096 systemd[1]: docker-prepare.service: Main process exited, code=exited, status=1/FAILURE
Jan 01 00:00:24 apq8096 systemd[1]: Failed to start docker-prepare.service.
Jan 01 00:00:24 apq8096 systemd[1]: docker-prepare.service: Unit entered failed state.
Jan 01 00:00:24 apq8096 systemd[1]: docker-prepare.service: Failed with result 'exit-code'.
Hint: Some lines were ellipsized, use -l to show in full.
yocto:~#
yocto:~#
yocto:~# systemctl status docker-start.service
● docker-start.service
   Loaded: loaded (/etc/systemd/system/docker-start.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Thu 2021-05-13 15:57:11 UTC; 3min 11s ago
 Main PID: 3873 (code=exited, status=1/FAILURE)

May 13 15:57:11 apq8096 systemd[1]: Started docker-start.service.
May 13 15:57:11 apq8096 docker[3873]: time="2021-05-13T15:57:11.453820000Z" level=info msg="API listen on /var/....sock"
May 13 15:57:11 apq8096 docker[3873]: time="2021-05-13T15:57:11.477567000Z" level=info msg="[graphdriver] using...lay\""
May 13 15:57:11 apq8096 docker[3873]: time="2021-05-13T15:57:11.527540000Z" level=info msg="Firewalld running: false"
May 13 15:57:11 apq8096 docker[3873]: time="2021-05-13T15:57:11.595538000Z" level=fatal msg="Error starting dae...oints"
May 13 15:57:11 apq8096 systemd[1]: docker-start.service: Main process exited, code=exited, status=1/FAILURE
May 13 15:57:11 apq8096 systemd[1]: docker-start.service: Unit entered failed state.
May 13 15:57:11 apq8096 systemd[1]: docker-start.service: Failed with result 'exit-code'.
Hint: Some lines were ellipsized, use -l to show in full.
yocto:~#
yocto:~# df
Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/root        2837860 1890496    930980  68% /
devtmpfs         1861956       0   1861956   0% /dev
tmpfs            1930660       0   1930660   0% /dev/shm
tmpfs            1930660   17136   1913524   1% /run
tmpfs            1930660       0   1930660   0% /sys/fs/cgroup
tmpfs            1930660       4   1930656   1% /var/volatile
/dev/sda2           2168      80      1964   4% /persist
/dev/sde12         12016    4184      7508  36% /dsp
/dev/sda3          58888      40     57568   1% /cache
/dev/sde11         97232   33616     63616  35% /firmware
/dev/sda9       15248568 7060916   8171268  47% /data
tmpfs             386132       0    386132   0% /run/user/0

kiprock · 13 May 2021, 16:06

I had a battery go pretty low and the system crashed while running mavros in docker last night, after reboot, docker deamon is not responding and I cannot see any images.

Eric Katzfey · 13 May 2021, 16:39

@kiprock I would reflash the system, setup your IP connection, configure Docker and just verify that it works as expected.

kiprock · 13 May 2021, 16:48

just looking at the logs, the docker deamon fails here:

level=fatal msg="Error starting daemon: Error initializing network controller: could not delete the default bridge network: network bridge has active endpoints"

Most forums suggest removing the docker network folder: rm -r /var/lib/docker/network

I did this but still having issues. Just curious if the modalAI setup stores docker files somewhere else?

Eric Katzfey · 13 May 2021, 16:51

@kiprock Everything is stored on /data. But again, I would reflash and start over just to verify Docker operation.

kiprock · 13 May 2021, 16:56

rm /data/network/files/local-kv.db

did the trick. Thanks for providing that missing piece of data!

shlee853 · 10 Jun 2021, 20:49

@kiprock Hi I have a exactly same issue.

I flashed image and reset factory configuration on my VOXL, but, It has still issue.

Did you resolve this problem?

Thanks,

kiprock · 11 Jun 2021, 16:44

@Eric-Katzfey if you cannot start docker images, you can try the command above to clear them. Good luck.

Eric Katzfey · 11 Jun 2021, 20:16

Thanks, but I think you meant to ping @shlee853

shlee853 · 12 Jun 2021, 00:12

@kiprock thank you for your reply
I’ll try it