ModalAI Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Hi-Res camera disconnect

    Ask your questions right here!
    3
    6
    140
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • wilkinsafW
      wilkinsaf Dev Team
      last edited by

      Hello,

      We are experiencing interruptions with our cameras. We are actively seeing it in voxl-streamer, but believe it might be having an effect on flight stability as well (we are using VIO)

      When using voxl-streamer we are able to see the video feed for a few minutes, and then it cuts out.

      The main error we see is this

      ERROR: Got 0 bytes. Expecting 40
closing FDs for client channel 0

      here it is in context
      ---------- hi-res camera -----------------

      yocto:~# voxl-streamer
Using configuration file /etc/modalai/voxl-streamer.conf
requesting name voxl-streamer0
with complete path /run/mpa/hires_preview/voxl-streamer0
connected after 1 attempt(s)
Stream available at rtsp://127.0.0.1:8900/live
A new client has connected to the RTSP server
An existing client has disconnected from the RTSP server
A new client has connected to the RTSP server
WARNING: Got 59392 bytes. Expecting 460800. skipping frame.
WARNING: Got 59432 bytes. Expecting 460800. skipping frame.
WARNING: Got 61400 bytes. Expecting 460800. skipping frame.
WARNING: Got 59392 bytes. Expecting 460800. skipping frame.
ERROR: Got 0 bytes. Expecting 40
closing FDs for client channel 0
Trying to stop loop
An existing client has disconnected from the RTSP server
voxl-streamer ending
yocto:~#

      We also see this with the tracking camera
      ----------- tracking --------------

      yocto:~# voxl-streamer -c tracking-mpa
Using configuration tracking-mpa
Using configuration file /etc/modalai/voxl-streamer.conf
requesting name voxl-streamer0
with complete path /run/mpa/tracking/voxl-streamer0
connected after 2 attempt(s)
WARNING: Got 212992 bytes. Expecting 307200. skipping frame.
Stream available at rtsp://127.0.0.1:8900/live
A new client has connected to the RTSP server
An existing client has disconnected from the RTSP server
A new client has connected to the RTSP server
WARNING: Got 208896 bytes. Expecting 307200. skipping frame.
WARNING: Got 212992 bytes. Expecting 307200. skipping frame.
WARNING: Got 208896 bytes. Expecting 307200. skipping frame.
WARNING: Got 208896 bytes. Expecting 307200. skipping frame.
WARNING: Got 208896 bytes. Expecting 307200. skipping frame.
WARNING: Got 212992 bytes. Expecting 307200. skipping frame.
WARNING: Got 208896 bytes. Expecting 307200. skipping frame.
WARNING: Got 212992 bytes. Expecting 307200. skipping frame.
WARNING: Got 212992 bytes. Expecting 307200. skipping frame.
WARNING: Got 208896 bytes. Expecting 307200. skipping frame.
WARNING: Got 208896 bytes. Expecting 307200. skipping frame.
WARNING: Got 208896 bytes. Expecting 307200. skipping frame.
WARNING: Got 208896 bytes. Expecting 307200. skipping frame.
WARNING: Got 208896 bytes. Expecting 307200. skipping frame.
WARNING: Got 208896 bytes. Expecting 307200. skipping frame.
WARNING: Got 208896 bytes. Expecting 307200. skipping frame.
WARNING: Got 208896 bytes. Expecting 307200. skipping frame.
WARNING: Got 208896 bytes. Expecting 307200. skipping frame.
ERROR: Got 0 bytes. Expecting 40
closing FDs for client channel 0
Trying to stop loop
An existing client has disconnected from the RTSP server
voxl-streamer ending
yocto:~#

      I wanted to include some information from the camera server as well.
      ------------- voxl-camera-server ---------------------

      Jan 01 00:08:59 apq8096 bash[2367]: voxl-camera-server ERROR: Framenumber: 15506 ErrorCode: 4
Jan 01 00:08:59 apq8096 bash[2367]: voxl-camera-server ERROR: Framenumber: 15507 ErrorCode: 4
Jan 01 00:08:59 apq8096 bash[2367]: voxl-camera-server ERROR: Framenumber: 15508 ErrorCode: 4
Jan 01 00:08:59 apq8096 bash[2367]: voxl-camera-server ERROR: Framenumber: 15513 ErrorCode: 4
Jan 01 00:08:59 apq8096 bash[2367]: voxl-camera-server ERROR: Framenumber: 15514 ErrorCode: 4
Jan 01 00:08:59 apq8096 bash[2367]: voxl-camera-server ERROR: Framenumber: 15516 ErrorCode: 4
Jan 01 00:09:00 apq8096 bash[2367]: ERROR in pipe_server_send_to_client, tried to write 460800 bytes but write returned 59392
Jan 01 00:09:00 apq8096 bash[2367]: ERROR in pipe_server_send_to_client, tried to write 460800 bytes but write returned 61440
Jan 01 00:09:00 apq8096 bash[2367]: ERROR in pipe_server_send_to_client, tried to write 460800 bytes but write returned 59392
Jan 01 00:09:01 apq8096 bash[2367]: voxl-camera-server ERROR: Framenumber: 15551 ErrorCode: 4
Jan 01 00:09:01 apq8096 bash[2367]: voxl-camera-server ERROR: Framenumber: 15551 ErrorCode: 4
Jan 01 00:09:01 apq8096 bash[2367]: voxl-camera-server ERROR: Framenumber: 15552 ErrorCode: 4
Jan 01 00:09:01 apq8096 bash[2367]: voxl-camera-server ERROR: Framenumber: 15552 ErrorCode: 4

      and QVIO, but there does not appear to be a lot here

      Jan 01 00:00:08 apq8096 voxl-qvio-server[2072]: loading /data/modalai/opencv_tracking_intrinsics.yml
Jan 01 00:00:08 apq8096 voxl-qvio-server[2072]: WARNING: disconnected from camera server, resetting VIO
Jan 01 00:09:21 apq8096 voxl-qvio-server[2072]: WARNING: output data thread fell behind
Jan 01 00:09:25 apq8096 voxl-qvio-server[2072]: WARNING: output data thread fell behind
Jan 01 00:09:32 apq8096 voxl-qvio-server[2072]: WARNING: output data thread fell behind
Jan 01 00:09:34 apq8096 voxl-qvio-server[2072]: WARNING: output data thread fell behind
Jan 01 00:09:35 apq8096 voxl-qvio-server[2072]: WARNING: output data thread fell behind
Jan 01 00:09:38 apq8096 voxl-qvio-server[2072]: WARNING: output data thread fell behind
Jan 01 00:09:42 apq8096 voxl-qvio-server[2072]: WARNING: output data thread fell behind
Jan 01 00:09:46 apq8096 voxl-qvio-server[2072]: WARNING: output data thread fell behind
Jan 01 00:10:18 apq8096 voxl-qvio-server[2072]: WARNING: output data thread fell behind

      Questions we asked ourselves

      Q: Could it be hardware setup? Loose cable?
      A: I tried this on 2 different setups and had the same issue

      Q: Have you changed any default camera settings?
      A: No

      Q: What is your camera settings setup
      A: Hi-res + tracking (we have stereo too, but I can not seem to get any stream from them. tried voxl-inspect-cam but still not nothing)

      Q: What cameras are you using?
      A: Both tracking and hi-res from modalai

      Any insight would be much appreciated. Thank you for your time

      1 Reply Last reply Reply Quote 0
      • Chad SweetC
        Chad Sweet Dev Team
        last edited by

        these issues don't seem to be common.

        Which version of software are you using?

        What else is running? Those data fell behind errors indicate that too much is running on the cpu. Can you test while also running voxl-inspect-cpu?

        1 Reply Last reply Reply Quote 0
        • wilkinsafW
          wilkinsaf Dev Team
          last edited by

          Thank you for the quick reply @Chad-Sweet
          Here is the version information

          yocto:~# voxl-version
cat: /etc/modalai/voxl-software-bundle-version.txt: No such file or directory
--------------------------------------------------------------------------------
system-image:    ModalAI 3.2.0 BUILDER: ekatzfey BUILD_TIME: 2021-02-05_19:54
kernel:          #1 SMP PREEMPT Fri Feb 5 20:03:22 UTC 2021 3.18.71-perf
factory-bundle:  1.0.1 (Yocto installation)
--------------------------------------------------------------------------------
architecture:    aarch64
processor:       apq8096
os:              GNU/Linux
--------------------------------------------------------------------------------
voxl-suite:
Package: voxl-suite
Version: 0.3.4
Depends: voxl-utils (>= 0.6.0), libmodal_json (>= 0.3.4), libmodal_pipe (>= 1.7.8), librc_math (>= 1.1.4), libvoxl_io (>= 0.5.4), mavlink-camera-manager (>= 0.0.1), opencv (>= 4.5.1), openmp (>= 10.0.1), voxl-camera-server (>= 0.5.6), voxl-dfs-server (>= 0.0.7), voxl-docker-support (>= 1.1.1), voxl-gphoto2 (>= 0.0.5), voxl-hal3-tof-cam-ros (>= 0.0.5), voxl-imu-server (>= 0.7.8), voxl-modem (>= 0.11.0), voxl-mpa-tflite-server (>= 0.0.2), voxl-mpa-tools (>= 0.1.6), voxl-nodes (>= 0.1.3), voxl-qvio-server (>= 0.2.1), voxl-rtsp (>= 1.0.3), voxl-streamer (>= 0.2.1), voxl-vision-px4 (>= 0.8.1), voxl-vpn (>= 0.0.3)
Status: install user installed
Section: base
Architecture: all
Maintainer: james@modalai.com
MD5Sum: e1e5bf1a125affe5036099f6ce895bed
Size: 1818
Filename: voxl-suite_0.3.4.ipk
Description: meta-package for voxl-suite stable release
Installed-Time: 333

Package: voxl-suite
Version: 0.2.0
Depends: docker, imu_app (= 0.0.6), libvoxl_io (= 0.5.2), voxl-cam-manager (= 0.2.2), voxl-docker-support (= 1.1.1), voxl-hal3-tof-cam-ros (= 0.0.2), voxl-modem (= 0.10.0), voxl-nodes (= 0.0.8), voxl-rtsp (= 1.0.2), voxl-utils (= 0.5.2), voxl-vision-px4 (= 0.6.8), voxl_imu (= 0.0.4), voxl-time-sync (= 0.0.1), voxl-vpn (= 0.0.2), librc_math (= 1.1.2), libmodal_pipe (= 1.2.2), modalai-vl (= 0.1.3)
Status: unknown ok not-installed
Section: base
Architecture: armv7a
Maintainer: james@modalai.com
MD5Sum: af706cd3c1ea59f274f2ed9b93141f1d
Size: 870
Filename: voxl-suite_0.2.0.ipk
Description: meta-package to install all of the voxl-suite

--------------------------------------------------------------------------------

          I will run voxl-inspect-cpu now

          1 Reply Last reply Reply Quote 0
          • wilkinsafW
            wilkinsaf Dev Team
            last edited by wilkinsaf

            It appears I do not have voxl-inspect-cpu

            yocto:~# ls /usr/bin/ | grep voxl
voxl-backup
voxl-calibrate-imu
voxl-camera-server
voxl-configure-camera-server
voxl-configure-cameras
voxl-configure-dfs
voxl-configure-docker-support.sh
voxl-configure-extrinsics
voxl-configure-modem
voxl-configure-mpa
voxl-configure-qvio
voxl-configure-vision-px4
voxl-dfs-server
voxl-docker
voxl-env
voxl-fan
voxl-gphoto2
voxl-gpio
voxl-gpio-loopback
voxl-hwscan
voxl-i2c
voxl-imu-server
voxl-inspect-apriltag-config
voxl-inspect-cam
voxl-inspect-extrinsics
voxl-inspect-imu
voxl-inspect-points
voxl-inspect-pose
voxl-inspect-qvio
voxl-inspect-services
voxl-inspect-vibration
voxl-kill-pipe
voxl-logger
voxl-modem
voxl-modem-start.sh
voxl-mpa-tflite-server
voxl-perfmon
voxl-px4-shell
voxl-qvio-server
voxl-replay
voxl-reset-qvio
voxl-rtsp
voxl-spi
voxl-streamer
voxl-test-mavparser
voxl-test-time
voxl-time-sync
voxl-uart-loopback
voxl-version
voxl-vision-px4
voxl-vpn-init
voxl-wifi

            Looking into how to download this tool now

            1 Reply Last reply Reply Quote 0
            • ?
              A Former User
              last edited by

              Hi,
              It looks like you are using an older version of our software bundle (0.3.2 sys image + 0.3.4 sdk). Our current (0.3.3. sys image + 0.4.6 sdk) platform release can be found on our downloads page. That release contains significant stability improvements to camera server as well as a number of other tools and will make it much easier to debug the problem if it persists.

              1 Reply Last reply Reply Quote 0
              • wilkinsafW
                wilkinsaf Dev Team
                last edited by

                That is great to hear. Thanks @Alex-Gardner!
                I will update.

                I assumed I was on the latest as I had seen I was on factory bundle 1.0.1 and saw on this page: https://docs.modalai.com/downloads/

                That the latest factory bundle was 1.0.1.
                32eb6578-ebff-4d43-b1ae-d2dc43d70f7e-image.png

                Should I be paying attention to the

                system-image:    ModalAI 3.2.0

                When checking whether the system needs an update?

                1 Reply Last reply Reply Quote 0
                • First post
                  Last post
                Powered by NodeBB | Contributors