ModalAI Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Hi-Res camera disconnect

    Ask your questions right here!
    3
    6
    190
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • wilkinsafW
      wilkinsaf ModalAI Team
      last edited by

      Hello,

      We are experiencing interruptions with our cameras. We are actively seeing it in voxl-streamer, but believe it might be having an effect on flight stability as well (we are using VIO)

      When using voxl-streamer we are able to see the video feed for a few minutes, and then it cuts out.

      The main error we see is this

      ERROR: Got 0 bytes. Expecting 40
      closing FDs for client channel 0
      

      here it is in context
      ---------- hi-res camera -----------------

      yocto:~# voxl-streamer
      Using configuration file /etc/modalai/voxl-streamer.conf
      requesting name voxl-streamer0
      with complete path /run/mpa/hires_preview/voxl-streamer0
      connected after 1 attempt(s)
      Stream available at rtsp://127.0.0.1:8900/live
      A new client has connected to the RTSP server
      An existing client has disconnected from the RTSP server
      A new client has connected to the RTSP server
      WARNING: Got 59392 bytes. Expecting 460800. skipping frame.
      WARNING: Got 59432 bytes. Expecting 460800. skipping frame.
      WARNING: Got 61400 bytes. Expecting 460800. skipping frame.
      WARNING: Got 59392 bytes. Expecting 460800. skipping frame.
      ERROR: Got 0 bytes. Expecting 40
      closing FDs for client channel 0
      Trying to stop loop
      An existing client has disconnected from the RTSP server
      voxl-streamer ending
      yocto:~#
      

      We also see this with the tracking camera
      ----------- tracking --------------

      yocto:~# voxl-streamer -c tracking-mpa
      Using configuration tracking-mpa
      Using configuration file /etc/modalai/voxl-streamer.conf
      requesting name voxl-streamer0
      with complete path /run/mpa/tracking/voxl-streamer0
      connected after 2 attempt(s)
      WARNING: Got 212992 bytes. Expecting 307200. skipping frame.
      Stream available at rtsp://127.0.0.1:8900/live
      A new client has connected to the RTSP server
      An existing client has disconnected from the RTSP server
      A new client has connected to the RTSP server
      WARNING: Got 208896 bytes. Expecting 307200. skipping frame.
      WARNING: Got 212992 bytes. Expecting 307200. skipping frame.
      WARNING: Got 208896 bytes. Expecting 307200. skipping frame.
      WARNING: Got 208896 bytes. Expecting 307200. skipping frame.
      WARNING: Got 208896 bytes. Expecting 307200. skipping frame.
      WARNING: Got 212992 bytes. Expecting 307200. skipping frame.
      WARNING: Got 208896 bytes. Expecting 307200. skipping frame.
      WARNING: Got 212992 bytes. Expecting 307200. skipping frame.
      WARNING: Got 212992 bytes. Expecting 307200. skipping frame.
      WARNING: Got 208896 bytes. Expecting 307200. skipping frame.
      WARNING: Got 208896 bytes. Expecting 307200. skipping frame.
      WARNING: Got 208896 bytes. Expecting 307200. skipping frame.
      WARNING: Got 208896 bytes. Expecting 307200. skipping frame.
      WARNING: Got 208896 bytes. Expecting 307200. skipping frame.
      WARNING: Got 208896 bytes. Expecting 307200. skipping frame.
      WARNING: Got 208896 bytes. Expecting 307200. skipping frame.
      WARNING: Got 208896 bytes. Expecting 307200. skipping frame.
      WARNING: Got 208896 bytes. Expecting 307200. skipping frame.
      ERROR: Got 0 bytes. Expecting 40
      closing FDs for client channel 0
      Trying to stop loop
      An existing client has disconnected from the RTSP server
      voxl-streamer ending
      yocto:~#
      

      I wanted to include some information from the camera server as well.
      ------------- voxl-camera-server ---------------------

      Jan 01 00:08:59 apq8096 bash[2367]: voxl-camera-server ERROR: Framenumber: 15506 ErrorCode: 4
      Jan 01 00:08:59 apq8096 bash[2367]: voxl-camera-server ERROR: Framenumber: 15507 ErrorCode: 4
      Jan 01 00:08:59 apq8096 bash[2367]: voxl-camera-server ERROR: Framenumber: 15508 ErrorCode: 4
      Jan 01 00:08:59 apq8096 bash[2367]: voxl-camera-server ERROR: Framenumber: 15513 ErrorCode: 4
      Jan 01 00:08:59 apq8096 bash[2367]: voxl-camera-server ERROR: Framenumber: 15514 ErrorCode: 4
      Jan 01 00:08:59 apq8096 bash[2367]: voxl-camera-server ERROR: Framenumber: 15516 ErrorCode: 4
      Jan 01 00:09:00 apq8096 bash[2367]: ERROR in pipe_server_send_to_client, tried to write 460800 bytes but write returned 59392
      Jan 01 00:09:00 apq8096 bash[2367]: ERROR in pipe_server_send_to_client, tried to write 460800 bytes but write returned 61440
      Jan 01 00:09:00 apq8096 bash[2367]: ERROR in pipe_server_send_to_client, tried to write 460800 bytes but write returned 59392
      Jan 01 00:09:01 apq8096 bash[2367]: voxl-camera-server ERROR: Framenumber: 15551 ErrorCode: 4
      Jan 01 00:09:01 apq8096 bash[2367]: voxl-camera-server ERROR: Framenumber: 15551 ErrorCode: 4
      Jan 01 00:09:01 apq8096 bash[2367]: voxl-camera-server ERROR: Framenumber: 15552 ErrorCode: 4
      Jan 01 00:09:01 apq8096 bash[2367]: voxl-camera-server ERROR: Framenumber: 15552 ErrorCode: 4
      

      and QVIO, but there does not appear to be a lot here

      Jan 01 00:00:08 apq8096 voxl-qvio-server[2072]: loading /data/modalai/opencv_tracking_intrinsics.yml
      Jan 01 00:00:08 apq8096 voxl-qvio-server[2072]: WARNING: disconnected from camera server, resetting VIO
      Jan 01 00:09:21 apq8096 voxl-qvio-server[2072]: WARNING: output data thread fell behind
      Jan 01 00:09:25 apq8096 voxl-qvio-server[2072]: WARNING: output data thread fell behind
      Jan 01 00:09:32 apq8096 voxl-qvio-server[2072]: WARNING: output data thread fell behind
      Jan 01 00:09:34 apq8096 voxl-qvio-server[2072]: WARNING: output data thread fell behind
      Jan 01 00:09:35 apq8096 voxl-qvio-server[2072]: WARNING: output data thread fell behind
      Jan 01 00:09:38 apq8096 voxl-qvio-server[2072]: WARNING: output data thread fell behind
      Jan 01 00:09:42 apq8096 voxl-qvio-server[2072]: WARNING: output data thread fell behind
      Jan 01 00:09:46 apq8096 voxl-qvio-server[2072]: WARNING: output data thread fell behind
      Jan 01 00:10:18 apq8096 voxl-qvio-server[2072]: WARNING: output data thread fell behind
      
      

      Questions we asked ourselves

      Q: Could it be hardware setup? Loose cable?
      A: I tried this on 2 different setups and had the same issue

      Q: Have you changed any default camera settings?
      A: No

      Q: What is your camera settings setup
      A: Hi-res + tracking (we have stereo too, but I can not seem to get any stream from them. tried voxl-inspect-cam but still not nothing)

      Q: What cameras are you using?
      A: Both tracking and hi-res from modalai

      Any insight would be much appreciated. Thank you for your time

      1 Reply Last reply Reply Quote 0
      • Chad SweetC
        Chad Sweet ModalAI Team
        last edited by

        these issues don't seem to be common.

        Which version of software are you using?

        What else is running? Those data fell behind errors indicate that too much is running on the cpu. Can you test while also running voxl-inspect-cpu?

        1 Reply Last reply Reply Quote 0
        • wilkinsafW
          wilkinsaf ModalAI Team
          last edited by

          Thank you for the quick reply @Chad-Sweet
          Here is the version information

          yocto:~# voxl-version
          cat: /etc/modalai/voxl-software-bundle-version.txt: No such file or directory
          --------------------------------------------------------------------------------
          system-image:    ModalAI 3.2.0 BUILDER: ekatzfey BUILD_TIME: 2021-02-05_19:54
          kernel:          #1 SMP PREEMPT Fri Feb 5 20:03:22 UTC 2021 3.18.71-perf
          factory-bundle:  1.0.1 (Yocto installation)
          --------------------------------------------------------------------------------
          architecture:    aarch64
          processor:       apq8096
          os:              GNU/Linux
          --------------------------------------------------------------------------------
          voxl-suite:
          Package: voxl-suite
          Version: 0.3.4
          Depends: voxl-utils (>= 0.6.0), libmodal_json (>= 0.3.4), libmodal_pipe (>= 1.7.8), librc_math (>= 1.1.4), libvoxl_io (>= 0.5.4), mavlink-camera-manager (>= 0.0.1), opencv (>= 4.5.1), openmp (>= 10.0.1), voxl-camera-server (>= 0.5.6), voxl-dfs-server (>= 0.0.7), voxl-docker-support (>= 1.1.1), voxl-gphoto2 (>= 0.0.5), voxl-hal3-tof-cam-ros (>= 0.0.5), voxl-imu-server (>= 0.7.8), voxl-modem (>= 0.11.0), voxl-mpa-tflite-server (>= 0.0.2), voxl-mpa-tools (>= 0.1.6), voxl-nodes (>= 0.1.3), voxl-qvio-server (>= 0.2.1), voxl-rtsp (>= 1.0.3), voxl-streamer (>= 0.2.1), voxl-vision-px4 (>= 0.8.1), voxl-vpn (>= 0.0.3)
          Status: install user installed
          Section: base
          Architecture: all
          Maintainer: james@modalai.com
          MD5Sum: e1e5bf1a125affe5036099f6ce895bed
          Size: 1818
          Filename: voxl-suite_0.3.4.ipk
          Description: meta-package for voxl-suite stable release
          Installed-Time: 333
          
          Package: voxl-suite
          Version: 0.2.0
          Depends: docker, imu_app (= 0.0.6), libvoxl_io (= 0.5.2), voxl-cam-manager (= 0.2.2), voxl-docker-support (= 1.1.1), voxl-hal3-tof-cam-ros (= 0.0.2), voxl-modem (= 0.10.0), voxl-nodes (= 0.0.8), voxl-rtsp (= 1.0.2), voxl-utils (= 0.5.2), voxl-vision-px4 (= 0.6.8), voxl_imu (= 0.0.4), voxl-time-sync (= 0.0.1), voxl-vpn (= 0.0.2), librc_math (= 1.1.2), libmodal_pipe (= 1.2.2), modalai-vl (= 0.1.3)
          Status: unknown ok not-installed
          Section: base
          Architecture: armv7a
          Maintainer: james@modalai.com
          MD5Sum: af706cd3c1ea59f274f2ed9b93141f1d
          Size: 870
          Filename: voxl-suite_0.2.0.ipk
          Description: meta-package to install all of the voxl-suite
          
          --------------------------------------------------------------------------------
          
          

          I will run voxl-inspect-cpu now

          1 Reply Last reply Reply Quote 0
          • wilkinsafW
            wilkinsaf ModalAI Team
            last edited by wilkinsaf

            It appears I do not have voxl-inspect-cpu

            yocto:~# ls /usr/bin/ | grep voxl
            voxl-backup
            voxl-calibrate-imu
            voxl-camera-server
            voxl-configure-camera-server
            voxl-configure-cameras
            voxl-configure-dfs
            voxl-configure-docker-support.sh
            voxl-configure-extrinsics
            voxl-configure-modem
            voxl-configure-mpa
            voxl-configure-qvio
            voxl-configure-vision-px4
            voxl-dfs-server
            voxl-docker
            voxl-env
            voxl-fan
            voxl-gphoto2
            voxl-gpio
            voxl-gpio-loopback
            voxl-hwscan
            voxl-i2c
            voxl-imu-server
            voxl-inspect-apriltag-config
            voxl-inspect-cam
            voxl-inspect-extrinsics
            voxl-inspect-imu
            voxl-inspect-points
            voxl-inspect-pose
            voxl-inspect-qvio
            voxl-inspect-services
            voxl-inspect-vibration
            voxl-kill-pipe
            voxl-logger
            voxl-modem
            voxl-modem-start.sh
            voxl-mpa-tflite-server
            voxl-perfmon
            voxl-px4-shell
            voxl-qvio-server
            voxl-replay
            voxl-reset-qvio
            voxl-rtsp
            voxl-spi
            voxl-streamer
            voxl-test-mavparser
            voxl-test-time
            voxl-time-sync
            voxl-uart-loopback
            voxl-version
            voxl-vision-px4
            voxl-vpn-init
            voxl-wifi
            
            

            Looking into how to download this tool now

            1 Reply Last reply Reply Quote 0
            • ?
              A Former User
              last edited by

              Hi,
              It looks like you are using an older version of our software bundle (0.3.2 sys image + 0.3.4 sdk). Our current (0.3.3. sys image + 0.4.6 sdk) platform release can be found on our downloads page. That release contains significant stability improvements to camera server as well as a number of other tools and will make it much easier to debug the problem if it persists.

              1 Reply Last reply Reply Quote 0
              • wilkinsafW
                wilkinsaf ModalAI Team
                last edited by

                That is great to hear. Thanks @Alex-Gardner!
                I will update.

                I assumed I was on the latest as I had seen I was on factory bundle 1.0.1 and saw on this page: https://docs.modalai.com/downloads/

                That the latest factory bundle was 1.0.1.
                32eb6578-ebff-4d43-b1ae-d2dc43d70f7e-image.png

                Should I be paying attention to the

                system-image:    ModalAI 3.2.0
                

                When checking whether the system needs an update?

                1 Reply Last reply Reply Quote 0
                • First post
                  Last post
                Powered by NodeBB | Contributors