Failed to start Stereo Camera Nodelet

abdullah.sorathi

I get this error every other time. for the first launch it executed perfectly but subsequent lanches throw error. Basically if it run perfectly and if i interrupt it by pressing ctrl + c and than run again it starts saying could not load nodelet. And many a time it says could not open camera ID = 2. Unable to open camera.

yocto:/# roslaunch voxl_cam_ros stereo.launch 
... logging to /home/root/.ros/log/a0237ab4-4d5a-11eb-9155-ec5c68cd1745/roslaunch-apq8096-4987.log
Checking log directory for disk usage. This may take awhile.
Press Ctrl-C to interrupt
Done checking log file disk usage. Usage is <1GB.

started roslaunch server http://10.42.0.157:48704/

SUMMARY
========

PARAMETERS
 * /rosdistro: indigo
 * /rosversion: 1.11.21
 * /stereo/stereo_cam_nodelet/cam_id: 2
 * /stereo/stereo_cam_nodelet/cam_name: stereo
 * /stereo/stereo_cam_nodelet/format: 1
 * /stereo/stereo_cam_nodelet/frame_id: left_stereo
 * /stereo/stereo_cam_nodelet/frame_id_right: right_stereo
 * /stereo/stereo_cam_nodelet/frame_rate: 15
 * /stereo/stereo_cam_nodelet/height: 480
 * /stereo/stereo_cam_nodelet/is_cam_master: True
 * /stereo/stereo_cam_nodelet/is_stereo: True
 * /stereo/stereo_cam_nodelet/skip_n_frames: 0
 * /stereo/stereo_cam_nodelet/width: 640

NODES
  /stereo/
    stereo_cam_nodelet (nodelet/nodelet)
    stereo_nodelet_manager (nodelet/nodelet)

auto-starting new master
process[master]: started with pid [5006]
ROS_MASTER_URI=http://localhost:11311/

setting /run_id to a0237ab4-4d5a-11eb-9155-ec5c68cd1745
process[rosout-1]: started with pid [5020]
started core service [/rosout]
process[stereo/stereo_nodelet_manager-2]: started with pid [5036]
process[stereo/stereo_cam_nodelet-3]: started with pid [5038]
[ INFO] [1609633720.545179265]: Initializing nodelet with 4 worker threads.
[ INFO] [1609633720.762515687]: SnapCamDriver Starting
[ INFO] [1609633720.762910065]: Monotonic offset: 1609633473.790296944
[FATAL] [1609633726.430193936]: Failed to load nodelet '/stereo/stereo_cam_nodelet` of type `voxl_cam_ros/stereo_driver` to manager `stereo_nodelet_manager'
[stereo/stereo_nodelet_manager-2] process has died [pid 5036, exit code -11, cmd /opt/ros/indigo/lib/nodelet/nodelet manager __name:=stereo_nodelet_manager __log:=/home/root/.ros/log/a0237ab4-4d5a-11eb-9155-ec5c68cd1745/stereo-stereo_nodelet_manager-2.log].
log file: /home/root/.ros/log/a0237ab4-4d5a-11eb-9155-ec5c68cd1745/stereo-stereo_nodelet_manager-2*.log
[stereo/stereo_cam_nodelet-3] process has died [pid 5038, exit code 255, cmd /opt/ros/indigo/lib/nodelet/nodelet load voxl_cam_ros/stereo_driver stereo_nodelet_manager __name:=stereo_cam_nodelet __log:=/home/root/.ros/log/a0237ab4-4d5a-11eb-9155-ec5c68cd1745/stereo-stereo_cam_nodelet-3.log].
log file: /home/root/.ros/log/a0237ab4-4d5a-11eb-9155-ec5c68cd1745/stereo-stereo_cam_nodelet-3*.log
^C[rosout-1] killing on exit
[master] killing on exit
shutting down processing monitor...
... shutting down processing monitor complete
done
yocto:/#

I even tried setting it as slave and not master as recommended over here [https://docs.modalai.com/voxl-cam-ros/](link url)

but after showing Monotonic offset it throws error

modaltb

Hi @abdullah-sorathi,

Can you please provide us with your hardware setup? What camera(s) do you have connected?

Can you also run voxl-version and provide us with the output?

Thanks!
Travis

Chad Sweet

Also, only one camera can be master in the libcamera architecture. If voxl-vision-px4 has VOA enabled, and is using the stereo cameras, is_cam_master needs to be set to false. In the log above is_cam_master is set to true

abdullah.sorathi

@modaltb
I am using voxl-flight deck
cameras are same that comes with flight deck-> stereo camera, hires and optical flow camera

This is the voxl version currently in use in my setup

yocto:/# voxl-version
--------------------------------------------------------------------------------
system-image:    ModalAI 2.5.2 BUILDER: ekatzfey BUILD_TIME: 2020-08-24_00:08
kernel:          #1 SMP PREEMPT Wed Sep 9 22:28:09 UTC 2020 3.18.71-perf
factory-bundle:  1.0.1
--------------------------------------------------------------------------------
architecture:    aarch64
processor:       apq8096
os:              GNU/Linux
--------------------------------------------------------------------------------
voxl-suite:
Package: voxl-suite
Version: 0.2.0
Depends: docker, imu_app (= 0.0.6), libvoxl_io (= 0.5.2), voxl-cam-manager (= 0.2.2), voxl-docker-support (= 1.1.1), voxl-hal3-tof-cam-ros (= 0.0.2), voxl-modem (= 0.10.0), voxl-nodes (= 0.0.8), voxl-rtsp (= 1.0.2), voxl-utils (= 0.5.2), voxl-vision-px4 (= 0.6.8), voxl_imu (= 0.0.4), voxl-time-sync (= 0.0.1), voxl-vpn (= 0.0.2), librc_math (= 1.1.2), libmodal_pipe (= 1.2.2), modalai-vl (= 0.1.3)
Status: install user installed
Section: base
Architecture: armv7a
Maintainer: james@modalai.com
MD5Sum: af706cd3c1ea59f274f2ed9b93141f1d
Size: 870
Filename: voxl-suite_0.2.0.ipk
Description: meta-package to install all of the voxl-suite
Installed-Time: 4024

--------------------------------------------------------------------------------
yocto:/#

I am currently working with SNAV and followed this [https://docs.modalai.com/snapdragon-navigator/]

abdullah.sorathi

@Chad-Sweet
Yes I tried both setting it to true and false. But once you start launch camera and the terminated and try to launch again it does not launch.

kiprock

@abdullah-sorathi said in Failed to start Stereo Camera Nodelet:

voxl_cam_ros stereo.launch

I have the same problem:

yocto:~# roslaunch voxl_cam_ros stereo.launch is_cam_master:=false
... logging to /home/root/.ros/log/357f3634-ac60-11eb-a8f6-ec5c68cd21c7/roslaunch-apq8096-4646.log
Checking log directory for disk usage. This may take awhile.
Press Ctrl-C to interrupt
Done checking log file disk usage. Usage is <1GB.

started roslaunch server http://10.0.0.2:34910/

SUMMARY
========

PARAMETERS
 * /rosdistro: indigo
 * /rosversion: 1.11.21
 * /stereo/stereo_cam_nodelet/cam_id: 2
 * /stereo/stereo_cam_nodelet/cam_name: stereo
 * /stereo/stereo_cam_nodelet/format: 1
 * /stereo/stereo_cam_nodelet/frame_id: left_stereo
 * /stereo/stereo_cam_nodelet/frame_id_right: right_stereo
 * /stereo/stereo_cam_nodelet/frame_rate: 15
 * /stereo/stereo_cam_nodelet/height: 480
 * /stereo/stereo_cam_nodelet/is_cam_master: False
 * /stereo/stereo_cam_nodelet/is_stereo: True
 * /stereo/stereo_cam_nodelet/skip_n_frames: 0
 * /stereo/stereo_cam_nodelet/width: 640

NODES
  /stereo/
    stereo_cam_nodelet (nodelet/nodelet)
    stereo_nodelet_manager (nodelet/nodelet)

auto-starting new master
process[master]: started with pid [4665]
ROS_MASTER_URI=http://localhost:11311/

setting /run_id to 357f3634-ac60-11eb-a8f6-ec5c68cd21c7
process[rosout-1]: started with pid [4678]
started core service [/rosout]
process[stereo/stereo_nodelet_manager-2]: started with pid [4691]
process[stereo/stereo_cam_nodelet-3]: started with pid [4696]
[ INFO] [1620081479.191290494]: Initializing nodelet with 4 worker threads.
[ INFO] [1620081479.450798205]: SnapCamDriver Starting
[ INFO] [1620081479.451256697]: Monotonic offset: 1620078140.011942802
libcam.enable.publish  1
libcam.enable.publish.dump 0
libcam.publish.buffer.policy max
libcam.publish.buffer.max 3
libcam.debug.level info
libcam.dump.dir /data/misc/camera/dumps
ERROR: could not open camera subscriber for cam id 2
[ERROR] [1620081496.766822127]: CameraManager::Initialize() failed
[ERROR] [1620081496.767214629]: Unable to open camera.

Any suggestion on debugging?

Chad Sweet

The two likely reasons for this error:

incorrect camera ID
is_cam_master not set properly.

If you are not running anything that uses the stereo camera, is_cam_master should be true

You can check the configured camera IDs by:

cat /etc/modalai/camera_env.sh

kiprock

If I set to true, it does launch but I cannot see any frame data coming out. I can for the other cameras though.

Here is the output of cat /etc/modalai/camera_env.sh


export CAM_CONFIG_ID=3
export HIRES_CAM_ID=0
export TRACKING_CAM_ID=1
export STEREO_CAM_ID=2
export TOF_CAM_ID=-1

I replaced the stereo camera kit (ordered from you guys). Cannot get the RTSP stream to show anything on VLC either, 0 and 1 works but not 2:

yocto:/etc/snav# voxl-rtsp -c 2
Camera number: 2
Simple RTSP starting
Connected
Started camera 2
Created session
Video track created
Updated camera parameters
AE Mode: 1, Exposure: 19425, Gain: 100, AWB Mode: 1
rtsp://10.0.0.2:8900/live
Started session
Ctrl-c or 'kill 4287' to end

Chad Sweet

Are you sure they are all plugged in correctly? Does pin 1 on the stereo flex align for both connecting to the PCB and each sensor to the flex?

kiprock

@Chad-Sweet Pretty sure everything is connected properly. Earlier today I started RTSP on the stereo cams and it was working, however I noticed that the left side was super blurry though, so I connected an old cam in it's place, which looked better, then I connected the orig back to double check that it wasn't just a blurry lens. At that point, I could not get anything.

If I disconnect the cameras, I cannot run the RTSP server in mode 2, if I connect the cameras, RTSP server runs but nothing shows up in VLC. I could use some help tracking down what is happening.

Chad Sweet

Hopefully you are removing power from the board before changing the camera modules?

The board is not designed to handle hot-swapping of MIPI components. The connectors also are only designed for a few dozen insertions, so they are pretty delicate.

In short, it seems like you had it working and then changed some components? Did anything else change?

kiprock

nothing else has changed.