Failed to start Stereo Camera Nodelet

abdullah.sorathi · 3 Jan 2021, 01:55

I get this error every other time. for the first launch it executed perfectly but subsequent lanches throw error. Basically if it run perfectly and if i interrupt it by pressing ctrl + c and than run again it starts saying could not load nodelet. And many a time it says could not open camera ID = 2. Unable to open camera.

yocto:/# roslaunch voxl_cam_ros stereo.launch 
... logging to /home/root/.ros/log/a0237ab4-4d5a-11eb-9155-ec5c68cd1745/roslaunch-apq8096-4987.log
Checking log directory for disk usage. This may take awhile.
Press Ctrl-C to interrupt
Done checking log file disk usage. Usage is <1GB.

started roslaunch server http://10.42.0.157:48704/

SUMMARY
========

PARAMETERS
 * /rosdistro: indigo
 * /rosversion: 1.11.21
 * /stereo/stereo_cam_nodelet/cam_id: 2
 * /stereo/stereo_cam_nodelet/cam_name: stereo
 * /stereo/stereo_cam_nodelet/format: 1
 * /stereo/stereo_cam_nodelet/frame_id: left_stereo
 * /stereo/stereo_cam_nodelet/frame_id_right: right_stereo
 * /stereo/stereo_cam_nodelet/frame_rate: 15
 * /stereo/stereo_cam_nodelet/height: 480
 * /stereo/stereo_cam_nodelet/is_cam_master: True
 * /stereo/stereo_cam_nodelet/is_stereo: True
 * /stereo/stereo_cam_nodelet/skip_n_frames: 0
 * /stereo/stereo_cam_nodelet/width: 640

NODES
  /stereo/
    stereo_cam_nodelet (nodelet/nodelet)
    stereo_nodelet_manager (nodelet/nodelet)

auto-starting new master
process[master]: started with pid [5006]
ROS_MASTER_URI=http://localhost:11311/

setting /run_id to a0237ab4-4d5a-11eb-9155-ec5c68cd1745
process[rosout-1]: started with pid [5020]
started core service [/rosout]
process[stereo/stereo_nodelet_manager-2]: started with pid [5036]
process[stereo/stereo_cam_nodelet-3]: started with pid [5038]
[ INFO] [1609633720.545179265]: Initializing nodelet with 4 worker threads.
[ INFO] [1609633720.762515687]: SnapCamDriver Starting
[ INFO] [1609633720.762910065]: Monotonic offset: 1609633473.790296944
[FATAL] [1609633726.430193936]: Failed to load nodelet '/stereo/stereo_cam_nodelet` of type `voxl_cam_ros/stereo_driver` to manager `stereo_nodelet_manager'
[stereo/stereo_nodelet_manager-2] process has died [pid 5036, exit code -11, cmd /opt/ros/indigo/lib/nodelet/nodelet manager __name:=stereo_nodelet_manager __log:=/home/root/.ros/log/a0237ab4-4d5a-11eb-9155-ec5c68cd1745/stereo-stereo_nodelet_manager-2.log].
log file: /home/root/.ros/log/a0237ab4-4d5a-11eb-9155-ec5c68cd1745/stereo-stereo_nodelet_manager-2*.log
[stereo/stereo_cam_nodelet-3] process has died [pid 5038, exit code 255, cmd /opt/ros/indigo/lib/nodelet/nodelet load voxl_cam_ros/stereo_driver stereo_nodelet_manager __name:=stereo_cam_nodelet __log:=/home/root/.ros/log/a0237ab4-4d5a-11eb-9155-ec5c68cd1745/stereo-stereo_cam_nodelet-3.log].
log file: /home/root/.ros/log/a0237ab4-4d5a-11eb-9155-ec5c68cd1745/stereo-stereo_cam_nodelet-3*.log
^C[rosout-1] killing on exit
[master] killing on exit
shutting down processing monitor...
... shutting down processing monitor complete
done
yocto:/#

I even tried setting it as slave and not master as recommended over here [https://docs.modalai.com/voxl-cam-ros/](link url)

but after showing Monotonic offset it throws error

modaltb · 4 Jan 2021, 20:30

Hi @abdullah-sorathi,

Can you please provide us with your hardware setup? What camera(s) do you have connected?

Can you also run voxl-version and provide us with the output?

Thanks!
Travis

Chad Sweet · 4 Jan 2021, 20:34

Also, only one camera can be master in the libcamera architecture. If voxl-vision-px4 has VOA enabled, and is using the stereo cameras, is_cam_master needs to be set to false. In the log above is_cam_master is set to true

abdullah.sorathi · 4 Jan 2021, 21:20

@modaltb
I am using voxl-flight deck
cameras are same that comes with flight deck-> stereo camera, hires and optical flow camera

This is the voxl version currently in use in my setup

yocto:/# voxl-version
--------------------------------------------------------------------------------
system-image:    ModalAI 2.5.2 BUILDER: ekatzfey BUILD_TIME: 2020-08-24_00:08
kernel:          #1 SMP PREEMPT Wed Sep 9 22:28:09 UTC 2020 3.18.71-perf
factory-bundle:  1.0.1
--------------------------------------------------------------------------------
architecture:    aarch64
processor:       apq8096
os:              GNU/Linux
--------------------------------------------------------------------------------
voxl-suite:
Package: voxl-suite
Version: 0.2.0
Depends: docker, imu_app (= 0.0.6), libvoxl_io (= 0.5.2), voxl-cam-manager (= 0.2.2), voxl-docker-support (= 1.1.1), voxl-hal3-tof-cam-ros (= 0.0.2), voxl-modem (= 0.10.0), voxl-nodes (= 0.0.8), voxl-rtsp (= 1.0.2), voxl-utils (= 0.5.2), voxl-vision-px4 (= 0.6.8), voxl_imu (= 0.0.4), voxl-time-sync (= 0.0.1), voxl-vpn (= 0.0.2), librc_math (= 1.1.2), libmodal_pipe (= 1.2.2), modalai-vl (= 0.1.3)
Status: install user installed
Section: base
Architecture: armv7a
Maintainer: james@modalai.com
MD5Sum: af706cd3c1ea59f274f2ed9b93141f1d
Size: 870
Filename: voxl-suite_0.2.0.ipk
Description: meta-package to install all of the voxl-suite
Installed-Time: 4024

--------------------------------------------------------------------------------
yocto:/#

I am currently working with SNAV and followed this [https://docs.modalai.com/snapdragon-navigator/]

abdullah.sorathi · 5 Jan 2021, 23:11

@Chad-Sweet
Yes I tried both setting it to true and false. But once you start launch camera and the terminated and try to launch again it does not launch.

kiprock · 3 May 2021, 22:39

@abdullah-sorathi said in Failed to start Stereo Camera Nodelet:

voxl_cam_ros stereo.launch

I have the same problem:

yocto:~# roslaunch voxl_cam_ros stereo.launch is_cam_master:=false
... logging to /home/root/.ros/log/357f3634-ac60-11eb-a8f6-ec5c68cd21c7/roslaunch-apq8096-4646.log
Checking log directory for disk usage. This may take awhile.
Press Ctrl-C to interrupt
Done checking log file disk usage. Usage is <1GB.

started roslaunch server http://10.0.0.2:34910/

SUMMARY
========

PARAMETERS
 * /rosdistro: indigo
 * /rosversion: 1.11.21
 * /stereo/stereo_cam_nodelet/cam_id: 2
 * /stereo/stereo_cam_nodelet/cam_name: stereo
 * /stereo/stereo_cam_nodelet/format: 1
 * /stereo/stereo_cam_nodelet/frame_id: left_stereo
 * /stereo/stereo_cam_nodelet/frame_id_right: right_stereo
 * /stereo/stereo_cam_nodelet/frame_rate: 15
 * /stereo/stereo_cam_nodelet/height: 480
 * /stereo/stereo_cam_nodelet/is_cam_master: False
 * /stereo/stereo_cam_nodelet/is_stereo: True
 * /stereo/stereo_cam_nodelet/skip_n_frames: 0
 * /stereo/stereo_cam_nodelet/width: 640

NODES
  /stereo/
    stereo_cam_nodelet (nodelet/nodelet)
    stereo_nodelet_manager (nodelet/nodelet)

auto-starting new master
process[master]: started with pid [4665]
ROS_MASTER_URI=http://localhost:11311/

setting /run_id to 357f3634-ac60-11eb-a8f6-ec5c68cd21c7
process[rosout-1]: started with pid [4678]
started core service [/rosout]
process[stereo/stereo_nodelet_manager-2]: started with pid [4691]
process[stereo/stereo_cam_nodelet-3]: started with pid [4696]
[ INFO] [1620081479.191290494]: Initializing nodelet with 4 worker threads.
[ INFO] [1620081479.450798205]: SnapCamDriver Starting
[ INFO] [1620081479.451256697]: Monotonic offset: 1620078140.011942802
libcam.enable.publish  1
libcam.enable.publish.dump 0
libcam.publish.buffer.policy max
libcam.publish.buffer.max 3
libcam.debug.level info
libcam.dump.dir /data/misc/camera/dumps
ERROR: could not open camera subscriber for cam id 2
[ERROR] [1620081496.766822127]: CameraManager::Initialize() failed
[ERROR] [1620081496.767214629]: Unable to open camera.

Any suggestion on debugging?

Chad Sweet · 3 May 2021, 23:21

The two likely reasons for this error:

incorrect camera ID
is_cam_master not set properly.

If you are not running anything that uses the stereo camera, is_cam_master should be true

You can check the configured camera IDs by:

cat /etc/modalai/camera_env.sh

kiprock · 4 May 2021, 00:03

If I set to true, it does launch but I cannot see any frame data coming out. I can for the other cameras though.

Here is the output of cat /etc/modalai/camera_env.sh


export CAM_CONFIG_ID=3
export HIRES_CAM_ID=0
export TRACKING_CAM_ID=1
export STEREO_CAM_ID=2
export TOF_CAM_ID=-1

I replaced the stereo camera kit (ordered from you guys). Cannot get the RTSP stream to show anything on VLC either, 0 and 1 works but not 2:

yocto:/etc/snav# voxl-rtsp -c 2
Camera number: 2
Simple RTSP starting
Connected
Started camera 2
Created session
Video track created
Updated camera parameters
AE Mode: 1, Exposure: 19425, Gain: 100, AWB Mode: 1
rtsp://10.0.0.2:8900/live
Started session
Ctrl-c or 'kill 4287' to end

Chad Sweet · 4 May 2021, 00:11

Are you sure they are all plugged in correctly? Does pin 1 on the stereo flex align for both connecting to the PCB and each sensor to the flex?

kiprock · 4 May 2021, 00:17

@Chad-Sweet Pretty sure everything is connected properly. Earlier today I started RTSP on the stereo cams and it was working, however I noticed that the left side was super blurry though, so I connected an old cam in it's place, which looked better, then I connected the orig back to double check that it wasn't just a blurry lens. At that point, I could not get anything.

If I disconnect the cameras, I cannot run the RTSP server in mode 2, if I connect the cameras, RTSP server runs but nothing shows up in VLC. I could use some help tracking down what is happening.

Chad Sweet · 4 May 2021, 00:40

Hopefully you are removing power from the board before changing the camera modules?

The board is not designed to handle hot-swapping of MIPI components. The connectors also are only designed for a few dozen insertions, so they are pretty delicate.

In short, it seems like you had it working and then changed some components? Did anything else change?

kiprock · 4 May 2021, 00:44

nothing else has changed.