Camera server keep crashing everytime it is launched
-
@Judoor-0 , ok, yes it looks like TOF is the cause of the crash.
Can you please capture output of
voxl-camera-server -d 0(to enable full debug) with only TOF enabled. The output will be quite long, can you please paste it using a "code block" (hopefully will get scroll bars), or a link to a file, otherwise the thread will get super long!You can stop the background service using
systemctl stop voxl-camera-serverand just runvoxl-camera-server -d 0in foreground in a terminal.Alex
-
@Alex-Kushleyev Yes here it is :
oxl2:/etc/modalai$ voxl-camera-server -d 0 detected system image 1.7.6 using new imx214 defaults ================================================================= configuration for 3 cameras: cam #0 name: tof sensor type: pmd-tof isEnabled: 1 camId: 0 camId2: -1 fps: 5 en_rotate: 0 en_rotate2: 0 en_preview: 1 pre_width: 224 pre_height: 1557 en_raw_preview: 1 en_small_video: 0 small_video_width: -1 small_video_height: -1 en_large_video: 0 large_video_width: -1 large_video_height: -1 en_snapshot: 0 snap_width: -1 snap_height: -1 ae_mode: off standby_enabled: 0 decimator: 5 independent_exposure:0 cam #1 name: hires sensor type: imx214 isEnabled: 0 camId: 1 camId2: -1 fps: 30 en_rotate: 0 en_rotate2: 0 en_preview: 0 pre_width: 640 pre_height: 480 en_raw_preview: 0 en_small_video: 1 small_video_width: 1024 small_video_height: 768 en_large_video: 1 large_video_width: 4208 large_video_height: 3120 en_snapshot: 1 snap_width: 4208 snap_height: 3120 ae_mode: isp standby_enabled: 0 decimator: 1 independent_exposure:0 cam #2 name: tracking sensor type: ov7251 isEnabled: 0 camId: 2 camId2: -1 fps: 30 en_rotate: 0 en_rotate2: 0 en_preview: 1 pre_width: 640 pre_height: 480 en_raw_preview: 1 en_small_video: 0 small_video_width: -1 small_video_height: -1 en_large_video: 0 large_video_width: -1 large_video_height: -1 en_snapshot: 0 snap_width: -1 snap_height: -1 ae_mode: lme_msv standby_enabled: 0 decimator: 1 independent_exposure:0 fsync_en: 0 fsync_gpio: 111 ================================================================= DEBUG: Attempting to open the hal module DEBUG: SUCCESS: Camera module opened on attempt 0 DEBUG: ----------- Number of cameras: 2 DEBUG: Cam idx: 0, Cam slot: 2, Slave Address: 0x0020, Sensor Id: 0x0214 DEBUG: Cam idx: 1, Cam slot: 3, Slave Address: 0x00E2, Sensor Id: 0x7750 connected to GPS mavlink pipe DEBUG: ------ voxl-camera-server: Starting 3 cameras Starting Camera: tof (id #0) Connected to cpu-monitor DEBUG: Checking Gain limits for Camera: tof Using gain limits min: 54 max: 431 DEBUG: cam ID 0 checking for fmt: 38 w: 224 h: 1557 o: 0 DEBUG: ==> option 0: format= 34 width=4208 height=3120 is_input=0 DEBUG: ==> option 4: format= 34 width=4208 height=3120 is_input=1 DEBUG: ==> option 8: format= 35 width=4208 height=3120 is_input=0 DEBUG: ==> option 12: format= 35 width=4208 height=3120 is_input=1 DEBUG: ==> option 16: format= 33 width=4208 height=3120 is_input=0 DEBUG: ==> option 20: format= 34 width=4160 height=3120 is_input=0 DEBUG: ==> option 24: format= 34 width=4160 height=3120 is_input=1 DEBUG: ==> option 28: format= 35 width=4160 height=3120 is_input=0 DEBUG: ==> option 32: format= 35 width=4160 height=3120 is_input=1 DEBUG: ==> option 36: format= 33 width=4160 height=3120 is_input=0 DEBUG: ==> option 40: format= 34 width=4096 height=2304 is_input=0 DEBUG: ==> option 44: format= 34 width=4096 height=2304 is_input=1 DEBUG: ==> option 48: format= 35 width=4096 height=2304 is_input=0 DEBUG: ==> option 52: format= 35 width=4096 height=2304 is_input=1 DEBUG: ==> option 56: format= 33 width=4096 height=2304 is_input=0 DEBUG: ==> option 60: format= 34 width=4096 height=2160 is_input=0 DEBUG: ==> option 64: format= 34 width=4096 height=2160 is_input=1 DEBUG: ==> option 68: format= 35 width=4096 height=2160 is_input=0 DEBUG: ==> option 72: format= 35 width=4096 height=2160 is_input=1 DEBUG: ==> option 76: format= 33 width=4096 height=2160 is_input=0 DEBUG: ==> option 80: format= 34 width=4056 height=3040 is_input=0 DEBUG: ==> option 84: format= 34 width=4056 height=3040 is_input=1 DEBUG: ==> option 88: format= 35 width=4056 height=3040 is_input=0 DEBUG: ==> option 92: format= 35 width=4056 height=3040 is_input=1 DEBUG: ==> option 96: format= 33 width=4056 height=3040 is_input=0 DEBUG: ==> option 100: format= 34 width=4000 height=3000 is_input=0 DEBUG: ==> option 104: format= 34 width=4000 height=3000 is_input=1 DEBUG: ==> option 108: format= 35 width=4000 height=3000 is_input=0 DEBUG: ==> option 112: format= 35 width=4000 height=3000 is_input=1 DEBUG: ==> option 116: format= 33 width=4000 height=3000 is_input=0 DEBUG: ==> option 120: format= 34 width=3840 height=2160 is_input=0 DEBUG: ==> option 124: format= 34 width=3840 height=2160 is_input=1 DEBUG: ==> option 128: format= 35 width=3840 height=2160 is_input=0 DEBUG: ==> option 132: format= 35 width=3840 height=2160 is_input=1 DEBUG: ==> option 136: format= 33 width=3840 height=2160 is_input=0 DEBUG: ==> option 140: format= 34 width=3264 height=2448 is_input=0 DEBUG: ==> option 144: format= 34 width=3264 height=2448 is_input=1 DEBUG: ==> option 148: format= 35 width=3264 height=2448 is_input=0 DEBUG: ==> option 152: format= 35 width=3264 height=2448 is_input=1 DEBUG: ==> option 156: format= 33 width=3264 height=2448 is_input=0 DEBUG: ==> option 160: format= 34 width=3200 height=2400 is_input=0 DEBUG: ==> option 164: format= 34 width=3200 height=2400 is_input=1 DEBUG: ==> option 168: format= 35 width=3200 height=2400 is_input=0 DEBUG: ==> option 172: format= 35 width=3200 height=2400 is_input=1 DEBUG: ==> option 176: format= 33 width=3200 height=2400 is_input=0 DEBUG: ==> option 180: format= 34 width=2976 height=2976 is_input=0 DEBUG: ==> option 184: format= 34 width=2976 height=2976 is_input=1 DEBUG: ==> option 188: format= 35 width=2976 height=2976 is_input=0 DEBUG: ==> option 192: format= 35 width=2976 height=2976 is_input=1 DEBUG: ==> option 196: format= 33 width=2976 height=2976 is_input=0 DEBUG: ==> option 200: format= 34 width=2688 height=1512 is_input=0 DEBUG: ==> option 204: format= 35 width=2688 height=1512 is_input=0 DEBUG: ==> option 208: format= 33 width=2688 height=1512 is_input=0 DEBUG: ==> option 212: format= 34 width=2592 height=1944 is_input=0 DEBUG: ==> option 216: format= 34 width=2592 height=1944 is_input=1 DEBUG: ==> option 220: format= 35 width=2592 height=1944 is_input=0 DEBUG: ==> option 224: format= 35 width=2592 height=1944 is_input=1 DEBUG: ==> option 228: format= 33 width=2592 height=1944 is_input=0 DEBUG: ==> option 232: format= 34 width=2048 height=1536 is_input=0 DEBUG: ==> option 236: format= 35 width=2048 height=1536 is_input=0 DEBUG: ==> option 240: format= 33 width=2048 height=1536 is_input=0 DEBUG: ==> option 244: format= 34 width=1920 height=1440 is_input=0 DEBUG: ==> option 248: format= 35 width=1920 height=1440 is_input=0 DEBUG: ==> option 252: format= 33 width=1920 height=1440 is_input=0 DEBUG: ==> option 256: format= 34 width=1928 height=1208 is_input=0 DEBUG: ==> option 260: format= 35 width=1928 height=1208 is_input=0 DEBUG: ==> option 264: format= 33 width=1928 height=1208 is_input=0 DEBUG: ==> option 268: format= 34 width=1920 height=1080 is_input=0 DEBUG: ==> option 272: format= 35 width=1920 height=1080 is_input=0 DEBUG: ==> option 276: format= 33 width=1920 height=1080 is_input=0 DEBUG: ==> option 280: format= 34 width=1600 height=1200 is_input=0 DEBUG: ==> option 284: format= 35 width=1600 height=1200 is_input=0 DEBUG: ==> option 288: format= 33 width=1600 height=1200 is_input=0 DEBUG: ==> option 292: format= 34 width=1440 height=1080 is_input=0 DEBUG: ==> option 296: format= 35 width=1440 height=1080 is_input=0 DEBUG: ==> option 300: format= 33 width=1440 height=1080 is_input=0 DEBUG: ==> option 304: format= 34 width=1280 height= 960 is_input=0 DEBUG: ==> option 308: format= 35 width=1280 height= 960 is_input=0 DEBUG: ==> option 312: format= 33 width=1280 height= 960 is_input=0 DEBUG: ==> option 316: format= 34 width=1280 height= 800 is_input=0 DEBUG: ==> option 320: format= 35 width=1280 height= 800 is_input=0 DEBUG: ==> option 324: format= 33 width=1280 height= 800 is_input=0 DEBUG: ==> option 328: format= 34 width=1280 height= 768 is_input=0 DEBUG: ==> option 332: format= 35 width=1280 height= 768 is_input=0 DEBUG: ==> option 336: format= 33 width=1280 height= 768 is_input=0 DEBUG: ==> option 340: format= 34 width=1280 height= 720 is_input=0 DEBUG: ==> option 344: format= 35 width=1280 height= 720 is_input=0 DEBUG: ==> option 348: format= 33 width=1280 height= 720 is_input=0 DEBUG: ==> option 352: format= 34 width=1080 height=1080 is_input=0 DEBUG: ==> option 356: format= 35 width=1080 height=1080 is_input=0 DEBUG: ==> option 360: format= 33 width=1080 height=1080 is_input=0 DEBUG: ==> option 364: format= 34 width=1024 height= 738 is_input=0 DEBUG: ==> option 368: format= 35 width=1024 height= 738 is_input=0 DEBUG: ==> option 372: format= 33 width=1024 height= 738 is_input=0 DEBUG: ==> option 376: format= 34 width=1024 height= 768 is_input=0 DEBUG: ==> option 380: format= 35 width=1024 height= 768 is_input=0 DEBUG: ==> option 384: format= 33 width=1024 height= 768 is_input=0 DEBUG: ==> option 388: format= 34 width= 864 height= 480 is_input=0 DEBUG: ==> option 392: format= 35 width= 864 height= 480 is_input=0 DEBUG: ==> option 396: format= 33 width= 864 height= 480 is_input=0 DEBUG: ==> option 400: format= 34 width= 800 height= 600 is_input=0 DEBUG: ==> option 404: format= 35 width= 800 height= 600 is_input=0 DEBUG: ==> option 408: format= 33 width= 800 height= 600 is_input=0 DEBUG: ==> option 412: format= 34 width= 800 height= 480 is_input=0 DEBUG: ==> option 416: format= 35 width= 800 height= 480 is_input=0 DEBUG: ==> option 420: format= 33 width= 800 height= 480 is_input=0 DEBUG: ==> option 424: format= 34 width= 720 height=1280 is_input=0 DEBUG: ==> option 428: format= 35 width= 720 height=1280 is_input=0 DEBUG: ==> option 432: format= 33 width= 720 height=1280 is_input=0 DEBUG: ==> option 436: format= 34 width= 720 height= 480 is_input=0 DEBUG: ==> option 440: format= 35 width= 720 height= 480 is_input=0 DEBUG: ==> option 444: format= 33 width= 720 height= 480 is_input=0 DEBUG: ==> option 448: format= 34 width= 640 height= 480 is_input=0 DEBUG: ==> option 452: format= 35 width= 640 height= 480 is_input=0 DEBUG: ==> option 456: format= 33 width= 640 height= 480 is_input=0 DEBUG: ==> option 460: format= 34 width= 640 height= 400 is_input=0 DEBUG: ==> option 464: format= 35 width= 640 height= 400 is_input=0 DEBUG: ==> option 468: format= 33 width= 640 height= 400 is_input=0 DEBUG: ==> option 472: format= 34 width= 640 height= 360 is_input=0 DEBUG: ==> option 476: format= 35 width= 640 height= 360 is_input=0 DEBUG: ==> option 480: format= 33 width= 640 height= 360 is_input=0 DEBUG: ==> option 484: format= 34 width= 352 height= 288 is_input=0 DEBUG: ==> option 488: format= 35 width= 352 height= 288 is_input=0 DEBUG: ==> option 492: format= 33 width= 352 height= 288 is_input=0 DEBUG: ==> option 496: format= 34 width= 320 height= 240 is_input=0 DEBUG: ==> option 500: format= 35 width= 320 height= 240 is_input=0 DEBUG: ==> option 504: format= 33 width= 320 height= 240 is_input=0 DEBUG: ==> option 508: format= 34 width= 240 height= 320 is_input=0 DEBUG: ==> option 512: format= 35 width= 240 height= 320 is_input=0 DEBUG: ==> option 516: format= 33 width= 240 height= 320 is_input=0 DEBUG: ==> option 520: format= 34 width= 176 height= 144 is_input=0 DEBUG: ==> option 524: format= 35 width= 176 height= 144 is_input=0 DEBUG: ==> option 528: format= 33 width= 176 height= 144 is_input=0 DEBUG: ==> option 532: format= 37 width=4208 height=3120 is_input=0 DEBUG: ==> option 536: format= 38 width=4208 height=3120 is_input=0 DEBUG: ==> option 540: format= 32 width=4208 height=3120 is_input=0 DEBUG: ==> option 544: format= 36 width=4208 height=3120 is_input=0 ERROR: Camera 0 failed to find supported preview config: 224x1557 WARNING: Failed to start cam tof due to invalid resolution WARNING: assuming cam is missing and trying to compensate Skipping Camera: hires, configuration marked disabled Skipping Camera: tracking, configuration marked disabled ERROR: failed to initialize any cameras ------ voxl-camera-server: Started 0 of 3 cameras ------ voxl-camera-server: Camera server is now running ------ voxl-camera-server: Camera server is now stopping DEBUG: Erasing all managers ------ voxl-camera-server: Camera server exited gracefully, returning -1 -
@Judoor-0 , it seems in this test TOF has not even been detected, you can see two cameras found in slots 2 and 3 (imx214 hi-res camera and ov7251 tracking camera).
Did anything change? (Tof unplugged?)
Just fyi, even if you disable cameras in the config, they will still be detected.
-
I am experiencing the same issue with the new TOF in cam config 6, which is also selection 21 in configure-cameras. I cannot get the beta tof sensor to work with the hires and tracking although hires/tracking and tof all work when configured independently. I tried changing the id:0 type to "pmd-tof-liow2" as suggested in the tof datasheet for the config file but that didn't work either. the only config that gets the new beta tof to work for me is #24.
-
@Alex-Kushleyev Well the tof sensor work from time to time but only for a short period.
When it works, I've got point cloud in voxl Portal (but it's upside down) and after a bit, it just stop functioning. It's quite random and hard to reproduce. -
@Gary-Holmgren , can you send me a picture or diagram that shows where hires, tracking and new TOF sensor are plugged in? Are they plugged in as C06 configuration, except instead of old TOF, you have the new TOF module? https://docs.modalai.com/voxl2-camera-configs/#c-06-tof-hires-and-tracking-d0005-starling
-
@Judoor-0 , have you used the TOF sensor recently for extended periods of time (before upgrading to SDK 1.2.0 ?). Extended, i mean, longer than just a few minutes.
I am just trying to figure out whether this is an old hardware problem or a new software problem (or potentially new hardware problem).
Also, what adapter do you use to plug the TOF sensor into VOXL2 board (the first adapter that plugs into VOXL2). Is it M0076 or M0135 or something else?
Alex
-
@Alex-Kushleyev No I didn't use the TOF sensor before, I almost directly updated the sdk.
But what I can recall is that even on the original 1.1.1 sdk I didn't saw any point cloud on the tof_pc on voxl portal. But I thought it was because I had to set something up for it to work.
On the version 1.2.0 I did saw once (as mention before) some point cloud appear on voxl-portal but after 3 minutes it crashed and I've never seen it since.
The cable that come from the TOF sensor is MFPC-M0040 REV A and the adapter for the board is M0135.
Julien -
@Alex-Kushleyev Yes, my setup is the same as that photo in the C06 link you posted
-
@Alex-Kushleyev I really need this sensor to work so can you tell me if you think that it is a hardware problem I need to change the faulty piece as quick as possible. I'm right now making a new order for another starling 2 so I need to know.
Otherwise, what can I do to debug this sensor.
Julien -
@Judoor-0 , I am sorry that this happened to you.
It seems that there is some hardware issue.
Let's do a quick test by connecting TOF to a different camera port and updating camera server configuration to reflect this:
- i am assuming your 3 cameras (tof, hires, tracking) are connected in the following configuration : https://docs.modalai.com/voxl2-camera-configs/#c-06-tof-hires-and-tracking-d0005-starling
- this means TOF should be in slot 0, IMX214 in slot 2, ov7251 (tracking) in slot 3. the last two i can see confirmed from the log that you posted (where TOF was not detected).
- power off voxl2
- remove the flex cable witih IMX214 and tracking camera from J7
- disconnect M0135 + TOF from J6 and plug it into J7 (same orientation as it was plugged into J6)
- power on voxl2
- copy the TOF driver for slot 2: /usr/share/modalai/chi-cdk/irs1645/com.qti.sensormodule.irs1645_2.bin
- run
voxl-camera-server -lto see if TOF is detected in slot 2 - edit /etc/voxl-camera-server.conf to disable hires and tracking cameras (enabled : false), confirm that TOF camera ID is set to 0 and TOF is enabled in the config file
- start voxl-camera-server in foreground and see if TOF is functioning normally.
Please let me know the result.
Alex
-
@Alex-Kushleyev Here is my starling 2 camera configuration the only difference is that I also have a M0135 module to connect the imx214 and tracking.
So, I unpluged the cameras, plug the TOF sensor on the other M0135 adapter.
I copied the driver in /usr/lib/camera and delete the existing one for slot 2 (it was imx214)
Now when I run voxl-camera-server -l I got this, so I suppose it correctly detected the TOF sensor.voxl2:/$ voxl-camera-server -l DEBUG: Attempting to open the hal module DEBUG: SUCCESS: Camera module opened on attempt 0 DEBUG: ----------- Number of cameras: 1 DEBUG: Cam idx: 0, Cam slot: 2, Slave Address: 0x007A, Sensor Id: 0x003D DEBUG: Note: This list comes from the HAL module and may not be indicative DEBUG: of configurations that have full pipelines DEBUG: Number of cameras: 1 ==================================== Stats for camera: 0 ANDROID_SCALER_AVAILABLE_RAW_SIZES: These are likely supported by the sensor 224 x 1557 ANDROID_SCALER_AVAILABLE_STREAM_CONFIGURATIONS: These are NOT necessarily supported by the sensor 176 x 144 HAL_PIXEL_FORMAT_YCbCr_420_888 176 x 144 HAL_PIXEL_FORMAT_BLOB 224 x 1557 HAL_PIXEL_FORMAT_RAW10 224 x 1557 HAL_PIXEL_FORMAT_RAW12 224 x 1557 HAL_PIXEL_FORMAT_RAW16 224 x 1557 HAL_PIXEL_FORMAT_RAW_OPAQUE ANDROID_SENSOR_INFO_SENSITIVITY_RANGE min = 54 max = 0 ANDROID_SENSOR_MAX_ANALOG_SENSITIVITY 0 ANDROID_SENSOR_INFO_EXPOSURE_TIME_RANGE min = 0ns max = 0ns ==================================== Number of cameras detected: 1 ====================================Just a note here, I found weird that the recommended resolution for the camera is 224x1557. Shouldn't it be 224x171 ?
Then when I disable the other cam and start the voxl-camera-server, I still got this with voxl-inspect-cam -a
And I've got the same errors when I directly run voxl-camera-server, the command end up by aborted and nothing's working.
voxl2:/$ voxl-camera-server detected system image 1.7.6 using new imx214 defaults ================================================================= configuration for 3 cameras: cam #0 name: tof sensor type: pmd-tof isEnabled: 1 camId: 0 camId2: -1 fps: 5 en_rotate: 0 en_rotate2: 0 en_preview: 1 pre_width: 224 pre_height: 1557 en_raw_preview: 1 en_small_video: 0 small_video_width: -1 small_video_height: -1 en_large_video: 0 large_video_width: -1 large_video_height: -1 en_snapshot: 0 snap_width: -1 snap_height: -1 ae_mode: off standby_enabled: 0 decimator: 5 independent_exposure:0 cam #1 name: hires sensor type: imx214 isEnabled: 0 camId: 1 camId2: -1 fps: 30 en_rotate: 0 en_rotate2: 0 en_preview: 0 pre_width: 640 pre_height: 480 en_raw_preview: 0 en_small_video: 1 small_video_width: 1024 small_video_height: 768 en_large_video: 1 large_video_width: 4208 large_video_height: 3120 en_snapshot: 1 snap_width: 4208 snap_height: 3120 ae_mode: isp standby_enabled: 0 decimator: 1 independent_exposure:0 cam #2 name: tracking sensor type: ov7251 isEnabled: 0 camId: 2 camId2: -1 fps: 30 en_rotate: 0 en_rotate2: 0 en_preview: 1 pre_width: 640 pre_height: 480 en_raw_preview: 1 en_small_video: 0 small_video_width: -1 small_video_height: -1 en_large_video: 0 large_video_width: -1 large_video_height: -1 en_snapshot: 0 snap_width: -1 snap_height: -1 ae_mode: lme_msv standby_enabled: 0 decimator: 1 independent_exposure:0 fsync_en: 0 fsync_gpio: 111 ================================================================= connected to GPS mavlink pipe Starting Camera: tof (id #0) Connected to cpu-monitor Using gain limits min: 54 max: 8000 gbm_create_device(156): Info: backend name is: msm_drm Skipping Camera: hires, configuration marked disabled Skipping Camera: tracking, configuration marked disabled ------ voxl-camera-server: Started 1 of 3 cameras ------ voxl-camera-server: Camera server is now running Aborted -
@Judoor-0 , since the issue is the same when using voxl2 J7, it seems there may be a hw issue either in the m0135 adapter or the tof module. If you are able, please try the second m0135 adapter (disconnect from the hires and tracking cameras and connect to tof), then plug back into j7. If the symptoms repeat, the TOF sensor is most likely bad, so we can send a replacement one to you.
Alex
-
@Alex-Kushleyev I tried with both and it didn't work either.
Please, if you can send a replacement one as soon as possible it would be great because my schedule to deliver the finish product is very tight.
Julien -
@Alex-Kushleyev Also why does it says that the resolution for the camera should be 224x1557 and not 224x171 like on the datasheet of the sensor?
Julien -
@Judoor-0 , the tof sensor takes multiple exposures and returns stacked IR image (kind of like HDR). That image is processed by the TOF library on the cpu to output the final image at the proper resolution.
I will check about RMA, you probably need to fill out the form: https://www.modalai.com/pages/rma
-
@Alex-Kushleyev Ok, I filled up the RMA form in cas it's needed.
My S/N is M23000001E9.
Julien -
@Judoor-0 I'm facing a similar problem with ToF configuration on a Starling setup. Through RMA was anything wrt. to hardware found? If not how did RMA help you?
-
@Jetson-Nano , if you run the camera server in foreground (and with
-d 0option to enable debugging), it could help localize the issue. Also try disabling cameras one by one invoxl-camera-server.confto find which camera is causing an issue.Is the issue you are seeing specific to TOF? Was it previously working?
Alex
-
@Alex-Kushleyev I tried that and have posted the output here (https://forum.modalai.com/topic/3688/vision-sensors-not-detected-in-voxl2). Initially the issue was with all the cameras, now it is only ToF.
