CAM_MISSING_STALLED

Alex Kushleyev

@Jetson-Nano, Normally, the camera connectors would not wiggle out of place (while staying inserted) if they are inserted correctly and they are not damaged. That is to say, if the connector stays physically inserted, vibration should not cause it to have bad contact.

It is possible that the flex cable is damaged or has a fine crack in the copper traces. You can test it out by gently moving the flex cable while the camera is streaming (but do not pull hard or do not try to disconnect the connector by pulling on the flex).

Alex

Jetson Nano

@Alex-Kushleyev But, the camera server is stopping. When the there is no vibrations, the drone is on ground and not even armed I am getting this error. When I run camera server it shows either killed or aborted. If there is damage in the flex cable it should not work even in the ground when it is stationary. It is stopping exactly after 5 sec of starting the server.

If I configure the camera server only for tracking it is working fine, when I configure along with the ToF it is showing the error.

Jetson Nano

@Alex-Kushleyev Here is the output for voxl-camera-server -d 0 only with ToF

voxl-camera-server -d 0
=================================================================
configuration for 1 cameras:

cam #0
    name:                tof
    sensor type:         pmd-tof
    isEnabled:           1
    camId:               0
    camId2:              -1
    fps:                 5
    en_rotate:           0
    en_rotate2:          0

    en_preview:          1
    pre_width:           224
    pre_height:          1557
    en_raw_preview:      1

    en_small_video:      0
    small_video_width:   -1
    small_video_height:  -1

    en_large_video:      0
    large_video_width:   -1
    large_video_height:  -1

    en_snapshot:         0
    snap_width:          -1
    snap_height:         -1

    ae_mode:             off
    standby_enabled:     0
    decimator:           5
    independent_exposure:0

fsync_en:            1
fsync_gpio:          111
=================================================================
DEBUG:   Attempting to open the hal module
DEBUG:   SUCCESS: Camera module opened on attempt 0
DEBUG:   ----------- Number of cameras: 1

DEBUG:   Cam idx: 0, Cam slot: 0, Slave Address: 0x007A, Sensor Id: 0x003D
DEBUG:   ------ voxl-camera-server: Starting 1 cameras
Starting Camera: tof (id #0)
Connected to cpu-monitor
DEBUG:   Checking Gain limits for Camera: tof
Using gain limits min: 54 max: 8000
DEBUG:   cam ID 0 checking for fmt:   38  w:  224 h: 1557 o:   0
DEBUG:   	==> option   0: format= 34 width= 176 height= 144 is_input=0
DEBUG:   	==> option   4: format= 34 width= 176 height= 144 is_input=1
DEBUG:   	==> option   8: format= 35 width= 176 height= 144 is_input=0
DEBUG:   	==> option  12: format= 35 width= 176 height= 144 is_input=1
DEBUG:   	==> option  16: format= 33 width= 176 height= 144 is_input=0
DEBUG:   	==> option  20: format= 37 width= 224 height=1557 is_input=0
DEBUG:   	==> option  24: format= 38 width= 224 height=1557 is_input=0
DEBUG:   	==> option  28: format= 32 width= 224 height=1557 is_input=0
DEBUG:   	==> option  32: format= 36 width= 224 height=1557 is_input=0
VERBOSE: Successfully found configuration match for camera 0: 224x1557
VERBOSE: Adding preview stream for camera: 0
DEBUG:   Converted gralloc flags 0x20000 to GBM flags 0x400000
VERBOSE: Dumping GBM flags
DEBUG:   	Found flag GBM_BO_USAGE_CAMERA_WRITE_QTI	
VERBOSE: Opened GBM fd
gbm_create_device(156): Info: backend name is: msm_drm
VERBOSE: Created GBM device
DEBUG:   Allocated BO with width=224 height=1557 stride=336 aligned_w=336 aligned_h=1557 size=524288 flags=0x20000 format=GBM_FORMAT_RAW12
DEBUG:   Converted gralloc flags 0x20000 to GBM flags 0x400000
VERBOSE: Dumping GBM flags
DEBUG:   	Found flag GBM_BO_USAGE_CAMERA_WRITE_QTI	
DEBUG:   Allocated BO with width=224 height=1557 stride=336 aligned_w=336 aligned_h=1557 size=524288 flags=0x20000 format=GBM_FORMAT_RAW12
DEBUG:   Converted gralloc flags 0x20000 to GBM flags 0x400000
VERBOSE: Dumping GBM flags
DEBUG:   	Found flag GBM_BO_USAGE_CAMERA_WRITE_QTI	
DEBUG:   Allocated BO with width=224 height=1557 stride=336 aligned_w=336 aligned_h=1557 size=524288 flags=0x20000 format=GBM_FORMAT_RAW12
DEBUG:   Converted gralloc flags 0x20000 to GBM flags 0x400000
VERBOSE: Dumping GBM flags
DEBUG:   	Found flag GBM_BO_USAGE_CAMERA_WRITE_QTI	
DEBUG:   Allocated BO with width=224 height=1557 stride=336 aligned_w=336 aligned_h=1557 size=524288 flags=0x20000 format=GBM_FORMAT_RAW12
DEBUG:   Converted gralloc flags 0x20000 to GBM flags 0x400000
VERBOSE: Dumping GBM flags
DEBUG:   	Found flag GBM_BO_USAGE_CAMERA_WRITE_QTI	
DEBUG:   Allocated BO with width=224 height=1557 stride=336 aligned_w=336 aligned_h=1557 size=524288 flags=0x20000 format=GBM_FORMAT_RAW12
DEBUG:   Converted gralloc flags 0x20000 to GBM flags 0x400000
VERBOSE: Dumping GBM flags
DEBUG:   	Found flag GBM_BO_USAGE_CAMERA_WRITE_QTI	
DEBUG:   Allocated BO with width=224 height=1557 stride=336 aligned_w=336 aligned_h=1557 size=524288 flags=0x20000 format=GBM_FORMAT_RAW12
DEBUG:   Converted gralloc flags 0x20000 to GBM flags 0x400000
VERBOSE: Dumping GBM flags
DEBUG:   	Found flag GBM_BO_USAGE_CAMERA_WRITE_QTI	
DEBUG:   Allocated BO with width=224 height=1557 stride=336 aligned_w=336 aligned_h=1557 size=524288 flags=0x20000 format=GBM_FORMAT_RAW12
DEBUG:   Converted gralloc flags 0x20000 to GBM flags 0x400000
VERBOSE: Dumping GBM flags
DEBUG:   	Found flag GBM_BO_USAGE_CAMERA_WRITE_QTI	
DEBUG:   Allocated BO with width=224 height=1557 stride=336 aligned_w=336 aligned_h=1557 size=524288 flags=0x20000 format=GBM_FORMAT_RAW12
DEBUG:   Converted gralloc flags 0x20000 to GBM flags 0x400000
VERBOSE: Dumping GBM flags
DEBUG:   	Found flag GBM_BO_USAGE_CAMERA_WRITE_QTI	
DEBUG:   Allocated BO with width=224 height=1557 stride=336 aligned_w=336 aligned_h=1557 size=524288 flags=0x20000 format=GBM_FORMAT_RAW12
DEBUG:   Converted gralloc flags 0x20000 to GBM flags 0x400000
VERBOSE: Dumping GBM flags
DEBUG:   	Found flag GBM_BO_USAGE_CAMERA_WRITE_QTI	
DEBUG:   Allocated BO with width=224 height=1557 stride=336 aligned_w=336 aligned_h=1557 size=524288 flags=0x20000 format=GBM_FORMAT_RAW12
DEBUG:   Converted gralloc flags 0x20000 to GBM flags 0x400000
VERBOSE: Dumping GBM flags
DEBUG:   	Found flag GBM_BO_USAGE_CAMERA_WRITE_QTI	
DEBUG:   Allocated BO with width=224 height=1557 stride=336 aligned_w=336 aligned_h=1557 size=524288 flags=0x20000 format=GBM_FORMAT_RAW12
DEBUG:   Converted gralloc flags 0x20000 to GBM flags 0x400000
VERBOSE: Dumping GBM flags
DEBUG:   	Found flag GBM_BO_USAGE_CAMERA_WRITE_QTI	
DEBUG:   Allocated BO with width=224 height=1557 stride=336 aligned_w=336 aligned_h=1557 size=524288 flags=0x20000 format=GBM_FORMAT_RAW12
DEBUG:   Converted gralloc flags 0x20000 to GBM flags 0x400000
VERBOSE: Dumping GBM flags
DEBUG:   	Found flag GBM_BO_USAGE_CAMERA_WRITE_QTI	
DEBUG:   Allocated BO with width=224 height=1557 stride=336 aligned_w=336 aligned_h=1557 size=524288 flags=0x20000 format=GBM_FORMAT_RAW12
DEBUG:   Converted gralloc flags 0x20000 to GBM flags 0x400000
VERBOSE: Dumping GBM flags
DEBUG:   	Found flag GBM_BO_USAGE_CAMERA_WRITE_QTI	
DEBUG:   Allocated BO with width=224 height=1557 stride=336 aligned_w=336 aligned_h=1557 size=524288 flags=0x20000 format=GBM_FORMAT_RAW12
DEBUG:   Converted gralloc flags 0x20000 to GBM flags 0x400000
VERBOSE: Dumping GBM flags
DEBUG:   	Found flag GBM_BO_USAGE_CAMERA_WRITE_QTI	
DEBUG:   Allocated BO with width=224 height=1557 stride=336 aligned_w=336 aligned_h=1557 size=524288 flags=0x20000 format=GBM_FORMAT_RAW12
DEBUG:   Converted gralloc flags 0x20000 to GBM flags 0x400000
VERBOSE: Dumping GBM flags
DEBUG:   	Found flag GBM_BO_USAGE_CAMERA_WRITE_QTI	
DEBUG:   Allocated BO with width=224 height=1557 stride=336 aligned_w=336 aligned_h=1557 size=524288 flags=0x20000 format=GBM_FORMAT_RAW12
DEBUG:   Successfully set up pipeline for stream: PREVIEW
VERBOSE: Entered thread: cam0-request(tid: 5577)
VERBOSE: Entered thread: cam0-result(tid: 5578)
DEBUG:   Started Camera: tof

------ voxl-camera-server: Started 1 of 1 cameras

------ voxl-camera-server: Camera server is now running
VERBOSE: Found Royale module config: imagerType 2, illuminationConfig.dutyCycle: 2 temp_sensor_type: 1
VERBOSE: Found Royale module config: maxImgW 224 maxImgH 172 frameTxMode 1 camName CUSTOM
VERBOSE: Found Royale module config: tempLimitSoft 60.000000 tempLimitHard 65.000000 autoExpoSupported yes
VERBOSE: useCaseList size is 11
VERBOSE: Usecase name MODE9_5
VERBOSE: Found Royale usecase: MODE9_5 - phases: 9 fps: 5
VERBOSE: Found Royale usecase: MODE9_5 - exposure group[0] = gray
VERBOSE: Found Royale usecase: MODE9_5 - exposure group[1] = mod1
VERBOSE: Found Royale usecase: MODE9_5 - exposure group[2] = mod2
VERBOSE: Found Royale usecase: MODE9_5 - exp_limit[0] = (100 100)
VERBOSE: Found Royale usecase: MODE9_5 - exp_limit[1] = (1 1200)
VERBOSE: Found Royale usecase: MODE9_5 - exp_limit[2] = (1 1200)
VERBOSE: Found Royale usecase: MODE9_5 - exp_time[0] = 100
VERBOSE: Found Royale usecase: MODE9_5 - exp_time[1] = 1200
VERBOSE: Found Royale usecase: MODE9_5 - exp_time[2] = 1200
VERBOSE: Usecase name MODE9_1
VERBOSE: Found Royale usecase: MODE9_1 - phases: 9 fps: 1
VERBOSE: Found Royale usecase: MODE9_1 - exposure group[0] = gray
VERBOSE: Found Royale usecase: MODE9_1 - exposure group[1] = mod1
VERBOSE: Found Royale usecase: MODE9_1 - exposure group[2] = mod2
VERBOSE: Found Royale usecase: MODE9_1 - exp_limit[0] = (100 100)
VERBOSE: Found Royale usecase: MODE9_1 - exp_limit[1] = (1 1200)
VERBOSE: Found Royale usecase: MODE9_1 - exp_limit[2] = (1 1200)
VERBOSE: Found Royale usecase: MODE9_1 - exp_time[0] = 100
VERBOSE: Found Royale usecase: MODE9_1 - exp_time[1] = 1200
VERBOSE: Found Royale usecase: MODE9_1 - exp_time[2] = 1200
VERBOSE: Usecase name MODE9_2
VERBOSE: Found Royale usecase: MODE9_2 - phases: 9 fps: 2
VERBOSE: Found Royale usecase: MODE9_2 - exposure group[0] = gray
VERBOSE: Found Royale usecase: MODE9_2 - exposure group[1] = mod1
VERBOSE: Found Royale usecase: MODE9_2 - exposure group[2] = mod2
VERBOSE: Found Royale usecase: MODE9_2 - exp_limit[0] = (100 100)
VERBOSE: Found Royale usecase: MODE9_2 - exp_limit[1] = (1 1200)
VERBOSE: Found Royale usecase: MODE9_2 - exp_limit[2] = (1 1200)
VERBOSE: Found Royale usecase: MODE9_2 - exp_time[0] = 100
VERBOSE: Found Royale usecase: MODE9_2 - exp_time[1] = 1200
VERBOSE: Found Royale usecase: MODE9_2 - exp_time[2] = 1200
VERBOSE: Usecase name MODE9_3
VERBOSE: Found Royale usecase: MODE9_3 - phases: 9 fps: 3
VERBOSE: Found Royale usecase: MODE9_3 - exposure group[0] = gray
VERBOSE: Found Royale usecase: MODE9_3 - exposure group[1] = mod1
VERBOSE: Found Royale usecase: MODE9_3 - exposure group[2] = mod2
VERBOSE: Found Royale usecase: MODE9_3 - exp_limit[0] = (100 100)
VERBOSE: Found Royale usecase: MODE9_3 - exp_limit[1] = (1 1200)
VERBOSE: Found Royale usecase: MODE9_3 - exp_limit[2] = (1 1200)
VERBOSE: Found Royale usecase: MODE9_3 - exp_time[0] = 100
VERBOSE: Found Royale usecase: MODE9_3 - exp_time[1] = 1200
VERBOSE: Found Royale usecase: MODE9_3 - exp_time[2] = 1200
VERBOSE: Usecase name MODE9_10
VERBOSE: Found Royale usecase: MODE9_10 - phases: 9 fps: 10
VERBOSE: Found Royale usecase: MODE9_10 - exposure group[0] = gray
VERBOSE: Found Royale usecase: MODE9_10 - exposure group[1] = mod1
VERBOSE: Found Royale usecase: MODE9_10 - exposure group[2] = mod2
VERBOSE: Found Royale usecase: MODE9_10 - exp_limit[0] = (100 100)
VERBOSE: Found Royale usecase: MODE9_10 - exp_limit[1] = (1 650)
VERBOSE: Found Royale usecase: MODE9_10 - exp_limit[2] = (1 650)
VERBOSE: Found Royale usecase: MODE9_10 - exp_time[0] = 100
VERBOSE: Found Royale usecase: MODE9_10 - exp_time[1] = 650
VERBOSE: Found Royale usecase: MODE9_10 - exp_time[2] = 650
VERBOSE: Usecase name MODE9_15
VERBOSE: Found Royale usecase: MODE9_15 - phases: 9 fps: 15
VERBOSE: Found Royale usecase: MODE9_15 - exposure group[0] = gray
VERBOSE: Found Royale usecase: MODE9_15 - exposure group[1] = mod1
VERBOSE: Found Royale usecase: MODE9_15 - exposure group[2] = mod2
VERBOSE: Found Royale usecase: MODE9_15 - exp_limit[0] = (100 100)
VERBOSE: Found Royale usecase: MODE9_15 - exp_limit[1] = (1 450)
VERBOSE: Found Royale usecase: MODE9_15 - exp_limit[2] = (1 450)
VERBOSE: Found Royale usecase: MODE9_15 - exp_time[0] = 100
VERBOSE: Found Royale usecase: MODE9_15 - exp_time[1] = 450
VERBOSE: Found Royale usecase: MODE9_15 - exp_time[2] = 450
VERBOSE: Usecase name MODE5_5
VERBOSE: Found Royale usecase: MODE5_5 - phases: 5 fps: 5
VERBOSE: Found Royale usecase: MODE5_5 - exposure group[0] = gray
VERBOSE: Found Royale usecase: MODE5_5 - exposure group[1] = mod
VERBOSE: Found Royale usecase: MODE5_5 - exp_limit[0] = (100 100)
VERBOSE: Found Royale usecase: MODE5_5 - exp_limit[1] = (1 250)
VERBOSE: Found Royale usecase: MODE5_5 - exp_time[0] = 100
VERBOSE: Found Royale usecase: MODE5_5 - exp_time[1] = 250
VERBOSE: Usecase name MODE5_10
VERBOSE: Found Royale usecase: MODE5_10 - phases: 5 fps: 10
VERBOSE: Found Royale usecase: MODE5_10 - exposure group[0] = gray
VERBOSE: Found Royale usecase: MODE5_10 - exposure group[1] = mod
VERBOSE: Found Royale usecase: MODE5_10 - exp_limit[0] = (100 100)
VERBOSE: Found Royale usecase: MODE5_10 - exp_limit[1] = (1 250)
VERBOSE: Found Royale usecase: MODE5_10 - exp_time[0] = 100
VERBOSE: Found Royale usecase: MODE5_10 - exp_time[1] = 250
VERBOSE: Usecase name MODE5_15
VERBOSE: Found Royale usecase: MODE5_15 - phases: 5 fps: 15
VERBOSE: Found Royale usecase: MODE5_15 - exposure group[0] = gray
VERBOSE: Found Royale usecase: MODE5_15 - exposure group[1] = mod
VERBOSE: Found Royale usecase: MODE5_15 - exp_limit[0] = (100 100)
VERBOSE: Found Royale usecase: MODE5_15 - exp_limit[1] = (1 250)
VERBOSE: Found Royale usecase: MODE5_15 - exp_time[0] = 100
VERBOSE: Found Royale usecase: MODE5_15 - exp_time[1] = 250
VERBOSE: Usecase name MODE5_30
VERBOSE: Found Royale usecase: MODE5_30 - phases: 5 fps: 30
VERBOSE: Found Royale usecase: MODE5_30 - exposure group[0] = gray
VERBOSE: Found Royale usecase: MODE5_30 - exposure group[1] = mod
VERBOSE: Found Royale usecase: MODE5_30 - exp_limit[0] = (100 100)
VERBOSE: Found Royale usecase: MODE5_30 - exp_limit[1] = (1 250)
VERBOSE: Found Royale usecase: MODE5_30 - exp_time[0] = 100
VERBOSE: Found Royale usecase: MODE5_30 - exp_time[1] = 250
VERBOSE: Usecase name MODE5_45
VERBOSE: Found Royale usecase: MODE5_45 - phases: 5 fps: 45
VERBOSE: Found Royale usecase: MODE5_45 - exposure group[0] = gray
VERBOSE: Found Royale usecase: MODE5_45 - exposure group[1] = mod
VERBOSE: Found Royale usecase: MODE5_45 - exp_limit[0] = (100 100)
VERBOSE: Found Royale usecase: MODE5_45 - exp_limit[1] = (1 250)
VERBOSE: Found Royale usecase: MODE5_45 - exp_time[0] = 100
VERBOSE: Found Royale usecase: MODE5_45 - exp_time[1] = 250
DEBUG:   Configuring TOF module for A65
VERBOSE: sid: 0x00AC, addr: 0x0000, data: 0x0050
VERBOSE: sid: 0x00AC, addr: 0x0001, data: 0x004D
VERBOSE: sid: 0x00AC, addr: 0x0002, data: 0x0044
VERBOSE: sid: 0x00AC, addr: 0x0003, data: 0x0054
VERBOSE: sid: 0x00AC, addr: 0x0004, data: 0x0045
VERBOSE: sid: 0x00AC, addr: 0x0005, data: 0x0043
VERBOSE: sid: 0x00AC, addr: 0x0006, data: 0x0007
VERBOSE: sid: 0x00AC, addr: 0x0007, data: 0x0000
VERBOSE: sid: 0x00AC, addr: 0x0008, data: 0x004B
VERBOSE: sid: 0x00AC, addr: 0x0009, data: 0x0086
VERBOSE: sid: 0x00AC, addr: 0x000A, data: 0x0051
VERBOSE: sid: 0x00AC, addr: 0x000B, data: 0x0014
VERBOSE: sid: 0x00AC, addr: 0x000C, data: 0x002A
VERBOSE: sid: 0x00AC, addr: 0x000D, data: 0x00EB
VERBOSE: sid: 0x00AC, addr: 0x000E, data: 0x0001
VERBOSE: sid: 0x00AC, addr: 0x000F, data: 0x0000
VERBOSE: sid: 0x00AC, addr: 0x0010, data: 0x0003
VERBOSE: sid: 0x00AC, addr: 0x0011, data: 0x0000
VERBOSE: sid: 0x00AC, addr: 0x0012, data: 0x0000
VERBOSE: sid: 0x00AC, addr: 0x0013, data: 0x0000
VERBOSE: sid: 0x00AC, addr: 0x0014, data: 0x0000
VERBOSE: sid: 0x00AC, addr: 0x0015, data: 0x0000
VERBOSE: sid: 0x00AC, addr: 0x0016, data: 0x0000
VERBOSE: sid: 0x00AC, addr: 0x0017, data: 0x0000
VERBOSE: sid: 0x00AC, addr: 0x0018, data: 0x0000
VERBOSE: sid: 0x00AC, addr: 0x0019, data: 0x0000
VERBOSE: sid: 0x00AC, addr: 0x001A, data: 0x0000
VERBOSE: sid: 0x00AC, addr: 0x001B, data: 0x0000
VERBOSE: sid: 0x00AC, addr: 0x001C, data: 0x0000
VERBOSE: sid: 0x00AC, addr: 0x001D, data: 0x0000
VERBOSE: sid: 0x00AC, addr: 0x001E, data: 0x0000
VERBOSE: sid: 0x00AC, addr: 0x001F, data: 0x0071
VERBOSE: sid: 0x00AC, addr: 0x0020, data: 0x0050
VERBOSE: sid: 0x00AC, addr: 0x0021, data: 0x004D
VERBOSE: sid: 0x00AC, addr: 0x0022, data: 0x0044
VERBOSE: sid: 0x00AC, addr: 0x0023, data: 0x0054
VERBOSE: sid: 0x00AC, addr: 0x0024, data: 0x004F
VERBOSE: sid: 0x00AC, addr: 0x0025, data: 0x0046
VERBOSE: sid: 0x00AC, addr: 0x0026, data: 0x005F
VERBOSE: sid: 0x00AC, addr: 0x0027, data: 0x005F
VERBOSE: sid: 0x00AC, addr: 0x0028, data: 0x005F
VERBOSE: sid: 0x00AC, addr: 0x0029, data: 0x005F
VERBOSE: sid: 0x00AC, addr: 0x002A, data: 0x005F
VERBOSE: sid: 0x00AC, addr: 0x002B, data: 0x005F
VERBOSE: sid: 0x00AC, addr: 0x002C, data: 0x005F
VERBOSE: sid: 0x00AC, addr: 0x002D, data: 0x005F
VERBOSE: sid: 0x00AC, addr: 0x002E, data: 0x005F
VERBOSE: sid: 0x00AC, addr: 0x002F, data: 0x005F
VERBOSE: sid: 0x00AC, addr: 0x0030, data: 0x0041
VERBOSE: sid: 0x00AC, addr: 0x0031, data: 0x0043
VERBOSE: sid: 0x00AC, addr: 0x0032, data: 0x0033
VERBOSE: sid: 0x00AC, addr: 0x0033, data: 0x0030
VERBOSE: sid: 0x00AC, addr: 0x0034, data: 0x002D
VERBOSE: sid: 0x00AC, addr: 0x0035, data: 0x0030
VERBOSE: sid: 0x00AC, addr: 0x0036, data: 0x0039
VERBOSE: sid: 0x00AC, addr: 0x0037, data: 0x0044
VERBOSE: sid: 0x00AC, addr: 0x0038, data: 0x0042
VERBOSE: sid: 0x00AC, addr: 0x0039, data: 0x002D
VERBOSE: sid: 0x00AC, addr: 0x003A, data: 0x0041
VERBOSE: sid: 0x00AC, addr: 0x003B, data: 0x0030
VERBOSE: sid: 0x00AC, addr: 0x003C, data: 0x0032
VERBOSE: sid: 0x00AC, addr: 0x003D, data: 0x0034
VERBOSE: sid: 0x00AC, addr: 0x003E, data: 0x002D
VERBOSE: sid: 0x00AC, addr: 0x003F, data: 0x0031
VERBOSE: sid: 0x00AC, addr: 0x0040, data: 0x0041
VERBOSE: sid: 0x00AC, addr: 0x0041, data: 0x0033
VERBOSE: sid: 0x00AC, addr: 0x0042, data: 0x0030
VERBOSE: sid: 0x007A, addr: 0xA0A5, data: 0x0B12
VERBOSE: sid: 0x007A, addr: 0xA007, data: 0x0001
VERBOSE: sid: 0x007A, addr: 0xA00A, data: 0x0007
VERBOSE: sid: 0x007A, addr: 0xA014, data: 0x0007
VERBOSE: sid: 0x007A, addr: 0xA01C, data: 0x131A
VERBOSE: sid: 0x007A, addr: 0xA063, data: 0x01C5
VERBOSE: sid: 0x007A, addr: 0xA065, data: 0x0206
VERBOSE: sid: 0x007A, addr: 0xA066, data: 0x01C5
VERBOSE: sid: 0x007A, addr: 0xA068, data: 0x0206
VERBOSE: sid: 0x007A, addr: 0xA007, data: 0x0001
VERBOSE: sid: 0x007A, addr: 0xA00A, data: 0x0007
VERBOSE: sid: 0x007A, addr: 0xA014, data: 0x0007
VERBOSE: sid: 0x007A, addr: 0xA01C, data: 0x131A
VERBOSE: sid: 0x007A, addr: 0xA063, data: 0x01C5
VERBOSE: sid: 0x007A, addr: 0xA065, data: 0x0206
VERBOSE: sid: 0x007A, addr: 0xA066, data: 0x01C5
VERBOSE: sid: 0x007A, addr: 0xA068, data: 0x0206
VERBOSE: sid: 0x007A, addr: 0xA084, data: 0x1513
VERBOSE: sid: 0x007A, addr: 0x9838, data: 0x0000
VERBOSE: sid: 0x007A, addr: 0x9835, data: 0x0192
VERBOSE: sid: 0x007A, addr: 0xB402, data: 0x0080
VERBOSE: sid: 0x007A, addr: 0xA800, data: 0x00E3
VERBOSE: sid: 0x007A, addr: 0xA801, data: 0x06D6
VERBOSE: sid: 0x007A, addr: 0xA802, data: 0x00E0
VERBOSE: sid: 0x007A, addr: 0xA803, data: 0x0604
VERBOSE: sid: 0x007A, addr: 0xA804, data: 0x00E0
VERBOSE: sid: 0x007A, addr: 0xA805, data: 0x0F0D
VERBOSE: sid: 0x007A, addr: 0xA806, data: 0x0A16
VERBOSE: sid: 0x007A, addr: 0xA807, data: 0x0890
VERBOSE: sid: 0x007A, addr: 0xA808, data: 0x0442
VERBOSE: sid: 0x007A, addr: 0xA809, data: 0x0A02
VERBOSE: sid: 0x007A, addr: 0xA80A, data: 0x00B0
VERBOSE: sid: 0x007A, addr: 0xA80B, data: 0x0C40
VERBOSE: sid: 0x007A, addr: 0xA80C, data: 0x0040
VERBOSE: sid: 0x007A, addr: 0xA80D, data: 0x0200
VERBOSE: sid: 0x007A, addr: 0xA80E, data: 0x088E
VERBOSE: sid: 0x007A, addr: 0xA80F, data: 0x0472
VERBOSE: sid: 0x007A, addr: 0xA810, data: 0x0888
VERBOSE: sid: 0x007A, addr: 0xA811, data: 0x0459
VERBOSE: sid: 0x007A, addr: 0xA812, data: 0x0A0E
VERBOSE: sid: 0x007A, addr: 0xA813, data: 0x0B06
VERBOSE: sid: 0x007A, addr: 0xA814, data: 0x0890
VERBOSE: sid: 0x007A, addr: 0xA815, data: 0x0450
VERBOSE: sid: 0x007A, addr: 0xA816, data: 0x0888
VERBOSE: sid: 0x007A, addr: 0xA817, data: 0x0D59
VERBOSE: sid: 0x007A, addr: 0xA818, data: 0x0890
VERBOSE: sid: 0x007A, addr: 0xA819, data: 0x0C51
VERBOSE: sid: 0x007A, addr: 0xA81A, data: 0x0230
VERBOSE: sid: 0x007A, addr: 0xA81B, data: 0x0398
VERBOSE: sid: 0x007A, addr: 0xAD26, data: 0x053B
VERBOSE: sid: 0x007A, addr: 0xAD27, data: 0x003D
VERBOSE: sid: 0x007A, addr: 0xAD28, data: 0x001F
VERBOSE: sid: 0x007A, addr: 0xAD29, data: 0x0848
VERBOSE: sid: 0x007A, addr: 0xAD2A, data: 0x0811
VERBOSE: sid: 0x007A, addr: 0xAD2B, data: 0x0DBB
VERBOSE: sid: 0x007A, addr: 0xB3A5, data: 0x5712
VERBOSE: sid: 0x007A, addr: 0xB3A6, data: 0x0002
DEBUG:   [ROYALE_INFO] {ROYALE_PROCESSING}: Spectre backend : ng_arm
DEBUG:   setup accessLevel 3, ret 0
DEBUG:   Royale getUseCases returned No error
DEBUG:   List of usecases:
DEBUG:   	Usecase 0: MODE9_5
DEBUG:   	Usecase 1: MODE9_1
DEBUG:   	Usecase 2: MODE9_2
DEBUG:   	Usecase 3: MODE9_3
DEBUG:   	Usecase 4: MODE9_10
DEBUG:   	Usecase 5: MODE9_15
DEBUG:   	Usecase 6: MODE5_5
DEBUG:   	Usecase 7: MODE5_10
DEBUG:   	Usecase 8: MODE5_15
DEBUG:   	Usecase 9: MODE5_30
DEBUG:   	Usecase 10: MODE5_45
DEBUG:   Attempting to set use case 'MODE9_5'
DEBUG:   Changed to usecase MODE9_5 exposure time limits [ 1 - 1200 ], cur exposure_time 1200, exposure mode 1
DEBUG:   Successfully set Royale exposure mode to AUTOMATICVERBOSE: 
Setting use case: mode=9, fps = 5
DEBUG:   Royale getUseCases returned No error
DEBUG:   List of usecases:
DEBUG:   	Usecase 0: MODE9_5
DEBUG:   	Usecase 1: MODE9_1
DEBUG:   	Usecase 2: MODE9_2
DEBUG:   	Usecase 3: MODE9_3
DEBUG:   	Usecase 4: MODE9_10
DEBUG:   	Usecase 5: MODE9_15
DEBUG:   	Usecase 6: MODE5_5
DEBUG:   	Usecase 7: MODE5_10
DEBUG:   	Usecase 8: MODE5_15
DEBUG:   	Usecase 9: MODE5_30
DEBUG:   	Usecase 10: MODE5_45
DEBUG:   Attempting to set use case 'MODE9_5'
DEBUG:   Changed to usecase MODE9_5 exposure time limits [ 1 - 1200 ], cur exposure_time 1200, exposure mode 1
VERBOSE: startCapture E
VERBOSE: sid: 0x007A, addr: 0x9005, data: 0x0002
VERBOSE: sid: 0x007A, addr: 0xB420, data: 0x1000
VERBOSE: sid: 0x007A, addr: 0xB420, data: 0x1040
VERBOSE: sid: 0x007A, addr: 0xB400, data: 0x0001
VERBOSE: sid: 0x007A, addr: 0x9000, data: 0x0005
VERBOSE: sid: 0x007A, addr: 0x9005, data: 0x0000
VERBOSE: sid: 0x007A, addr: 0x982F, data: 0x0000
VERBOSE: sid: 0x007A, addr: 0x9830, data: 0x0000
VERBOSE: sid: 0x007A, addr: 0x982F, data: 0x0001
VERBOSE: startCapture X
VERBOSE: TOF interface created!
VERBOSE: setting hal3 exposure for tof     2259763   200
VERBOSE: added request for preview stream
VERBOSE: Sending request for frame 0 for camera tof for 1 streams
VERBOSE: Received result from HAl3 for frame number 0
VERBOSE: Received 0 buffers from camera tof, partial result:1
VERBOSE: finished sending request for frame 0 for camera tof
VERBOSE: returning from SendOneCaptureRequest for frame 1 for camera tof
VERBOSE: setting hal3 exposure for tof     2259763   200
VERBOSE: added request for preview stream
VERBOSE: Sending request for frame 1 for camera tof for 1 streams
VERBOSE: finished sending request for frame 1 for camera tof
VERBOSE: returning from SendOneCaptureRequest for frame 2 for camera tof
VERBOSE: setting hal3 exposure for tof     2259763   200
VERBOSE: added request for preview stream
VERBOSE: Sending request for frame 2 for camera tof for 1 streams
VERBOSE: finished sending request for frame 2 for camera tof
VERBOSE: returning from SendOneCaptureRequest for frame 3 for camera tof
VERBOSE: setting hal3 exposure for tof     2259763   200
VERBOSE: added request for preview stream
VERBOSE: Sending request for frame 3 for camera tof for 1 streams
VERBOSE: finished sending request for frame 3 for camera tof
VERBOSE: returning from SendOneCaptureRequest for frame 4 for camera tof
VERBOSE: setting hal3 exposure for tof     2259763   200
VERBOSE: added request for preview stream
VERBOSE: Sending request for frame 4 for camera tof for 1 streams
VERBOSE: finished sending request for frame 4 for camera tof
VERBOSE: returning from SendOneCaptureRequest for frame 5 for camera tof
VERBOSE: setting hal3 exposure for tof     2259763   200
VERBOSE: added request for preview stream
VERBOSE: Sending request for frame 5 for camera tof for 1 streams
VERBOSE: Received result from HAl3 for frame number 1
VERBOSE: Received 0 buffers from camera tof, partial result:1
VERBOSE: Received result from HAl3 for frame number 2
VERBOSE: Received 0 buffers from camera tof, partial result:1
VERBOSE: Received result from HAl3 for frame number 3
VERBOSE: Received 0 buffers from camera tof, partial result:1
VERBOSE: Received result from HAl3 for frame number 0
VERBOSE: Received 0 buffers from camera tof, partial result:2
VERBOSE: Received metadata for frame 0 from camera tof
VERBOSE: 	Timestamp: 786183245685
VERBOSE: 	Gain: 0
VERBOSE: 	Exposure: 0
VERBOSE: Received result from HAl3 for frame number 4
VERBOSE: Received 0 buffers from camera tof, partial result:1
DEBUG:   Value of standby_active is: 0 
DEBUG:   Value of standby_active is: 0 
DEBUG:   Value of standby_active is: 0 
DEBUG:   Value of standby_active is: 0 
DEBUG:   Value of standby_active is: 0 
Aborted

Alex Kushleyev

@Jetson-Nano , Yes it seems like TOF sensor is not working properly. Can you please repeat the test with only TOF and enable some debugging:

echo 0xf > /sys/kernel/debug/camera_ife/ife_csid_debug

Then start dmesg -w in a separate terminal and restart camera server. Under normal conditions, you should be seeing messages like below (SOF stands for Start of Frame and EOF stands for End of Frame). This would also show up any errors in transmission between TOF camera and VOXL2.

CAM_INFO: CAM-ISP: cam_ife_csid_irq: 5210 CSID RDI:0 SOF received
CAM_INFO: CAM-ISP: cam_ife_csid_irq: 5219 CSID RDI:0 EOF received

Jetson Nano

@Alex-Kushleyev thanks for reverting, meanwhile running this test, should I have physically other cameras(ie. ov7251 tracking) disconnected from the system or just have them disabled in the camera config file?

Alex Kushleyev

@Jetson-Nano , just disabled in the config file is fine, but you can try to see if physically disconnecting them would make a difference. May be a good test to do that.

Jetson Nano

@Alex-Kushleyev I did the test, with ToF enabled in config with others disabled and physically also removed the other cameras(tracking, hires) and got the following output,

[  500.965614] qcom_rpmh DRV:apps_rsc TCS Busy, retrying RPMH message send: addr=0x40d00
[  513.797938] qcom_rpmh DRV:apps_rsc TCS Busy, retrying RPMH message send: addr=0x40d00
[  563.809259] CAM_ERR: CAM-MEM: cam_mem_mgr_create_debug_fs: 126 failed to create dentry
[  563.837666] CAM_INFO: CAM-HFI: cam_hfi_init: 878 Init IO1 : [0x10c00000 0xcf300000] IO2 [0xe0200000 0x1ed00000]
[  563.847825] CAM_INFO: CAM-ICP: cam_icp_mgr_hw_open: 3879 FW download done successfully
[  563.847979] CAM_INFO: CAM-ICP: cam_icp_mgr_process_dbg_buf: 2572 FW_DBG:CICP_FW_E : HFI  :QC_IMAGE_VERSION_STRING=CICP.FW.1.0-00079,OEM_IMAGE_VERSION_STRING=CRM,BUILD_TIME: Oct 17 2019 05:49:19,CACHE_ENABLED at icphostinterface.c:636 QC_IMAGE_VERSION_STRING=CICP.FW.1.0-00079 OEM_IMAGE_VERSION_STRING=CRM
[  563.847983] CAM_INFO: CAM-ICP: cam_icp_mgr_process_dbg_buf: 2572 FW_DBG:CICP_FW_E : HFI  :ELF variant: CACHE-ENABLED:T480:API_V2:USE_CDM_1_1: , API version: 0x2000049 at icphostinterface.c:637 QC_IMAGE_VERSION_STRING=CICP.FW.1.0-00079 OEM_IMAGE_VERSION_STRING=CRM
[  564.015546] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Linked as a consumer to regulator.60
[  564.015606] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Linked as a consumer to regulator.79
[  564.042685] CAM_ERR: CAM-CCI: cam_cci_irq: 238 Base:        pK-error, M0_Q1 NACK ERROR: 0x10000000
[  564.042909] CAM_ERR: CAM-CCI: cam_cci_read: 1329 ERROR with Slave 0x7a:
[  564.042926] CAM_ERR: CAM-CCI: cam_cci_read_bytes: 1613 failed to read rc:-22
[  564.042935] CAM_ERR: CAM-SENSOR: cam_cci_i2c_read: 35 rc = -22
[  564.042945] CAM_WARN: CAM-SENSOR: cam_sensor_match_id: 707 read id: 0x0 expected id 0x3d:
[  564.043137] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Dropping the link to regulator.79
[  564.047327] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Dropping the link to regulator.60
[  564.076125] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Linked as a consumer to regulator.60
[  564.076278] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Linked as a consumer to regulator.79
[  564.093922] CAM_ERR: CAM-CCI: cam_cci_irq: 238 Base:        pK-error, M0_Q1 NACK ERROR: 0x10000000
[  564.093988] CAM_ERR: CAM-CCI: cam_cci_read: 1329 ERROR with Slave 0x20:
[  564.094000] CAM_ERR: CAM-CCI: cam_cci_read_bytes: 1613 failed to read rc:-22
[  564.094008] CAM_ERR: CAM-SENSOR: cam_cci_i2c_read: 35 rc = -22
[  564.094016] CAM_WARN: CAM-SENSOR: cam_sensor_match_id: 707 read id: 0x0 expected id 0x214:
[  564.094138] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Dropping the link to regulator.79
[  564.098341] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Dropping the link to regulator.60
[  564.129852] qcom,camera ac50000.qcom,cci:qcom,cam-sensor3: Linked as a consumer to regulator.55
[  564.132377] qcom,camera ac50000.qcom,cci:qcom,cam-sensor3: Linked as a consumer to regulator.60
[  564.132522] qcom,camera ac50000.qcom,cci:qcom,cam-sensor3: Linked as a consumer to regulator.79
[  564.156581] CAM_INFO: CAM-SENSOR: cam_sensor_set_alt_id: 731 [Applying Alternative Address] : Sensor Index   : 391549108
[  564.156594] CAM_INFO: CAM-SENSOR: cam_sensor_set_alt_id: 732 [Applying Alternative Address] : Slave Address  : 0xE2
[  564.156602] CAM_INFO: CAM-SENSOR: cam_sensor_set_alt_id: 733 [Applying Alternative Address] : Slave sensor id: 0x7750
[  564.156610] CAM_INFO: CAM-SENSOR: cam_sensor_set_alt_id: 734 [Applying Alternative Address] : cci_client sid : 0x71
[  564.156830] CAM_ERR: CAM-CCI: cam_cci_irq: 258 Base:        pK-error, M1_Q0 NACK ERROR: 0x20000000
[  564.156882] CAM_ERR: CAM-CCI: cam_cci_wait: 270 failed rc -22
[  564.156891] CAM_ERR: CAM-CCI: cam_cci_transfer_end: 345 failed rc -22
[  564.156899] CAM_ERR: CAM-CCI: cam_cci_data_queue: 870 Slave: 0xe0 failed rc -22
[  564.156907] CAM_ERR: CAM-CCI: cam_cci_i2c_write: 1450 failed rc: -22
[  564.156916] CAM_ERR: CAM-SENSOR: cam_cci_i2c_write_table_cmd: 123 Failed rc = -22
[  564.156924] CAM_ERR: CAM-SENSOR: cam_sensor_set_alt_id: 768 [Applying Alternative Address] : Probe : camera_io_dev_write failed: rc=-22
[  564.157075] CAM_ERR: CAM-CCI: cam_cci_irq: 264 Base:        pK-error, M1_Q1 NACK ERROR: 0x40000000
[  564.157195] CAM_ERR: CAM-CCI: cam_cci_read: 1329 ERROR with Slave 0xe0:
[  564.157203] CAM_ERR: CAM-CCI: cam_cci_read_bytes: 1613 failed to read rc:-22
[  564.157210] CAM_ERR: CAM-SENSOR: cam_cci_i2c_read: 35 rc = -22
[  564.157219] CAM_WARN: CAM-SENSOR: cam_sensor_match_id: 707 read id: 0x0 expected id 0x7750:
[  564.157384] qcom,camera ac50000.qcom,cci:qcom,cam-sensor3: Dropping the link to regulator.79
[  564.161659] qcom,camera ac50000.qcom,cci:qcom,cam-sensor3: Dropping the link to regulator.60
[  564.161850] qcom,camera ac50000.qcom,cci:qcom,cam-sensor3: Dropping the link to regulator.55
[  564.214462] CAM_WARN: CAM-CRM: cam_req_mgr_close: 160 release invoked associated userspace process has died
[  593.797336] qcom_rpmh DRV:apps_rsc TCS Busy, retrying RPMH message send: addr=0x40d00
[  593.797362] qcom_rpmh DRV:apps_rsc TCS Busy, retrying RPMH message send: addr=0x40d00

Jetson Nano

@Jetson-Nano I tried doing the same above process of checking for Hires IMX214 cam also, found the following using dmesg -w. I configured C-8 configuration upon J7-U port.

[ 1414.886039] cam_csid_evt_bottom_half_handler: 12 callbacks suppressed
[ 1414.886043] CAM_ERR: CAM-ISP: cam_csid_evt_bottom_half_handler: 4621 status RDI0: 0
[ 1414.886046] CAM_ERR: CAM-ISP: cam_csid_evt_bottom_half_handler: 4621 status RDI1: 0
[ 1414.886049] CAM_ERR: CAM-ISP: cam_csid_evt_bottom_half_handler: 4621 status RDI2: 0
[ 1414.886053] CAM_ERR: CAM-ISP: cam_csid_evt_bottom_half_handler: 4621 status RDI3: 0
[ 1414.886056] CAM_ERR: CAM-ISP: cam_csid_evt_bottom_half_handler: 4621 status TOP: 0
[ 1414.886060] CAM_ERR: CAM-ISP: cam_csid_evt_bottom_half_handler: 4621 status RX: 38000
[ 1414.886063] CAM_ERR: CAM-ISP: cam_csid_evt_bottom_half_handler: 4621 status IPP: 0
[ 1414.886066] CAM_ERR: CAM-ISP: cam_csid_evt_bottom_half_handler: 4621 status PPP: 0
[ 1414.886070] CAM_ERR: CAM-ISP: cam_csid_evt_bottom_half_handler: 4621 status UDI0: 0
[ 1414.886073] CAM_ERR: CAM-ISP: cam_csid_evt_bottom_half_handler: 4621 status UDI1: 0
[ 1414.886081] CAM_ERR: CAM-ISP: cam_csid_evt_bottom_half_handler: 4616 idx 1 err 5 phy 3  lane type:0 ln num:4 ln cfg:0x3210 cnt 2
[ 1416.258821] msm_vidc:   err : ffffffff: .....: DBLP Set: status 0
[ 1420.000213] CAM_ERR: CAM-CRM: __cam_req_mgr_process_sof_freeze: 1800 SOF freeze for session 8520192 link 0xb50105
[ 1420.000250] CAM_INFO: CAM-ISP: cam_ife_csid_sof_irq_debug: 4250 SOF freeze: CSID SOF irq enabled
[ 1420.000265] CAM_INFO: CAM-ISP: cam_ife_csid_sof_irq_debug: 4250 SOF freeze: CSID SOF irq enabled

Alex Kushleyev

@Jetson-Nano , SOF freeze condition is when the ISP never gets a frame and times out. Something strange is going on.

Are any cameras working at all?

Your original post stated that your cameras would stop working in mid flight, but now it seems they dont work at all?

Alex

Jetson Nano

@Alex-Kushleyev Based on the above guidelines you mentioned I tried debugging the system messages along with pasting bin needed for respective ports. The ToF Also I made some interesting observation. ToF when plugged and seated firmly at certain angle works for me. Initially I tested with only ToF with required bin files in place. I receive the following output,

ToF is seated on J6L port

 643.046323] qcom_rpmh DRV:apps_rsc TCS Busy, retrying RPMH message send: addr=0x41b08
[  643.046350] qcom_rpmh DRV:apps_rsc TCS Busy, retrying RPMH message send: addr=0x41b08
[  643.046369] qcom_rpmh DRV:apps_rsc TCS Busy, retrying RPMH message send: addr=0x41b08
[  643.046388] qcom_rpmh DRV:apps_rsc TCS Busy, retrying RPMH message send: addr=0x41b08
[  643.046406] qcom_rpmh DRV:apps_rsc TCS Busy, retrying RPMH message send: addr=0x41b08
[  643.046425] qcom_rpmh DRV:apps_rsc TCS Busy, retrying RPMH message send: addr=0x41b08
[  643.046443] qcom_rpmh DRV:apps_rsc TCS Busy, retrying RPMH message send: addr=0x41b08
[  643.046461] qcom_rpmh DRV:apps_rsc TCS Busy, retrying RPMH message send: addr=0x41b08
[  643.046480] qcom_rpmh DRV:apps_rsc TCS Busy, retrying RPMH message send: addr=0x41b08
[  670.810479] ufshcd-qcom 1d84000.ufshc: ufshcd_ioctl: User buffer is NULL!
[  693.797692] qcom_rpmh DRV:apps_rsc TCS Busy, retrying RPMH message send: addr=0x40d00
[  700.336130] hrtimer: interrupt took 105417 ns
[  721.252079] ufshcd-qcom 1d84000.ufshc: ufshcd_ioctl: User buffer is NULL!
[  747.535072] CAM_ERR: CAM-MEM: cam_mem_mgr_create_debug_fs: 126 failed to create dentry
[  747.564180] CAM_INFO: CAM-HFI: cam_hfi_init: 878 Init IO1 : [0x10c00000 0xcf300000] IO2 [0xe0200000 0x1ed00000]
[  747.574301] CAM_INFO: CAM-ICP: cam_icp_mgr_hw_open: 3879 FW download done successfully
[  747.574309] CAM_INFO: CAM-ICP: cam_icp_mgr_process_dbg_buf: 2572 FW_DBG:CICP_FW_E : HFI  :QC_IMAGE_VERSION_STRING=CICP.FW.1.0-00079,OEM_IMAGE_VERSION_STRING=CRM,BUILD_TIME: Oct 17 2019 05:49:19,CACHE_ENABLED at icphostinterface.c:636 QC_IMAGE_VERSION_STRING=CICP.FW.1.0-00079 OEM_IMAGE_VERSION_STRING=CRM
[  747.574315] CAM_INFO: CAM-ICP: cam_icp_mgr_process_dbg_buf: 2572 FW_DBG:CICP_FW_E : HFI  :ELF variant: CACHE-ENABLED:T480:API_V2:USE_CDM_1_1: , API version: 0x2000049 at icphostinterface.c:637 QC_IMAGE_VERSION_STRING=CICP.FW.1.0-00079 OEM_IMAGE_VERSION_STRING=CRM
[  747.667487] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Linked as a consumer to regulator.60
[  747.667526] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Linked as a consumer to regulator.79
[  747.696358] CAM_INFO: CAM-SENSOR: cam_sensor_driver_cmd: 918 Probe success,slot:2,slave_addr:0x7a,sensor_id:0x3d
[  747.696481] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Dropping the link to regulator.79
[  747.700211] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Dropping the link to regulator.60
[  747.744388] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Linked as a consumer to regulator.60
[  747.744579] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Linked as a consumer to regulator.79
[  747.767225] CAM_INFO: CAM-SENSOR: cam_sensor_driver_cmd: 997 CAM_ACQUIRE_DEV Success, sensor_id:0x3d,sensor_slave_addr:0x7a
[  748.124228] CAM_ERR: CAM-ISP: cam_ife_hw_mgr_print_acquire_info: 710 Successfully acquire single IFE[6 -1] with [0 pix] [0 pd] [1 rdi] ports for ctx:0
[  748.124972] CAM_ERR: CAM-CRM: cam_req_mgr_cb_add_req: 2742 req 1 not found in in_q
[  748.125219] CAM_INFO: CAM-CSIPHY: cam_csiphy_core_cfg: 1137 START_DEV: CSIPHY_IDX: 2, Device_slot: 0, Datarate: 288000000, Settletime: 2800000000
[  748.133931] CAM_INFO: CAM-ISP: cam_vfe_bus_ver3_init_hw: 3659 Overriding clock gating at bus input
[  748.133934] CAM_INFO: CAM-ISP: cam_vfe_top_ver3_init_hw: 246 Disable clock gating at IFE top
[  748.134108] CAM_ERR: CAM-ISP: cam_ife_mgr_start_hw: 4510 ->Config HW, 00000000a3bb1d21
[  748.134411] CAM_INFO: CAM-SENSOR: cam_sensor_driver_cmd: 1089 CAM_START_DEV Success, sensor_id:0x3d,sensor_slave_addr:0x7a
[  748.146857] CAM_ERR: CAM-ISP: cam_ife_csid_irq: 4927 CSID:6 UNBOUNDED_FRAME
[  762.859114] qcom_rpmh DRV:apps_rsc TCS Busy, retrying RPMH message send: addr=0x41b08
[  762.859143] qcom_rpmh DRV:apps_rsc TCS Busy, retrying RPMH message send: addr=0x41b08
[  835.472101] perf: interrupt took too long (3965 > 3960), lowering kernel.perf_event_max_sample_rate to 50250

Further upon I tested with a Tracking Camera ov7251 attached, where I'm receiving data from the sensor and feed from sensors could be viewed upon the portal.

Here ToF is seated as earlier on J6 L port and Tracking ov7251 on J7 L port

[    5.505291] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Linked as a consumer to regulator.79
[    5.526921] CAM_INFO: CAM-SENSOR: cam_sensor_set_alt_id: 731 [Applying Alternative Address] : Sensor Index   : 391385268
[    5.526926] CAM_INFO: CAM-SENSOR: cam_sensor_set_alt_id: 732 [Applying Alternative Address] : Slave Address  : 0xE2
[    5.526930] CAM_INFO: CAM-SENSOR: cam_sensor_set_alt_id: 733 [Applying Alternative Address] : Slave sensor id: 0x7750
[    5.526934] CAM_INFO: CAM-SENSOR: cam_sensor_set_alt_id: 734 [Applying Alternative Address] : cci_client sid : 0x71
[    5.527197] CAM_INFO: CAM-SENSOR: cam_sensor_set_alt_id: 770 [Applying Alternative Address] : Probe : Alternate address updated
[    5.527203] CAM_INFO: CAM-SENSOR: cam_sensor_driver_cmd: 997 CAM_ACQUIRE_DEV Success, sensor_id:0x7750,sensor_slave_addr:0xe2
[    5.556310] CAM_ERR: CAM-ISP: cam_ife_hw_mgr_print_acquire_info: 710 Successfully acquire single IFE[6 -1] with [0 pix] [0 pd] [1 rdi] ports for ctx:0
[    5.557998] CAM_INFO: CAM-CSIPHY: cam_csiphy_core_cfg: 1137 START_DEV: CSIPHY_IDX: 2, Device_slot: 0, Datarate: 384000000, Settletime: 2800000000
[    5.567355] CAM_INFO: CAM-ISP: cam_vfe_bus_ver3_init_hw: 3659 Overriding clock gating at bus input
[    5.567362] CAM_INFO: CAM-ISP: cam_vfe_top_ver3_init_hw: 246 Disable clock gating at IFE top
[    5.567662] CAM_ERR: CAM-ISP: cam_ife_mgr_start_hw: 4510 ->Config HW, 00000000b07f5b5e
[    5.568263] CAM_INFO: CAM-SENSOR: cam_sensor_driver_cmd: 1089 CAM_START_DEV Success, sensor_id:0x7750,sensor_slave_addr:0xe2
[    5.710704] android_work: sent uevent USB_STATE=CONNECTED
[    5.720776] configfs-gadget gadget: high-speed config #1: c
[    5.721016] android_work: sent uevent USB_STATE=CONFIGURED
[    5.760593] CAM_ERR: CAM-ISP: cam_ife_hw_mgr_print_acquire_info: 710 Successfully acquire single IFE[5 -1] with [0 pix] [0 pd] [1 rdi] ports for ctx:1
[    5.761075] CAM_ERR: CAM-CRM: cam_req_mgr_cb_add_req: 2742 req 1 not found in in_q
[    5.761251] CAM_INFO: CAM-CSIPHY: cam_csiphy_core_cfg: 1137 START_DEV: CSIPHY_IDX: 0, Device_slot: 0, Datarate: 288000000, Settletime: 2800000000
[    5.769634] CAM_INFO: CAM-ISP: cam_vfe_bus_ver3_init_hw: 3659 Overriding clock gating at bus input
[    5.769638] CAM_INFO: CAM-ISP: cam_vfe_top_ver3_init_hw: 246 Disable clock gating at IFE top
[    5.769647] CAM_ERR: CAM-ISP: cam_ife_mgr_start_hw: 4510 ->Config HW, 00000000b07f5b5e
[    5.769921] CAM_INFO: CAM-SENSOR: cam_sensor_driver_cmd: 1089 CAM_START_DEV Success, sensor_id:0x3d,sensor_slave_addr:0x7a
[    5.783209] CAM_ERR: CAM-ISP: cam_ife_csid_irq: 4927 CSID:5 UNBOUNDED_FRAME
[    6.030729] spi_geni 98c000.spi: tx_fifo 16 rx_fifo 16 tx_width 32
[    6.215921] Started adbd. 
[    6.374976] IPv6: ADDRCONF(NETDEV_UP): wlan0: link is not ready
[    6.387472] IPv6: ADDRCONF(NETDEV_CHANGE): wlan0: link becomes ready
[    6.391291] QTI:Netlink Query to Kernel Success
[   10.090752] devfreq-qcom-fw 18590000.qcom,devfreq-l3:qcom,cdsp-cdsp-l3-lat: Successfully started CDSP L3 governor
[   11.435643] netlink: 'hostapd': attribute type 213 has an invalid length.
[   11.489516] RTW: assoc success
[   11.489998] RTW: set group key camid:0, addr:00:c0:ca:b3:a3:e9, kid:1, type:AES
[   28.000258] CAM_INFO: CAM-CRM: cam_req_mgr_process_flush_req: 2177 Last request id to flush is 115
[   28.000507] CAM_INFO: CAM-ISP: __cam_isp_ctx_flush_req_in_top_state: 3046 Last request id to flush is 115
[   28.000960] CAM_INFO: CAM-ISP: __cam_isp_ctx_flush_req_in_top_state: 3069 Stop HW complete. Reset HW next.
[   28.005985] CAM_INFO: CAM-SENSOR: cam_sensor_driver_cmd: 1117 CAM_STOP_DEV Success, sensor_id:0x3d,sensor_slave_addr:0x7a
[   28.006005] CAM_INFO: CAM-CSIPHY: cam_csiphy_core_cfg: 982 STOP_DEV: CSIPHY_IDX: 0, Device_slot: 0, Datarate: 288000000, Settletime: 2800000000
[   28.012879] CAM_INFO: CAM-ISP: cam_ife_mgr_release_hw: 4755 Release HW success ctx id: 1
[   28.014448] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Dropping the link to regulator.79
[   28.017762] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Dropping the link to regulator.60
[   28.018601] CAM_INFO: CAM-SENSOR: cam_sensor_driver_cmd: 1047 CAM_RELEASE_DEV Success, sensor_id:0x3d,sensor_slave_addr:0x7a
[   28.066082] CAM_INFO: CAM-CRM: cam_req_mgr_process_flush_req: 2177 Last request id to flush is 679
[   28.066153] CAM_INFO: CAM-ISP: __cam_isp_ctx_flush_req_in_top_state: 3046 Last request id to flush is 679
[   28.066283] CAM_INFO: CAM-ISP: __cam_isp_ctx_flush_req_in_top_state: 3069 Stop HW complete. Reset HW next.
[   28.071811] CAM_INFO: CAM-SENSOR: cam_sensor_driver_cmd: 1117 CAM_STOP_DEV Success, sensor_id:0x7750,sensor_slave_addr:0xe2
[   28.071831] CAM_INFO: CAM-CSIPHY: cam_csiphy_core_cfg: 982 STOP_DEV: CSIPHY_IDX: 2, Device_slot: 0, Datarate: 384000000, Settletime: 2800000000
[   28.078536] CAM_INFO: CAM-ISP: cam_ife_mgr_release_hw: 4755 Release HW success ctx id: 0
[   28.080413] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Dropping the link to regulator.79
[   28.083593] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Dropping the link to regulator.60
[   28.084953] CAM_INFO: CAM-SENSOR: cam_sensor_driver_cmd: 1047 CAM_RELEASE_DEV Success, sensor_id:0x7750,sensor_slave_addr:0xe2
[   28.092411] rpmh_rsc_send_data: 1 callbacks suppressed
[   28.092417] qcom_rpmh DRV:apps_rsc TCS Busy, retrying RPMH message send: addr=0x30080
[   28.102533] CAM_WARN: CAM-CRM: cam_req_mgr_close: 160 release invoked associated userspace process has died
[   28.173983] CAM_ERR: CAM-MEM: cam_mem_mgr_create_debug_fs: 126 failed to create dentry
[   28.208650] CAM_INFO: CAM-HFI: cam_hfi_init: 878 Init IO1 : [0x10c00000 0xcf300000] IO2 [0xe0200000 0x1ed00000]
[   28.218782] CAM_INFO: CAM-ICP: cam_icp_mgr_process_dbg_buf: 2572 FW_DBG:CICP_FW_E : HFI  :QC_IMAGE_VERSION_STRING=CICP.FW.1.0-00079,OEM_IMAGE_VERSION_STRING=CRM,BUILD_TIME: Oct 17 2019 05:49:19,CACHE_ENABLED at icphostinterface.c:636 QC_IMAGE_VERSION_STRING=CICP.FW.1.0-00079 OEM_IMAGE_VERSION_STRING=CRM
[   28.218793] CAM_INFO: CAM-ICP: cam_icp_mgr_process_dbg_buf: 2572 FW_DBG:CICP_FW_E : HFI  :ELF variant: CACHE-ENABLED:T480:API_V2:USE_CDM_1_1: , API version: 0x2000049 at icphostinterface.c:637 QC_IMAGE_VERSION_STRING=CICP.FW.1.0-00079 OEM_IMAGE_VERSION_STRING=CRM
[   28.218982] CAM_INFO: CAM-ICP: cam_icp_mgr_hw_open: 3879 FW download done successfully
[   28.385352] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Linked as a consumer to regulator.60
[   28.385413] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Linked as a consumer to regulator.79
[   28.410828] CAM_INFO: CAM-SENSOR: cam_sensor_driver_cmd: 918 Probe success,slot:0,slave_addr:0x7a,sensor_id:0x3d
[   28.410999] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Dropping the link to regulator.79
[   28.415312] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Dropping the link to regulator.60
[   28.448500] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Linked as a consumer to regulator.60
[   28.448594] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Linked as a consumer to regulator.79
[   28.471853] CAM_INFO: CAM-SENSOR: cam_sensor_set_alt_id: 731 [Applying Alternative Address] : Sensor Index   : 391385268
[   28.471863] CAM_INFO: CAM-SENSOR: cam_sensor_set_alt_id: 732 [Applying Alternative Address] : Slave Address  : 0xE2
[   28.471870] CAM_INFO: CAM-SENSOR: cam_sensor_set_alt_id: 733 [Applying Alternative Address] : Slave sensor id: 0x7750
[   28.471876] CAM_INFO: CAM-SENSOR: cam_sensor_set_alt_id: 734 [Applying Alternative Address] : cci_client sid : 0x71
[   28.472189] CAM_INFO: CAM-SENSOR: cam_sensor_set_alt_id: 770 [Applying Alternative Address] : Probe : Alternate address updated
[   28.472454] CAM_INFO: CAM-SENSOR: cam_sensor_driver_cmd: 918 Probe success,slot:2,slave_addr:0xe2,sensor_id:0x7750
[   28.472681] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Dropping the link to regulator.79
[   28.476332] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Dropping the link to regulator.60
[   28.537077] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Linked as a consumer to regulator.60
[   28.537166] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Linked as a consumer to regulator.79
[   28.559234] CAM_INFO: CAM-SENSOR: cam_sensor_driver_cmd: 997 CAM_ACQUIRE_DEV Success, sensor_id:0x3d,sensor_slave_addr:0x7a
[   28.681627] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Linked as a consumer to regulator.60
[   28.681728] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Linked as a consumer to regulator.79
[   28.702666] CAM_INFO: CAM-SENSOR: cam_sensor_set_alt_id: 731 [Applying Alternative Address] : Sensor Index   : 391385268
[   28.702672] CAM_INFO: CAM-SENSOR: cam_sensor_set_alt_id: 732 [Applying Alternative Address] : Slave Address  : 0xE2
[   28.702676] CAM_INFO: CAM-SENSOR: cam_sensor_set_alt_id: 733 [Applying Alternative Address] : Slave sensor id: 0x7750
[   28.702680] CAM_INFO: CAM-SENSOR: cam_sensor_set_alt_id: 734 [Applying Alternative Address] : cci_client sid : 0x71
[   28.702943] CAM_INFO: CAM-SENSOR: cam_sensor_set_alt_id: 770 [Applying Alternative Address] : Probe : Alternate address updated
[   28.702949] CAM_INFO: CAM-SENSOR: cam_sensor_driver_cmd: 997 CAM_ACQUIRE_DEV Success, sensor_id:0x7750,sensor_slave_addr:0xe2
[   28.737746] CAM_ERR: CAM-ISP: cam_ife_hw_mgr_print_acquire_info: 710 Successfully acquire single IFE[6 -1] with [0 pix] [0 pd] [1 rdi] ports for ctx:2
[   28.739627] CAM_INFO: CAM-CSIPHY: cam_csiphy_core_cfg: 1137 START_DEV: CSIPHY_IDX: 2, Device_slot: 0, Datarate: 384000000, Settletime: 2800000000
[   28.748886] CAM_INFO: CAM-ISP: cam_vfe_bus_ver3_init_hw: 3659 Overriding clock gating at bus input
[   28.748893] CAM_INFO: CAM-ISP: cam_vfe_top_ver3_init_hw: 246 Disable clock gating at IFE top
[   28.757430] CAM_ERR: CAM-ISP: cam_ife_mgr_start_hw: 4510 ->Config HW, 00000000b07f5b5e
[   28.757854] CAM_INFO: CAM-SENSOR: cam_sensor_driver_cmd: 1089 CAM_START_DEV Success, sensor_id:0x7750,sensor_slave_addr:0xe2
[   29.029065] CAM_ERR: CAM-ISP: cam_ife_hw_mgr_print_acquire_info: 710 Successfully acquire single IFE[5 -1] with [0 pix] [0 pd] [1 rdi] ports for ctx:3
[   29.029714] CAM_ERR: CAM-CRM: cam_req_mgr_cb_add_req: 2742 req 1 not found in in_q
[   29.030119] CAM_INFO: CAM-CSIPHY: cam_csiphy_core_cfg: 1137 START_DEV: CSIPHY_IDX: 0, Device_slot: 0, Datarate: 288000000, Settletime: 2800000000
[   29.039005] CAM_INFO: CAM-ISP: cam_vfe_bus_ver3_init_hw: 3659 Overriding clock gating at bus input
[   29.039012] CAM_INFO: CAM-ISP: cam_vfe_top_ver3_init_hw: 246 Disable clock gating at IFE top
[   29.039031] CAM_ERR: CAM-ISP: cam_ife_mgr_start_hw: 4510 ->Config HW, 00000000b07f5b5e
[   29.039547] CAM_INFO: CAM-SENSOR: cam_sensor_driver_cmd: 1089 CAM_START_DEV Success, sensor_id:0x3d,sensor_slave_addr:0x7a

Jetson Nano

@Alex-Kushleyev I have also attached the below output where ToF sensor fails sometimes during midflight, we tried reattaching the flex connectors to interposers and works fine for a while and shuts off. I tried to troubleshoot on the following note,

Case 1: Only Tracking enabled with ToF attached - I copied the required .bin files to /usr/lib/camera/. I restart the voxl-camera-server and the Tracking camera works fine on ground and midflight, without being frozen or feed being stalled from server.

Case 2: Only ToF enabled, with Tracking attached - I copied the required bin files to the destination. I restarted the voxl-camera-server, after a while data starts getting published. But the senosr doesn't always work. On a scale of comparison based on testing, the sensor stops publishing data 2 out of 5 run of tests.

Tried increasing the FPS of the sensor from 5 to 15, to check if would help. It helped improve mapping quality, but not well to solve the issue of losing feed.

Case 3: ToF + Tracking ov7251 attached and enabled - Configured the voxl-configure-cameras to configure 5. Then I restart the systemctl restart voxl-camera-server, the camera server restarts and feed is viewed in portal since cameras are publishing data. Sometimes upon restarting or running voxl-camera-server only, the cameras are detected and the server gets aborted abruptly or a message of gracefully exited appears.

Sometimes, the server goes of randomly on ground or mid-air and comes back again. On running dmesg -w on a terminal, I get the following output.

[  827.840180] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Linked as a consumer to regulator.58
[  827.842571] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Linked as a consumer to regulator.55
[  827.844131] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Linked as a consumer to regulator.60
[  827.844205] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Linked as a consumer to regulator.79
[  827.870672] CAM_INFO: CAM-SENSOR: cam_sensor_driver_cmd: 918 Probe success,slot:0,slave_addr:0x7a,sensor_id:0x3d
[  827.870946] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Dropping the link to regulator.79
[  827.874552] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Dropping the link to regulator.60
[  827.874719] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Dropping the link to regulator.55
[  827.874859] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Dropping the link to regulator.58
[  827.905837] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Linked as a consumer to regulator.59
[  827.908159] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Linked as a consumer to regulator.55
[  827.910035] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Linked as a consumer to regulator.60
[  827.910156] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Linked as a consumer to regulator.79
[  827.932850] CAM_INFO: CAM-SENSOR: cam_sensor_set_alt_id: 731 [Applying Alternative Address] : Sensor Index   : 1465270452
[  827.932859] CAM_INFO: CAM-SENSOR: cam_sensor_set_alt_id: 732 [Applying Alternative Address] : Slave Address  : 0xE2
[  827.932866] CAM_INFO: CAM-SENSOR: cam_sensor_set_alt_id: 733 [Applying Alternative Address] : Slave sensor id: 0x7750
[  827.932872] CAM_INFO: CAM-SENSOR: cam_sensor_set_alt_id: 734 [Applying Alternative Address] : cci_client sid : 0x71
[  827.933224] CAM_INFO: CAM-SENSOR: cam_sensor_set_alt_id: 770 [Applying Alternative Address] : Probe : Alternate address updated
[  827.933477] CAM_INFO: CAM-SENSOR: cam_sensor_driver_cmd: 918 Probe success,slot:2,slave_addr:0xe2,sensor_id:0x7750
[  827.933651] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Dropping the link to regulator.79
[  827.937888] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Dropping the link to regulator.60
[  827.938082] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Dropping the link to regulator.55
[  827.938221] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Dropping the link to regulator.59
[  828.000369] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Linked as a consumer to regulator.58
[  828.002330] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Linked as a consumer to regulator.55
[  828.004189] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Linked as a consumer to regulator.60
[  828.004269] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Linked as a consumer to regulator.79
[  828.028279] CAM_INFO: CAM-SENSOR: cam_sensor_driver_cmd: 997 CAM_ACQUIRE_DEV Success, sensor_id:0x3d,sensor_slave_addr:0x7a
[  828.478535] CAM_ERR: CAM-ISP: cam_ife_hw_mgr_print_acquire_info: 710 Successfully acquire single IFE[6 -1] with [0 pix] [0 pd] [1 rdi] ports for ctx:8
[  828.479181] CAM_ERR: CAM-CRM: cam_req_mgr_cb_add_req: 2742 req 1 not found in in_q
[  828.479457] CAM_INFO: CAM-CSIPHY: cam_csiphy_core_cfg: 1137 START_DEV: CSIPHY_IDX: 0, Device_slot: 0, Datarate: 288000000, Settletime: 2800000000
[  828.487975] CAM_INFO: CAM-ISP: cam_vfe_bus_ver3_init_hw: 3659 Overriding clock gating at bus input
[  828.487979] CAM_INFO: CAM-ISP: cam_vfe_top_ver3_init_hw: 246 Disable clock gating at IFE top
[  828.488207] CAM_ERR: CAM-ISP: cam_ife_mgr_start_hw: 4510 ->Config HW, 00000000851a209a
[  828.488561] CAM_INFO: CAM-SENSOR: cam_sensor_driver_cmd: 1089 CAM_START_DEV Success, sensor_id:0x3d,sensor_slave_addr:0x7a
[  841.529716] RTW: wlan0- hw port(0) mac_addr =00:c0:ca:b3:a3:e9
[  862.477578] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Dropping the link to regulator.79
[  862.481705] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Dropping the link to regulator.60
[  862.481785] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Dropping the link to regulator.55
[  862.481857] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Dropping the link to regulator.58
[  862.490614] CAM_ERR: CAM-SYNC: cam_sync_signal: 203 Error: accessing an uninitialized sync obj = 5
[  862.490620] CAM_ERR: CAM-SYNC: cam_sync_signal: 203 Error: accessing an uninitialized sync obj = 1
[  862.490623] CAM_ERR: CAM-SYNC: cam_sync_signal: 203 Error: accessing an uninitialized sync obj = 2
[  862.490626] CAM_ERR: CAM-SYNC: cam_sync_signal: 203 Error: accessing an uninitialized sync obj = 4
[  862.490630] CAM_ERR: CAM-SYNC: cam_sync_signal: 203 Error: accessing an uninitialized sync obj = 3
[  862.490634] CAM_ERR: CAM-ISP: __cam_isp_ctx_release_dev_in_top_state: 3789 releasing hw
[  862.490658] CAM_INFO: CAM-ISP: cam_ife_mgr_release_hw: 4755 Release HW success ctx id: 8
[  862.490757] CAM_WARN: CAM-CRM: cam_req_mgr_close: 160 release invoked associated userspace process has died
[  862.490838] CAM-REQ-MGR_UTIL cam_req_mgr_util_free_hdls:99 CAM_WARN: CAM-CRM: cam_req_mgr_util_free_hdls: 99 Dev handle = f10102 session_handle = 6e0200
[  862.493693] ion_buffer_destroy: 1 callbacks suppressed
[  862.493695] ION client likely missing a call to dma_buf_kunmap or dma_buf_vunmap
[  862.493706] ION client likely missing a call to dma_buf_kunmap or dma_buf_vunmap
[  862.493715] ION client likely missing a call to dma_buf_kunmap or dma_buf_vunmap
[  862.493803] ION client likely missing a call to dma_buf_kunmap or dma_buf_vunmap
[  862.493839] ION client likely missing a call to dma_buf_kunmap or dma_buf_vunmap
[  862.494021] ION client likely missing a call to dma_buf_kunmap or dma_buf_vunmap
[  862.494253] ION client likely missing a call to dma_buf_kunmap or dma_buf_vunmap
[  862.494262] ION client likely missing a call to dma_buf_kunmap or dma_buf_vunmap
[  862.494282] ION client likely missing a call to dma_buf_kunmap or dma_buf_vunmap
[  862.494288] ION client likely missing a call to dma_buf_kunmap or dma_buf_vunmap
[  863.270637] RTW: wlan0- hw port(0) mac_addr =00:c0:ca:b3:a3:e9
[  870.888214] qcom_rpmh DRV:apps_rsc TCS Busy, retrying RPMH message send: addr=0x41b08
[  870.888244] qcom_rpmh DRV:apps_rsc TCS Busy, retrying RPMH message send: addr=0x41b08
[  875.098507] CAM_ERR: CAM-MEM: cam_mem_mgr_create_debug_fs: 126 failed to create dentry
[  875.124437] CAM_INFO: CAM-HFI: cam_hfi_init: 878 Init IO1 : [0x10c00000 0xcf300000] IO2 [0xe0200000 0x1ed00000]
[  875.134626] CAM_INFO: CAM-ICP: cam_icp_mgr_process_dbg_buf: 2572 FW_DBG:CICP_FW_E : HFI  :QC_IMAGE_VERSION_STRING=CICP.FW.1.0-00079,OEM_IMAGE_VERSION_STRING=CRM,BUILD_TIME: Oct 17 2019 05:49:19,CACHE_ENABLED at icphostinterface.c:636 QC_IMAGE_VERSION_STRING=CICP.FW.1.0-00079 OEM_IMAGE_VERSION_STRING=CRM
[  875.134631] CAM_INFO: CAM-ICP: cam_icp_mgr_process_dbg_buf: 2572 FW_DBG:CICP_FW_E : HFI  :ELF variant: CACHE-ENABLED:T480:API_V2:USE_CDM_1_1: , API version: 0x2000049 at icphostinterface.c:637 QC_IMAGE_VERSION_STRING=CICP.FW.1.0-00079 OEM_IMAGE_VERSION_STRING=CRM
[  875.134702] CAM_INFO: CAM-ICP: cam_icp_mgr_hw_open: 3879 FW download done successfully
[  875.357708] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Linked as a consumer to regulator.58
[  875.359880] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Linked as a consumer to regulator.55
[  875.362038] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Linked as a consumer to regulator.60
[  875.362116] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Linked as a consumer to regulator.79
[  875.387690] CAM_INFO: CAM-SENSOR: cam_sensor_driver_cmd: 918 Probe success,slot:0,slave_addr:0x7a,sensor_id:0x3d
[  875.387887] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Dropping the link to regulator.79
[  875.390781] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Dropping the link to regulator.60
[  875.390890] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Dropping the link to regulator.55
[  875.390989] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Dropping the link to regulator.58
[  875.410081] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Linked as a consumer to regulator.59
[  875.412357] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Linked as a consumer to regulator.55
[  875.414556] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Linked as a consumer to regulator.60
[  875.414677] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Linked as a consumer to regulator.79
[  875.438443] CAM_INFO: CAM-SENSOR: cam_sensor_set_alt_id: 731 [Applying Alternative Address] : Sensor Index   : 1465270452
[  875.438452] CAM_INFO: CAM-SENSOR: cam_sensor_set_alt_id: 732 [Applying Alternative Address] : Slave Address  : 0xE2
[  875.438459] CAM_INFO: CAM-SENSOR: cam_sensor_set_alt_id: 733 [Applying Alternative Address] : Slave sensor id: 0x7750
[  875.438464] CAM_INFO: CAM-SENSOR: cam_sensor_set_alt_id: 734 [Applying Alternative Address] : cci_client sid : 0x71
[  875.438825] CAM_INFO: CAM-SENSOR: cam_sensor_set_alt_id: 770 [Applying Alternative Address] : Probe : Alternate address updated
[  875.439083] CAM_INFO: CAM-SENSOR: cam_sensor_driver_cmd: 918 Probe success,slot:2,slave_addr:0xe2,sensor_id:0x7750
[  875.439587] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Dropping the link to regulator.79
[  875.443944] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Dropping the link to regulator.60
[  875.444426] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Dropping the link to regulator.55
[  875.444571] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Dropping the link to regulator.59
[  875.511578] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Linked as a consumer to regulator.58
[  875.513201] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Linked as a consumer to regulator.55
[  875.515282] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Linked as a consumer to regulator.60
[  875.515350] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Linked as a consumer to regulator.79
[  875.538928] CAM_INFO: CAM-SENSOR: cam_sensor_driver_cmd: 997 CAM_ACQUIRE_DEV Success, sensor_id:0x3d,sensor_slave_addr:0x7a
[  876.005639] CAM_ERR: CAM-ISP: cam_ife_hw_mgr_print_acquire_info: 710 Successfully acquire single IFE[6 -1] with [0 pix] [0 pd] [1 rdi] ports for ctx:9
[  876.006492] CAM_ERR: CAM-CRM: cam_req_mgr_cb_add_req: 2742 req 1 not found in in_q
[  876.006720] CAM_INFO: CAM-CSIPHY: cam_csiphy_core_cfg: 1137 START_DEV: CSIPHY_IDX: 0, Device_slot: 0, Datarate: 288000000, Settletime: 2800000000
[  876.015994] CAM_INFO: CAM-ISP: cam_vfe_bus_ver3_init_hw: 3659 Overriding clock gating at bus input
[  876.015997] CAM_INFO: CAM-ISP: cam_vfe_top_ver3_init_hw: 246 Disable clock gating at IFE top
[  876.016401] CAM_ERR: CAM-ISP: cam_ife_mgr_start_hw: 4510 ->Config HW, 00000000851a209a
[  876.016793] CAM_INFO: CAM-SENSOR: cam_sensor_driver_cmd: 1089 CAM_START_DEV Success, sensor_id:0x3d,sensor_slave_addr:0x7a
[  876.019097] CAM_INFO: CAM-CRM: __cam_req_mgr_find_dev_name: 209 Skip Frame: req: 2 not ready on link: 0xa10104 for pd: 2 dev: cam-sensor open_req count: 4
[  880.848318] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Dropping the link to regulator.79
[  880.852399] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Dropping the link to regulator.60
[  880.852461] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Dropping the link to regulator.55
[  880.852506] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Dropping the link to regulator.58
[  880.865126] CAM_ERR: CAM-SYNC: cam_sync_signal: 203 Error: accessing an uninitialized sync obj = 2
[  880.865140] CAM_ERR: CAM-SYNC: cam_sync_signal: 203 Error: accessing an uninitialized sync obj = 3
[  880.865150] CAM_ERR: CAM-SYNC: cam_sync_signal: 203 Error: accessing an uninitialized sync obj = 4
[  880.865159] CAM_ERR: CAM-SYNC: cam_sync_signal: 203 Error: accessing an uninitialized sync obj = 5
[  880.865168] CAM_ERR: CAM-SYNC: cam_sync_signal: 203 Error: accessing an uninitialized sync obj = 1
[  880.865179] CAM_ERR: CAM-ISP: __cam_isp_ctx_release_dev_in_top_state: 3789 releasing hw
[  880.865285] CAM_INFO: CAM-ISP: cam_ife_mgr_release_hw: 4755 Release HW success ctx id: 9
[  880.865409] CAM_WARN: CAM-CRM: cam_req_mgr_close: 160 release invoked associated userspace process has died
[  880.865699] CAM-REQ-MGR_UTIL cam_req_mgr_util_free_hdls:99 CAM_WARN: CAM-CRM: cam_req_mgr_util_free_hdls: 99 Dev handle = df0102 session_handle = a60200
[  880.871851] ion_buffer_destroy: 1 callbacks suppressed
[  880.871858] ION client likely missing a call to dma_buf_kunmap or dma_buf_vunmap
[  880.871999] ION client likely missing a call to dma_buf_kunmap or dma_buf_vunmap
[  880.873664] ION client likely missing a call to dma_buf_kunmap or dma_buf_vunmap
[  880.873691] ION client likely missing a call to dma_buf_kunmap or dma_buf_vunmap
[  880.873866] ION client likely missing a call to dma_buf_kunmap or dma_buf_vunmap
[  880.873889] ION client likely missing a call to dma_buf_kunmap or dma_buf_vunmap
[  880.873988] ION client likely missing a call to dma_buf_kunmap or dma_buf_vunmap
[  880.874057] ION client likely missing a call to dma_buf_kunmap or dma_buf_vunmap
[  880.879726] ION client likely missing a call to dma_buf_kunmap or dma_buf_vunmap
[  880.879944] ION client likely missing a call to dma_buf_kunmap or dma_buf_vunmap
[  885.149685] RTW: wlan0- hw port(0) mac_addr =00:c0:ca:b3:a3:e9
[  890.823665] qcom_rpmh DRV:apps_rsc TCS Busy, retrying RPMH message send: addr=0x40d00
[  906.997270] RTW: wlan0- hw port(0) mac_addr =00:c0:ca:b3:a3:e9
[  920.855588] CAM_ERR: CAM-MEM: cam_mem_mgr_create_debug_fs: 126 failed to create dentry
[  920.884159] CAM_INFO: CAM-HFI: cam_hfi_init: 878 Init IO1 : [0x10c00000 0xcf300000] IO2 [0xe0200000 0x1ed00000]
[  920.894375] CAM_INFO: CAM-ICP: cam_icp_mgr_hw_open: 3879 FW download done successfully
[  920.894383] CAM_INFO: CAM-ICP: cam_icp_mgr_process_dbg_buf: 2572 FW_DBG:CICP_FW_E : HFI  :QC_IMAGE_VERSION_STRING=CICP.FW.1.0-00079,OEM_IMAGE_VERSION_STRING=CRM,BUILD_TIME: Oct 17 2019 05:49:19,CACHE_ENABLED at icphostinterface.c:636 QC_IMAGE_VERSION_STRING=CICP.FW.1.0-00079 OEM_IMAGE_VERSION_STRING=CRM
[  920.894389] CAM_INFO: CAM-ICP: cam_icp_mgr_process_dbg_buf: 2572 FW_DBG:CICP_FW_E : HFI  :ELF variant: CACHE-ENABLED:T480:API_V2:USE_CDM_1_1: , API version: 0x2000049 at icphostinterface.c:637 QC_IMAGE_VERSION_STRING=CICP.FW.1.0-00079 OEM_IMAGE_VERSION_STRING=CRM
[  921.122596] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Linked as a consumer to regulator.58
[  921.124189] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Linked as a consumer to regulator.55
[  921.126333] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Linked as a consumer to regulator.60
[  921.126411] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Linked as a consumer to regulator.79
[  921.153915] CAM_INFO: CAM-SENSOR: cam_sensor_driver_cmd: 918 Probe success,slot:0,slave_addr:0x7a,sensor_id:0x3d
[  921.154186] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Dropping the link to regulator.79
[  921.159914] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Dropping the link to regulator.60
[  921.160560] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Dropping the link to regulator.55
[  921.160703] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Dropping the link to regulator.58
[  921.188632] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Linked as a consumer to regulator.59
[  921.191511] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Linked as a consumer to regulator.55
[  921.193717] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Linked as a consumer to regulator.60
[  921.193854] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Linked as a consumer to regulator.79
[  921.218411] CAM_INFO: CAM-SENSOR: cam_sensor_set_alt_id: 731 [Applying Alternative Address] : Sensor Index   : 1465270452
[  921.218420] CAM_INFO: CAM-SENSOR: cam_sensor_set_alt_id: 732 [Applying Alternative Address] : Slave Address  : 0xE2
[  921.218427] CAM_INFO: CAM-SENSOR: cam_sensor_set_alt_id: 733 [Applying Alternative Address] : Slave sensor id: 0x7750
[  921.218432] CAM_INFO: CAM-SENSOR: cam_sensor_set_alt_id: 734 [Applying Alternative Address] : cci_client sid : 0x71
[  921.218777] CAM_INFO: CAM-SENSOR: cam_sensor_set_alt_id: 770 [Applying Alternative Address] : Probe : Alternate address updated
[  921.219067] CAM_INFO: CAM-SENSOR: cam_sensor_driver_cmd: 918 Probe success,slot:2,slave_addr:0xe2,sensor_id:0x7750
[  921.219268] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Dropping the link to regulator.79
[  921.223412] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Dropping the link to regulator.60
[  921.223568] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Dropping the link to regulator.55
[  921.223668] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Dropping the link to regulator.59
[  921.284538] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Linked as a consumer to regulator.58
[  921.285965] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Linked as a consumer to regulator.55
[  921.288211] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Linked as a consumer to regulator.60
[  921.288298] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Linked as a consumer to regulator.79
[  921.311468] CAM_INFO: CAM-SENSOR: cam_sensor_driver_cmd: 997 CAM_ACQUIRE_DEV Success, sensor_id:0x3d,sensor_slave_addr:0x7a
[  921.859799] CAM_ERR: CAM-ISP: cam_ife_hw_mgr_print_acquire_info: 710 Successfully acquire single IFE[6 -1] with [0 pix] [0 pd] [1 rdi] ports for ctx:1
[  921.861359] CAM_ERR: CAM-CRM: cam_req_mgr_cb_add_req: 2742 req 1 not found in in_q
[  921.861782] CAM_INFO: CAM-CSIPHY: cam_csiphy_core_cfg: 1137 START_DEV: CSIPHY_IDX: 0, Device_slot: 0, Datarate: 288000000, Settletime: 2800000000
[  921.872367] CAM_INFO: CAM-ISP: cam_vfe_bus_ver3_init_hw: 3659 Overriding clock gating at bus input
[  921.872375] CAM_INFO: CAM-ISP: cam_vfe_top_ver3_init_hw: 246 Disable clock gating at IFE top
[  921.872758] CAM_ERR: CAM-ISP: cam_ife_mgr_start_hw: 4510 ->Config HW, 00000000851a209a
[  921.873590] CAM_INFO: CAM-SENSOR: cam_sensor_driver_cmd: 1089 CAM_START_DEV Success, sensor_id:0x3d,sensor_slave_addr:0x7a
[  926.700983] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Dropping the link to regulator.79
[  926.704609] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Dropping the link to regulator.60
[  926.704746] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Dropping the link to regulator.55
[  926.704858] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Dropping the link to regulator.58
[  926.716260] CAM_ERR: CAM-SYNC: cam_sync_signal: 203 Error: accessing an uninitialized sync obj = 2
[  926.716270] CAM_ERR: CAM-SYNC: cam_sync_signal: 203 Error: accessing an uninitialized sync obj = 3
[  926.716278] CAM_ERR: CAM-SYNC: cam_sync_signal: 203 Error: accessing an uninitialized sync obj = 4
[  926.716286] CAM_ERR: CAM-SYNC: cam_sync_signal: 203 Error: accessing an uninitialized sync obj = 5
[  926.716293] CAM_ERR: CAM-SYNC: cam_sync_signal: 203 Error: accessing an uninitialized sync obj = 1
[  926.716302] CAM_ERR: CAM-ISP: __cam_isp_ctx_release_dev_in_top_state: 3789 releasing hw
[  926.716346] CAM_INFO: CAM-ISP: cam_ife_mgr_release_hw: 4755 Release HW success ctx id: 1
[  926.716439] CAM_WARN: CAM-CRM: cam_req_mgr_close: 160 release invoked associated userspace process has died
[  926.716536] CAM-REQ-MGR_UTIL cam_req_mgr_util_free_hdls:99 CAM_WARN: CAM-CRM: cam_req_mgr_util_free_hdls: 99 Dev handle = bb0102 session_handle = a10200
[  926.723046] ion_buffer_destroy: 1 callbacks suppressed
[  926.723050] ION client likely missing a call to dma_buf_kunmap or dma_buf_vunmap
[  926.723184] ION client likely missing a call to dma_buf_kunmap or dma_buf_vunmap
[  926.723315] ION client likely missing a call to dma_buf_kunmap or dma_buf_vunmap
[  926.723534] ION client likely missing a call to dma_buf_kunmap or dma_buf_vunmap
[  926.728238] ION client likely missing a call to dma_buf_kunmap or dma_buf_vunmap
[  926.728261] ION client likely missing a call to dma_buf_kunmap or dma_buf_vunmap
[  926.728311] ION client likely missing a call to dma_buf_kunmap or dma_buf_vunmap
[  926.728332] ION client likely missing a call to dma_buf_kunmap or dma_buf_vunmap
[  926.728425] ION client likely missing a call to dma_buf_kunmap or dma_buf_vunmap
[  926.728641] ION client likely missing a call to dma_buf_kunmap or dma_buf_vunmap
[  928.697163] RTW: wlan0- hw port(0) mac_addr =00:c0:ca:b3:a3:e9

Sometimes, when we tweak the cables slightly or power the system after a while, the camera server including the ToF and tracking work fine, even though CAM ERR message appear on terminal line system call messages.

Through these I'm not able to actually conclude if it's a hardware issue or software issue in the voxl-camera-server while running multiple camera sensors simultanelously through the server.

Would be great if you could help out.

Jetson Nano

@Alex-Kushleyev Is there something we could do to tweak the way the data is being published through the voxl-camera-server pipeline?

Jetson Nano

@Alex-Kushleyev could you look into?

Alex Kushleyev

Hi @Jetson-Nano ,

Thank you for sending the logs. Unfortunately, I do not see anything specific that would indicate an error in the last few logs.

Do you have more than one VOXL2 you could use for testing? also, do you have another TOF sensor you could use? It seems the issue is tied to the TOF sensor but it is not clear whether it is on the VOXL2 side (maybe connector issue) or the TOF sensor side.

What you are describing is not normal behavior - the camera server from any recent official release should be pretty stable when used with these standard configurations (tracking camera, TOF, etc).

Since the tracking camera by itself seems to be working fine, please focus on testing TOF only (you can disconnect the tracking camera) and make sure you can get that to work reliably, testing on a different VOXL2 or a different TOF module if possible (to rule out one of the two being an issue)

Also, which SDK are you using?

Alex

Jetson Nano

@Alex-Kushleyev Thanks for reverting. Yes, I have another VOXL2 board. I'll test and share the results. I'm currently testing with SDKs 1.1.1 and 1.2 releases.

Jetson Nano

@Alex-Kushleyev I tested the ToF with another VOXL2 board I have. Modified the camera config as only for ToF. On ground tests, the camera server seems to work fine. On terminal system logger publishes similar type of CAM_INFO, CAM_ERR dmesg -w messages as earlier.

We used the same interposer as used in earlier VOXL2 compute.

ModalAI Forum

CAM_MISSING_STALLED