CAM_MISSING_STALLED

Jetson Nano

@Alex-Kushleyev I did the test, with ToF enabled in config with others disabled and physically also removed the other cameras(tracking, hires) and got the following output,

[  500.965614] qcom_rpmh DRV:apps_rsc TCS Busy, retrying RPMH message send: addr=0x40d00
[  513.797938] qcom_rpmh DRV:apps_rsc TCS Busy, retrying RPMH message send: addr=0x40d00
[  563.809259] CAM_ERR: CAM-MEM: cam_mem_mgr_create_debug_fs: 126 failed to create dentry
[  563.837666] CAM_INFO: CAM-HFI: cam_hfi_init: 878 Init IO1 : [0x10c00000 0xcf300000] IO2 [0xe0200000 0x1ed00000]
[  563.847825] CAM_INFO: CAM-ICP: cam_icp_mgr_hw_open: 3879 FW download done successfully
[  563.847979] CAM_INFO: CAM-ICP: cam_icp_mgr_process_dbg_buf: 2572 FW_DBG:CICP_FW_E : HFI  :QC_IMAGE_VERSION_STRING=CICP.FW.1.0-00079,OEM_IMAGE_VERSION_STRING=CRM,BUILD_TIME: Oct 17 2019 05:49:19,CACHE_ENABLED at icphostinterface.c:636 QC_IMAGE_VERSION_STRING=CICP.FW.1.0-00079 OEM_IMAGE_VERSION_STRING=CRM
[  563.847983] CAM_INFO: CAM-ICP: cam_icp_mgr_process_dbg_buf: 2572 FW_DBG:CICP_FW_E : HFI  :ELF variant: CACHE-ENABLED:T480:API_V2:USE_CDM_1_1: , API version: 0x2000049 at icphostinterface.c:637 QC_IMAGE_VERSION_STRING=CICP.FW.1.0-00079 OEM_IMAGE_VERSION_STRING=CRM
[  564.015546] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Linked as a consumer to regulator.60
[  564.015606] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Linked as a consumer to regulator.79
[  564.042685] CAM_ERR: CAM-CCI: cam_cci_irq: 238 Base:        pK-error, M0_Q1 NACK ERROR: 0x10000000
[  564.042909] CAM_ERR: CAM-CCI: cam_cci_read: 1329 ERROR with Slave 0x7a:
[  564.042926] CAM_ERR: CAM-CCI: cam_cci_read_bytes: 1613 failed to read rc:-22
[  564.042935] CAM_ERR: CAM-SENSOR: cam_cci_i2c_read: 35 rc = -22
[  564.042945] CAM_WARN: CAM-SENSOR: cam_sensor_match_id: 707 read id: 0x0 expected id 0x3d:
[  564.043137] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Dropping the link to regulator.79
[  564.047327] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Dropping the link to regulator.60
[  564.076125] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Linked as a consumer to regulator.60
[  564.076278] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Linked as a consumer to regulator.79
[  564.093922] CAM_ERR: CAM-CCI: cam_cci_irq: 238 Base:        pK-error, M0_Q1 NACK ERROR: 0x10000000
[  564.093988] CAM_ERR: CAM-CCI: cam_cci_read: 1329 ERROR with Slave 0x20:
[  564.094000] CAM_ERR: CAM-CCI: cam_cci_read_bytes: 1613 failed to read rc:-22
[  564.094008] CAM_ERR: CAM-SENSOR: cam_cci_i2c_read: 35 rc = -22
[  564.094016] CAM_WARN: CAM-SENSOR: cam_sensor_match_id: 707 read id: 0x0 expected id 0x214:
[  564.094138] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Dropping the link to regulator.79
[  564.098341] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Dropping the link to regulator.60
[  564.129852] qcom,camera ac50000.qcom,cci:qcom,cam-sensor3: Linked as a consumer to regulator.55
[  564.132377] qcom,camera ac50000.qcom,cci:qcom,cam-sensor3: Linked as a consumer to regulator.60
[  564.132522] qcom,camera ac50000.qcom,cci:qcom,cam-sensor3: Linked as a consumer to regulator.79
[  564.156581] CAM_INFO: CAM-SENSOR: cam_sensor_set_alt_id: 731 [Applying Alternative Address] : Sensor Index   : 391549108
[  564.156594] CAM_INFO: CAM-SENSOR: cam_sensor_set_alt_id: 732 [Applying Alternative Address] : Slave Address  : 0xE2
[  564.156602] CAM_INFO: CAM-SENSOR: cam_sensor_set_alt_id: 733 [Applying Alternative Address] : Slave sensor id: 0x7750
[  564.156610] CAM_INFO: CAM-SENSOR: cam_sensor_set_alt_id: 734 [Applying Alternative Address] : cci_client sid : 0x71
[  564.156830] CAM_ERR: CAM-CCI: cam_cci_irq: 258 Base:        pK-error, M1_Q0 NACK ERROR: 0x20000000
[  564.156882] CAM_ERR: CAM-CCI: cam_cci_wait: 270 failed rc -22
[  564.156891] CAM_ERR: CAM-CCI: cam_cci_transfer_end: 345 failed rc -22
[  564.156899] CAM_ERR: CAM-CCI: cam_cci_data_queue: 870 Slave: 0xe0 failed rc -22
[  564.156907] CAM_ERR: CAM-CCI: cam_cci_i2c_write: 1450 failed rc: -22
[  564.156916] CAM_ERR: CAM-SENSOR: cam_cci_i2c_write_table_cmd: 123 Failed rc = -22
[  564.156924] CAM_ERR: CAM-SENSOR: cam_sensor_set_alt_id: 768 [Applying Alternative Address] : Probe : camera_io_dev_write failed: rc=-22
[  564.157075] CAM_ERR: CAM-CCI: cam_cci_irq: 264 Base:        pK-error, M1_Q1 NACK ERROR: 0x40000000
[  564.157195] CAM_ERR: CAM-CCI: cam_cci_read: 1329 ERROR with Slave 0xe0:
[  564.157203] CAM_ERR: CAM-CCI: cam_cci_read_bytes: 1613 failed to read rc:-22
[  564.157210] CAM_ERR: CAM-SENSOR: cam_cci_i2c_read: 35 rc = -22
[  564.157219] CAM_WARN: CAM-SENSOR: cam_sensor_match_id: 707 read id: 0x0 expected id 0x7750:
[  564.157384] qcom,camera ac50000.qcom,cci:qcom,cam-sensor3: Dropping the link to regulator.79
[  564.161659] qcom,camera ac50000.qcom,cci:qcom,cam-sensor3: Dropping the link to regulator.60
[  564.161850] qcom,camera ac50000.qcom,cci:qcom,cam-sensor3: Dropping the link to regulator.55
[  564.214462] CAM_WARN: CAM-CRM: cam_req_mgr_close: 160 release invoked associated userspace process has died
[  593.797336] qcom_rpmh DRV:apps_rsc TCS Busy, retrying RPMH message send: addr=0x40d00
[  593.797362] qcom_rpmh DRV:apps_rsc TCS Busy, retrying RPMH message send: addr=0x40d00

Jetson Nano

@Jetson-Nano I tried doing the same above process of checking for Hires IMX214 cam also, found the following using dmesg -w. I configured C-8 configuration upon J7-U port.

[ 1414.886039] cam_csid_evt_bottom_half_handler: 12 callbacks suppressed
[ 1414.886043] CAM_ERR: CAM-ISP: cam_csid_evt_bottom_half_handler: 4621 status RDI0: 0
[ 1414.886046] CAM_ERR: CAM-ISP: cam_csid_evt_bottom_half_handler: 4621 status RDI1: 0
[ 1414.886049] CAM_ERR: CAM-ISP: cam_csid_evt_bottom_half_handler: 4621 status RDI2: 0
[ 1414.886053] CAM_ERR: CAM-ISP: cam_csid_evt_bottom_half_handler: 4621 status RDI3: 0
[ 1414.886056] CAM_ERR: CAM-ISP: cam_csid_evt_bottom_half_handler: 4621 status TOP: 0
[ 1414.886060] CAM_ERR: CAM-ISP: cam_csid_evt_bottom_half_handler: 4621 status RX: 38000
[ 1414.886063] CAM_ERR: CAM-ISP: cam_csid_evt_bottom_half_handler: 4621 status IPP: 0
[ 1414.886066] CAM_ERR: CAM-ISP: cam_csid_evt_bottom_half_handler: 4621 status PPP: 0
[ 1414.886070] CAM_ERR: CAM-ISP: cam_csid_evt_bottom_half_handler: 4621 status UDI0: 0
[ 1414.886073] CAM_ERR: CAM-ISP: cam_csid_evt_bottom_half_handler: 4621 status UDI1: 0
[ 1414.886081] CAM_ERR: CAM-ISP: cam_csid_evt_bottom_half_handler: 4616 idx 1 err 5 phy 3  lane type:0 ln num:4 ln cfg:0x3210 cnt 2
[ 1416.258821] msm_vidc:   err : ffffffff: .....: DBLP Set: status 0
[ 1420.000213] CAM_ERR: CAM-CRM: __cam_req_mgr_process_sof_freeze: 1800 SOF freeze for session 8520192 link 0xb50105
[ 1420.000250] CAM_INFO: CAM-ISP: cam_ife_csid_sof_irq_debug: 4250 SOF freeze: CSID SOF irq enabled
[ 1420.000265] CAM_INFO: CAM-ISP: cam_ife_csid_sof_irq_debug: 4250 SOF freeze: CSID SOF irq enabled

Alex Kushleyev

@Jetson-Nano , SOF freeze condition is when the ISP never gets a frame and times out. Something strange is going on.

Are any cameras working at all?

Your original post stated that your cameras would stop working in mid flight, but now it seems they dont work at all?

Alex

Jetson Nano

@Alex-Kushleyev Based on the above guidelines you mentioned I tried debugging the system messages along with pasting bin needed for respective ports. The ToF Also I made some interesting observation. ToF when plugged and seated firmly at certain angle works for me. Initially I tested with only ToF with required bin files in place. I receive the following output,

ToF is seated on J6L port

 643.046323] qcom_rpmh DRV:apps_rsc TCS Busy, retrying RPMH message send: addr=0x41b08
[  643.046350] qcom_rpmh DRV:apps_rsc TCS Busy, retrying RPMH message send: addr=0x41b08
[  643.046369] qcom_rpmh DRV:apps_rsc TCS Busy, retrying RPMH message send: addr=0x41b08
[  643.046388] qcom_rpmh DRV:apps_rsc TCS Busy, retrying RPMH message send: addr=0x41b08
[  643.046406] qcom_rpmh DRV:apps_rsc TCS Busy, retrying RPMH message send: addr=0x41b08
[  643.046425] qcom_rpmh DRV:apps_rsc TCS Busy, retrying RPMH message send: addr=0x41b08
[  643.046443] qcom_rpmh DRV:apps_rsc TCS Busy, retrying RPMH message send: addr=0x41b08
[  643.046461] qcom_rpmh DRV:apps_rsc TCS Busy, retrying RPMH message send: addr=0x41b08
[  643.046480] qcom_rpmh DRV:apps_rsc TCS Busy, retrying RPMH message send: addr=0x41b08
[  670.810479] ufshcd-qcom 1d84000.ufshc: ufshcd_ioctl: User buffer is NULL!
[  693.797692] qcom_rpmh DRV:apps_rsc TCS Busy, retrying RPMH message send: addr=0x40d00
[  700.336130] hrtimer: interrupt took 105417 ns
[  721.252079] ufshcd-qcom 1d84000.ufshc: ufshcd_ioctl: User buffer is NULL!
[  747.535072] CAM_ERR: CAM-MEM: cam_mem_mgr_create_debug_fs: 126 failed to create dentry
[  747.564180] CAM_INFO: CAM-HFI: cam_hfi_init: 878 Init IO1 : [0x10c00000 0xcf300000] IO2 [0xe0200000 0x1ed00000]
[  747.574301] CAM_INFO: CAM-ICP: cam_icp_mgr_hw_open: 3879 FW download done successfully
[  747.574309] CAM_INFO: CAM-ICP: cam_icp_mgr_process_dbg_buf: 2572 FW_DBG:CICP_FW_E : HFI  :QC_IMAGE_VERSION_STRING=CICP.FW.1.0-00079,OEM_IMAGE_VERSION_STRING=CRM,BUILD_TIME: Oct 17 2019 05:49:19,CACHE_ENABLED at icphostinterface.c:636 QC_IMAGE_VERSION_STRING=CICP.FW.1.0-00079 OEM_IMAGE_VERSION_STRING=CRM
[  747.574315] CAM_INFO: CAM-ICP: cam_icp_mgr_process_dbg_buf: 2572 FW_DBG:CICP_FW_E : HFI  :ELF variant: CACHE-ENABLED:T480:API_V2:USE_CDM_1_1: , API version: 0x2000049 at icphostinterface.c:637 QC_IMAGE_VERSION_STRING=CICP.FW.1.0-00079 OEM_IMAGE_VERSION_STRING=CRM
[  747.667487] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Linked as a consumer to regulator.60
[  747.667526] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Linked as a consumer to regulator.79
[  747.696358] CAM_INFO: CAM-SENSOR: cam_sensor_driver_cmd: 918 Probe success,slot:2,slave_addr:0x7a,sensor_id:0x3d
[  747.696481] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Dropping the link to regulator.79
[  747.700211] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Dropping the link to regulator.60
[  747.744388] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Linked as a consumer to regulator.60
[  747.744579] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Linked as a consumer to regulator.79
[  747.767225] CAM_INFO: CAM-SENSOR: cam_sensor_driver_cmd: 997 CAM_ACQUIRE_DEV Success, sensor_id:0x3d,sensor_slave_addr:0x7a
[  748.124228] CAM_ERR: CAM-ISP: cam_ife_hw_mgr_print_acquire_info: 710 Successfully acquire single IFE[6 -1] with [0 pix] [0 pd] [1 rdi] ports for ctx:0
[  748.124972] CAM_ERR: CAM-CRM: cam_req_mgr_cb_add_req: 2742 req 1 not found in in_q
[  748.125219] CAM_INFO: CAM-CSIPHY: cam_csiphy_core_cfg: 1137 START_DEV: CSIPHY_IDX: 2, Device_slot: 0, Datarate: 288000000, Settletime: 2800000000
[  748.133931] CAM_INFO: CAM-ISP: cam_vfe_bus_ver3_init_hw: 3659 Overriding clock gating at bus input
[  748.133934] CAM_INFO: CAM-ISP: cam_vfe_top_ver3_init_hw: 246 Disable clock gating at IFE top
[  748.134108] CAM_ERR: CAM-ISP: cam_ife_mgr_start_hw: 4510 ->Config HW, 00000000a3bb1d21
[  748.134411] CAM_INFO: CAM-SENSOR: cam_sensor_driver_cmd: 1089 CAM_START_DEV Success, sensor_id:0x3d,sensor_slave_addr:0x7a
[  748.146857] CAM_ERR: CAM-ISP: cam_ife_csid_irq: 4927 CSID:6 UNBOUNDED_FRAME
[  762.859114] qcom_rpmh DRV:apps_rsc TCS Busy, retrying RPMH message send: addr=0x41b08
[  762.859143] qcom_rpmh DRV:apps_rsc TCS Busy, retrying RPMH message send: addr=0x41b08
[  835.472101] perf: interrupt took too long (3965 > 3960), lowering kernel.perf_event_max_sample_rate to 50250

Further upon I tested with a Tracking Camera ov7251 attached, where I'm receiving data from the sensor and feed from sensors could be viewed upon the portal.

Here ToF is seated as earlier on J6 L port and Tracking ov7251 on J7 L port

[    5.505291] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Linked as a consumer to regulator.79
[    5.526921] CAM_INFO: CAM-SENSOR: cam_sensor_set_alt_id: 731 [Applying Alternative Address] : Sensor Index   : 391385268
[    5.526926] CAM_INFO: CAM-SENSOR: cam_sensor_set_alt_id: 732 [Applying Alternative Address] : Slave Address  : 0xE2
[    5.526930] CAM_INFO: CAM-SENSOR: cam_sensor_set_alt_id: 733 [Applying Alternative Address] : Slave sensor id: 0x7750
[    5.526934] CAM_INFO: CAM-SENSOR: cam_sensor_set_alt_id: 734 [Applying Alternative Address] : cci_client sid : 0x71
[    5.527197] CAM_INFO: CAM-SENSOR: cam_sensor_set_alt_id: 770 [Applying Alternative Address] : Probe : Alternate address updated
[    5.527203] CAM_INFO: CAM-SENSOR: cam_sensor_driver_cmd: 997 CAM_ACQUIRE_DEV Success, sensor_id:0x7750,sensor_slave_addr:0xe2
[    5.556310] CAM_ERR: CAM-ISP: cam_ife_hw_mgr_print_acquire_info: 710 Successfully acquire single IFE[6 -1] with [0 pix] [0 pd] [1 rdi] ports for ctx:0
[    5.557998] CAM_INFO: CAM-CSIPHY: cam_csiphy_core_cfg: 1137 START_DEV: CSIPHY_IDX: 2, Device_slot: 0, Datarate: 384000000, Settletime: 2800000000
[    5.567355] CAM_INFO: CAM-ISP: cam_vfe_bus_ver3_init_hw: 3659 Overriding clock gating at bus input
[    5.567362] CAM_INFO: CAM-ISP: cam_vfe_top_ver3_init_hw: 246 Disable clock gating at IFE top
[    5.567662] CAM_ERR: CAM-ISP: cam_ife_mgr_start_hw: 4510 ->Config HW, 00000000b07f5b5e
[    5.568263] CAM_INFO: CAM-SENSOR: cam_sensor_driver_cmd: 1089 CAM_START_DEV Success, sensor_id:0x7750,sensor_slave_addr:0xe2
[    5.710704] android_work: sent uevent USB_STATE=CONNECTED
[    5.720776] configfs-gadget gadget: high-speed config #1: c
[    5.721016] android_work: sent uevent USB_STATE=CONFIGURED
[    5.760593] CAM_ERR: CAM-ISP: cam_ife_hw_mgr_print_acquire_info: 710 Successfully acquire single IFE[5 -1] with [0 pix] [0 pd] [1 rdi] ports for ctx:1
[    5.761075] CAM_ERR: CAM-CRM: cam_req_mgr_cb_add_req: 2742 req 1 not found in in_q
[    5.761251] CAM_INFO: CAM-CSIPHY: cam_csiphy_core_cfg: 1137 START_DEV: CSIPHY_IDX: 0, Device_slot: 0, Datarate: 288000000, Settletime: 2800000000
[    5.769634] CAM_INFO: CAM-ISP: cam_vfe_bus_ver3_init_hw: 3659 Overriding clock gating at bus input
[    5.769638] CAM_INFO: CAM-ISP: cam_vfe_top_ver3_init_hw: 246 Disable clock gating at IFE top
[    5.769647] CAM_ERR: CAM-ISP: cam_ife_mgr_start_hw: 4510 ->Config HW, 00000000b07f5b5e
[    5.769921] CAM_INFO: CAM-SENSOR: cam_sensor_driver_cmd: 1089 CAM_START_DEV Success, sensor_id:0x3d,sensor_slave_addr:0x7a
[    5.783209] CAM_ERR: CAM-ISP: cam_ife_csid_irq: 4927 CSID:5 UNBOUNDED_FRAME
[    6.030729] spi_geni 98c000.spi: tx_fifo 16 rx_fifo 16 tx_width 32
[    6.215921] Started adbd. 
[    6.374976] IPv6: ADDRCONF(NETDEV_UP): wlan0: link is not ready
[    6.387472] IPv6: ADDRCONF(NETDEV_CHANGE): wlan0: link becomes ready
[    6.391291] QTI:Netlink Query to Kernel Success
[   10.090752] devfreq-qcom-fw 18590000.qcom,devfreq-l3:qcom,cdsp-cdsp-l3-lat: Successfully started CDSP L3 governor
[   11.435643] netlink: 'hostapd': attribute type 213 has an invalid length.
[   11.489516] RTW: assoc success
[   11.489998] RTW: set group key camid:0, addr:00:c0:ca:b3:a3:e9, kid:1, type:AES
[   28.000258] CAM_INFO: CAM-CRM: cam_req_mgr_process_flush_req: 2177 Last request id to flush is 115
[   28.000507] CAM_INFO: CAM-ISP: __cam_isp_ctx_flush_req_in_top_state: 3046 Last request id to flush is 115
[   28.000960] CAM_INFO: CAM-ISP: __cam_isp_ctx_flush_req_in_top_state: 3069 Stop HW complete. Reset HW next.
[   28.005985] CAM_INFO: CAM-SENSOR: cam_sensor_driver_cmd: 1117 CAM_STOP_DEV Success, sensor_id:0x3d,sensor_slave_addr:0x7a
[   28.006005] CAM_INFO: CAM-CSIPHY: cam_csiphy_core_cfg: 982 STOP_DEV: CSIPHY_IDX: 0, Device_slot: 0, Datarate: 288000000, Settletime: 2800000000
[   28.012879] CAM_INFO: CAM-ISP: cam_ife_mgr_release_hw: 4755 Release HW success ctx id: 1
[   28.014448] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Dropping the link to regulator.79
[   28.017762] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Dropping the link to regulator.60
[   28.018601] CAM_INFO: CAM-SENSOR: cam_sensor_driver_cmd: 1047 CAM_RELEASE_DEV Success, sensor_id:0x3d,sensor_slave_addr:0x7a
[   28.066082] CAM_INFO: CAM-CRM: cam_req_mgr_process_flush_req: 2177 Last request id to flush is 679
[   28.066153] CAM_INFO: CAM-ISP: __cam_isp_ctx_flush_req_in_top_state: 3046 Last request id to flush is 679
[   28.066283] CAM_INFO: CAM-ISP: __cam_isp_ctx_flush_req_in_top_state: 3069 Stop HW complete. Reset HW next.
[   28.071811] CAM_INFO: CAM-SENSOR: cam_sensor_driver_cmd: 1117 CAM_STOP_DEV Success, sensor_id:0x7750,sensor_slave_addr:0xe2
[   28.071831] CAM_INFO: CAM-CSIPHY: cam_csiphy_core_cfg: 982 STOP_DEV: CSIPHY_IDX: 2, Device_slot: 0, Datarate: 384000000, Settletime: 2800000000
[   28.078536] CAM_INFO: CAM-ISP: cam_ife_mgr_release_hw: 4755 Release HW success ctx id: 0
[   28.080413] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Dropping the link to regulator.79
[   28.083593] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Dropping the link to regulator.60
[   28.084953] CAM_INFO: CAM-SENSOR: cam_sensor_driver_cmd: 1047 CAM_RELEASE_DEV Success, sensor_id:0x7750,sensor_slave_addr:0xe2
[   28.092411] rpmh_rsc_send_data: 1 callbacks suppressed
[   28.092417] qcom_rpmh DRV:apps_rsc TCS Busy, retrying RPMH message send: addr=0x30080
[   28.102533] CAM_WARN: CAM-CRM: cam_req_mgr_close: 160 release invoked associated userspace process has died
[   28.173983] CAM_ERR: CAM-MEM: cam_mem_mgr_create_debug_fs: 126 failed to create dentry
[   28.208650] CAM_INFO: CAM-HFI: cam_hfi_init: 878 Init IO1 : [0x10c00000 0xcf300000] IO2 [0xe0200000 0x1ed00000]
[   28.218782] CAM_INFO: CAM-ICP: cam_icp_mgr_process_dbg_buf: 2572 FW_DBG:CICP_FW_E : HFI  :QC_IMAGE_VERSION_STRING=CICP.FW.1.0-00079,OEM_IMAGE_VERSION_STRING=CRM,BUILD_TIME: Oct 17 2019 05:49:19,CACHE_ENABLED at icphostinterface.c:636 QC_IMAGE_VERSION_STRING=CICP.FW.1.0-00079 OEM_IMAGE_VERSION_STRING=CRM
[   28.218793] CAM_INFO: CAM-ICP: cam_icp_mgr_process_dbg_buf: 2572 FW_DBG:CICP_FW_E : HFI  :ELF variant: CACHE-ENABLED:T480:API_V2:USE_CDM_1_1: , API version: 0x2000049 at icphostinterface.c:637 QC_IMAGE_VERSION_STRING=CICP.FW.1.0-00079 OEM_IMAGE_VERSION_STRING=CRM
[   28.218982] CAM_INFO: CAM-ICP: cam_icp_mgr_hw_open: 3879 FW download done successfully
[   28.385352] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Linked as a consumer to regulator.60
[   28.385413] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Linked as a consumer to regulator.79
[   28.410828] CAM_INFO: CAM-SENSOR: cam_sensor_driver_cmd: 918 Probe success,slot:0,slave_addr:0x7a,sensor_id:0x3d
[   28.410999] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Dropping the link to regulator.79
[   28.415312] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Dropping the link to regulator.60
[   28.448500] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Linked as a consumer to regulator.60
[   28.448594] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Linked as a consumer to regulator.79
[   28.471853] CAM_INFO: CAM-SENSOR: cam_sensor_set_alt_id: 731 [Applying Alternative Address] : Sensor Index   : 391385268
[   28.471863] CAM_INFO: CAM-SENSOR: cam_sensor_set_alt_id: 732 [Applying Alternative Address] : Slave Address  : 0xE2
[   28.471870] CAM_INFO: CAM-SENSOR: cam_sensor_set_alt_id: 733 [Applying Alternative Address] : Slave sensor id: 0x7750
[   28.471876] CAM_INFO: CAM-SENSOR: cam_sensor_set_alt_id: 734 [Applying Alternative Address] : cci_client sid : 0x71
[   28.472189] CAM_INFO: CAM-SENSOR: cam_sensor_set_alt_id: 770 [Applying Alternative Address] : Probe : Alternate address updated
[   28.472454] CAM_INFO: CAM-SENSOR: cam_sensor_driver_cmd: 918 Probe success,slot:2,slave_addr:0xe2,sensor_id:0x7750
[   28.472681] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Dropping the link to regulator.79
[   28.476332] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Dropping the link to regulator.60
[   28.537077] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Linked as a consumer to regulator.60
[   28.537166] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Linked as a consumer to regulator.79
[   28.559234] CAM_INFO: CAM-SENSOR: cam_sensor_driver_cmd: 997 CAM_ACQUIRE_DEV Success, sensor_id:0x3d,sensor_slave_addr:0x7a
[   28.681627] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Linked as a consumer to regulator.60
[   28.681728] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Linked as a consumer to regulator.79
[   28.702666] CAM_INFO: CAM-SENSOR: cam_sensor_set_alt_id: 731 [Applying Alternative Address] : Sensor Index   : 391385268
[   28.702672] CAM_INFO: CAM-SENSOR: cam_sensor_set_alt_id: 732 [Applying Alternative Address] : Slave Address  : 0xE2
[   28.702676] CAM_INFO: CAM-SENSOR: cam_sensor_set_alt_id: 733 [Applying Alternative Address] : Slave sensor id: 0x7750
[   28.702680] CAM_INFO: CAM-SENSOR: cam_sensor_set_alt_id: 734 [Applying Alternative Address] : cci_client sid : 0x71
[   28.702943] CAM_INFO: CAM-SENSOR: cam_sensor_set_alt_id: 770 [Applying Alternative Address] : Probe : Alternate address updated
[   28.702949] CAM_INFO: CAM-SENSOR: cam_sensor_driver_cmd: 997 CAM_ACQUIRE_DEV Success, sensor_id:0x7750,sensor_slave_addr:0xe2
[   28.737746] CAM_ERR: CAM-ISP: cam_ife_hw_mgr_print_acquire_info: 710 Successfully acquire single IFE[6 -1] with [0 pix] [0 pd] [1 rdi] ports for ctx:2
[   28.739627] CAM_INFO: CAM-CSIPHY: cam_csiphy_core_cfg: 1137 START_DEV: CSIPHY_IDX: 2, Device_slot: 0, Datarate: 384000000, Settletime: 2800000000
[   28.748886] CAM_INFO: CAM-ISP: cam_vfe_bus_ver3_init_hw: 3659 Overriding clock gating at bus input
[   28.748893] CAM_INFO: CAM-ISP: cam_vfe_top_ver3_init_hw: 246 Disable clock gating at IFE top
[   28.757430] CAM_ERR: CAM-ISP: cam_ife_mgr_start_hw: 4510 ->Config HW, 00000000b07f5b5e
[   28.757854] CAM_INFO: CAM-SENSOR: cam_sensor_driver_cmd: 1089 CAM_START_DEV Success, sensor_id:0x7750,sensor_slave_addr:0xe2
[   29.029065] CAM_ERR: CAM-ISP: cam_ife_hw_mgr_print_acquire_info: 710 Successfully acquire single IFE[5 -1] with [0 pix] [0 pd] [1 rdi] ports for ctx:3
[   29.029714] CAM_ERR: CAM-CRM: cam_req_mgr_cb_add_req: 2742 req 1 not found in in_q
[   29.030119] CAM_INFO: CAM-CSIPHY: cam_csiphy_core_cfg: 1137 START_DEV: CSIPHY_IDX: 0, Device_slot: 0, Datarate: 288000000, Settletime: 2800000000
[   29.039005] CAM_INFO: CAM-ISP: cam_vfe_bus_ver3_init_hw: 3659 Overriding clock gating at bus input
[   29.039012] CAM_INFO: CAM-ISP: cam_vfe_top_ver3_init_hw: 246 Disable clock gating at IFE top
[   29.039031] CAM_ERR: CAM-ISP: cam_ife_mgr_start_hw: 4510 ->Config HW, 00000000b07f5b5e
[   29.039547] CAM_INFO: CAM-SENSOR: cam_sensor_driver_cmd: 1089 CAM_START_DEV Success, sensor_id:0x3d,sensor_slave_addr:0x7a

Jetson Nano

@Alex-Kushleyev I have also attached the below output where ToF sensor fails sometimes during midflight, we tried reattaching the flex connectors to interposers and works fine for a while and shuts off. I tried to troubleshoot on the following note,

Case 1: Only Tracking enabled with ToF attached - I copied the required .bin files to /usr/lib/camera/. I restart the voxl-camera-server and the Tracking camera works fine on ground and midflight, without being frozen or feed being stalled from server.

Case 2: Only ToF enabled, with Tracking attached - I copied the required bin files to the destination. I restarted the voxl-camera-server, after a while data starts getting published. But the senosr doesn't always work. On a scale of comparison based on testing, the sensor stops publishing data 2 out of 5 run of tests.

Tried increasing the FPS of the sensor from 5 to 15, to check if would help. It helped improve mapping quality, but not well to solve the issue of losing feed.

Case 3: ToF + Tracking ov7251 attached and enabled - Configured the voxl-configure-cameras to configure 5. Then I restart the systemctl restart voxl-camera-server, the camera server restarts and feed is viewed in portal since cameras are publishing data. Sometimes upon restarting or running voxl-camera-server only, the cameras are detected and the server gets aborted abruptly or a message of gracefully exited appears.

Sometimes, the server goes of randomly on ground or mid-air and comes back again. On running dmesg -w on a terminal, I get the following output.

[  827.840180] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Linked as a consumer to regulator.58
[  827.842571] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Linked as a consumer to regulator.55
[  827.844131] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Linked as a consumer to regulator.60
[  827.844205] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Linked as a consumer to regulator.79
[  827.870672] CAM_INFO: CAM-SENSOR: cam_sensor_driver_cmd: 918 Probe success,slot:0,slave_addr:0x7a,sensor_id:0x3d
[  827.870946] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Dropping the link to regulator.79
[  827.874552] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Dropping the link to regulator.60
[  827.874719] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Dropping the link to regulator.55
[  827.874859] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Dropping the link to regulator.58
[  827.905837] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Linked as a consumer to regulator.59
[  827.908159] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Linked as a consumer to regulator.55
[  827.910035] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Linked as a consumer to regulator.60
[  827.910156] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Linked as a consumer to regulator.79
[  827.932850] CAM_INFO: CAM-SENSOR: cam_sensor_set_alt_id: 731 [Applying Alternative Address] : Sensor Index   : 1465270452
[  827.932859] CAM_INFO: CAM-SENSOR: cam_sensor_set_alt_id: 732 [Applying Alternative Address] : Slave Address  : 0xE2
[  827.932866] CAM_INFO: CAM-SENSOR: cam_sensor_set_alt_id: 733 [Applying Alternative Address] : Slave sensor id: 0x7750
[  827.932872] CAM_INFO: CAM-SENSOR: cam_sensor_set_alt_id: 734 [Applying Alternative Address] : cci_client sid : 0x71
[  827.933224] CAM_INFO: CAM-SENSOR: cam_sensor_set_alt_id: 770 [Applying Alternative Address] : Probe : Alternate address updated
[  827.933477] CAM_INFO: CAM-SENSOR: cam_sensor_driver_cmd: 918 Probe success,slot:2,slave_addr:0xe2,sensor_id:0x7750
[  827.933651] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Dropping the link to regulator.79
[  827.937888] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Dropping the link to regulator.60
[  827.938082] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Dropping the link to regulator.55
[  827.938221] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Dropping the link to regulator.59
[  828.000369] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Linked as a consumer to regulator.58
[  828.002330] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Linked as a consumer to regulator.55
[  828.004189] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Linked as a consumer to regulator.60
[  828.004269] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Linked as a consumer to regulator.79
[  828.028279] CAM_INFO: CAM-SENSOR: cam_sensor_driver_cmd: 997 CAM_ACQUIRE_DEV Success, sensor_id:0x3d,sensor_slave_addr:0x7a
[  828.478535] CAM_ERR: CAM-ISP: cam_ife_hw_mgr_print_acquire_info: 710 Successfully acquire single IFE[6 -1] with [0 pix] [0 pd] [1 rdi] ports for ctx:8
[  828.479181] CAM_ERR: CAM-CRM: cam_req_mgr_cb_add_req: 2742 req 1 not found in in_q
[  828.479457] CAM_INFO: CAM-CSIPHY: cam_csiphy_core_cfg: 1137 START_DEV: CSIPHY_IDX: 0, Device_slot: 0, Datarate: 288000000, Settletime: 2800000000
[  828.487975] CAM_INFO: CAM-ISP: cam_vfe_bus_ver3_init_hw: 3659 Overriding clock gating at bus input
[  828.487979] CAM_INFO: CAM-ISP: cam_vfe_top_ver3_init_hw: 246 Disable clock gating at IFE top
[  828.488207] CAM_ERR: CAM-ISP: cam_ife_mgr_start_hw: 4510 ->Config HW, 00000000851a209a
[  828.488561] CAM_INFO: CAM-SENSOR: cam_sensor_driver_cmd: 1089 CAM_START_DEV Success, sensor_id:0x3d,sensor_slave_addr:0x7a
[  841.529716] RTW: wlan0- hw port(0) mac_addr =00:c0:ca:b3:a3:e9
[  862.477578] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Dropping the link to regulator.79
[  862.481705] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Dropping the link to regulator.60
[  862.481785] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Dropping the link to regulator.55
[  862.481857] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Dropping the link to regulator.58
[  862.490614] CAM_ERR: CAM-SYNC: cam_sync_signal: 203 Error: accessing an uninitialized sync obj = 5
[  862.490620] CAM_ERR: CAM-SYNC: cam_sync_signal: 203 Error: accessing an uninitialized sync obj = 1
[  862.490623] CAM_ERR: CAM-SYNC: cam_sync_signal: 203 Error: accessing an uninitialized sync obj = 2
[  862.490626] CAM_ERR: CAM-SYNC: cam_sync_signal: 203 Error: accessing an uninitialized sync obj = 4
[  862.490630] CAM_ERR: CAM-SYNC: cam_sync_signal: 203 Error: accessing an uninitialized sync obj = 3
[  862.490634] CAM_ERR: CAM-ISP: __cam_isp_ctx_release_dev_in_top_state: 3789 releasing hw
[  862.490658] CAM_INFO: CAM-ISP: cam_ife_mgr_release_hw: 4755 Release HW success ctx id: 8
[  862.490757] CAM_WARN: CAM-CRM: cam_req_mgr_close: 160 release invoked associated userspace process has died
[  862.490838] CAM-REQ-MGR_UTIL cam_req_mgr_util_free_hdls:99 CAM_WARN: CAM-CRM: cam_req_mgr_util_free_hdls: 99 Dev handle = f10102 session_handle = 6e0200
[  862.493693] ion_buffer_destroy: 1 callbacks suppressed
[  862.493695] ION client likely missing a call to dma_buf_kunmap or dma_buf_vunmap
[  862.493706] ION client likely missing a call to dma_buf_kunmap or dma_buf_vunmap
[  862.493715] ION client likely missing a call to dma_buf_kunmap or dma_buf_vunmap
[  862.493803] ION client likely missing a call to dma_buf_kunmap or dma_buf_vunmap
[  862.493839] ION client likely missing a call to dma_buf_kunmap or dma_buf_vunmap
[  862.494021] ION client likely missing a call to dma_buf_kunmap or dma_buf_vunmap
[  862.494253] ION client likely missing a call to dma_buf_kunmap or dma_buf_vunmap
[  862.494262] ION client likely missing a call to dma_buf_kunmap or dma_buf_vunmap
[  862.494282] ION client likely missing a call to dma_buf_kunmap or dma_buf_vunmap
[  862.494288] ION client likely missing a call to dma_buf_kunmap or dma_buf_vunmap
[  863.270637] RTW: wlan0- hw port(0) mac_addr =00:c0:ca:b3:a3:e9
[  870.888214] qcom_rpmh DRV:apps_rsc TCS Busy, retrying RPMH message send: addr=0x41b08
[  870.888244] qcom_rpmh DRV:apps_rsc TCS Busy, retrying RPMH message send: addr=0x41b08
[  875.098507] CAM_ERR: CAM-MEM: cam_mem_mgr_create_debug_fs: 126 failed to create dentry
[  875.124437] CAM_INFO: CAM-HFI: cam_hfi_init: 878 Init IO1 : [0x10c00000 0xcf300000] IO2 [0xe0200000 0x1ed00000]
[  875.134626] CAM_INFO: CAM-ICP: cam_icp_mgr_process_dbg_buf: 2572 FW_DBG:CICP_FW_E : HFI  :QC_IMAGE_VERSION_STRING=CICP.FW.1.0-00079,OEM_IMAGE_VERSION_STRING=CRM,BUILD_TIME: Oct 17 2019 05:49:19,CACHE_ENABLED at icphostinterface.c:636 QC_IMAGE_VERSION_STRING=CICP.FW.1.0-00079 OEM_IMAGE_VERSION_STRING=CRM
[  875.134631] CAM_INFO: CAM-ICP: cam_icp_mgr_process_dbg_buf: 2572 FW_DBG:CICP_FW_E : HFI  :ELF variant: CACHE-ENABLED:T480:API_V2:USE_CDM_1_1: , API version: 0x2000049 at icphostinterface.c:637 QC_IMAGE_VERSION_STRING=CICP.FW.1.0-00079 OEM_IMAGE_VERSION_STRING=CRM
[  875.134702] CAM_INFO: CAM-ICP: cam_icp_mgr_hw_open: 3879 FW download done successfully
[  875.357708] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Linked as a consumer to regulator.58
[  875.359880] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Linked as a consumer to regulator.55
[  875.362038] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Linked as a consumer to regulator.60
[  875.362116] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Linked as a consumer to regulator.79
[  875.387690] CAM_INFO: CAM-SENSOR: cam_sensor_driver_cmd: 918 Probe success,slot:0,slave_addr:0x7a,sensor_id:0x3d
[  875.387887] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Dropping the link to regulator.79
[  875.390781] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Dropping the link to regulator.60
[  875.390890] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Dropping the link to regulator.55
[  875.390989] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Dropping the link to regulator.58
[  875.410081] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Linked as a consumer to regulator.59
[  875.412357] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Linked as a consumer to regulator.55
[  875.414556] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Linked as a consumer to regulator.60
[  875.414677] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Linked as a consumer to regulator.79
[  875.438443] CAM_INFO: CAM-SENSOR: cam_sensor_set_alt_id: 731 [Applying Alternative Address] : Sensor Index   : 1465270452
[  875.438452] CAM_INFO: CAM-SENSOR: cam_sensor_set_alt_id: 732 [Applying Alternative Address] : Slave Address  : 0xE2
[  875.438459] CAM_INFO: CAM-SENSOR: cam_sensor_set_alt_id: 733 [Applying Alternative Address] : Slave sensor id: 0x7750
[  875.438464] CAM_INFO: CAM-SENSOR: cam_sensor_set_alt_id: 734 [Applying Alternative Address] : cci_client sid : 0x71
[  875.438825] CAM_INFO: CAM-SENSOR: cam_sensor_set_alt_id: 770 [Applying Alternative Address] : Probe : Alternate address updated
[  875.439083] CAM_INFO: CAM-SENSOR: cam_sensor_driver_cmd: 918 Probe success,slot:2,slave_addr:0xe2,sensor_id:0x7750
[  875.439587] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Dropping the link to regulator.79
[  875.443944] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Dropping the link to regulator.60
[  875.444426] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Dropping the link to regulator.55
[  875.444571] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Dropping the link to regulator.59
[  875.511578] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Linked as a consumer to regulator.58
[  875.513201] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Linked as a consumer to regulator.55
[  875.515282] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Linked as a consumer to regulator.60
[  875.515350] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Linked as a consumer to regulator.79
[  875.538928] CAM_INFO: CAM-SENSOR: cam_sensor_driver_cmd: 997 CAM_ACQUIRE_DEV Success, sensor_id:0x3d,sensor_slave_addr:0x7a
[  876.005639] CAM_ERR: CAM-ISP: cam_ife_hw_mgr_print_acquire_info: 710 Successfully acquire single IFE[6 -1] with [0 pix] [0 pd] [1 rdi] ports for ctx:9
[  876.006492] CAM_ERR: CAM-CRM: cam_req_mgr_cb_add_req: 2742 req 1 not found in in_q
[  876.006720] CAM_INFO: CAM-CSIPHY: cam_csiphy_core_cfg: 1137 START_DEV: CSIPHY_IDX: 0, Device_slot: 0, Datarate: 288000000, Settletime: 2800000000
[  876.015994] CAM_INFO: CAM-ISP: cam_vfe_bus_ver3_init_hw: 3659 Overriding clock gating at bus input
[  876.015997] CAM_INFO: CAM-ISP: cam_vfe_top_ver3_init_hw: 246 Disable clock gating at IFE top
[  876.016401] CAM_ERR: CAM-ISP: cam_ife_mgr_start_hw: 4510 ->Config HW, 00000000851a209a
[  876.016793] CAM_INFO: CAM-SENSOR: cam_sensor_driver_cmd: 1089 CAM_START_DEV Success, sensor_id:0x3d,sensor_slave_addr:0x7a
[  876.019097] CAM_INFO: CAM-CRM: __cam_req_mgr_find_dev_name: 209 Skip Frame: req: 2 not ready on link: 0xa10104 for pd: 2 dev: cam-sensor open_req count: 4
[  880.848318] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Dropping the link to regulator.79
[  880.852399] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Dropping the link to regulator.60
[  880.852461] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Dropping the link to regulator.55
[  880.852506] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Dropping the link to regulator.58
[  880.865126] CAM_ERR: CAM-SYNC: cam_sync_signal: 203 Error: accessing an uninitialized sync obj = 2
[  880.865140] CAM_ERR: CAM-SYNC: cam_sync_signal: 203 Error: accessing an uninitialized sync obj = 3
[  880.865150] CAM_ERR: CAM-SYNC: cam_sync_signal: 203 Error: accessing an uninitialized sync obj = 4
[  880.865159] CAM_ERR: CAM-SYNC: cam_sync_signal: 203 Error: accessing an uninitialized sync obj = 5
[  880.865168] CAM_ERR: CAM-SYNC: cam_sync_signal: 203 Error: accessing an uninitialized sync obj = 1
[  880.865179] CAM_ERR: CAM-ISP: __cam_isp_ctx_release_dev_in_top_state: 3789 releasing hw
[  880.865285] CAM_INFO: CAM-ISP: cam_ife_mgr_release_hw: 4755 Release HW success ctx id: 9
[  880.865409] CAM_WARN: CAM-CRM: cam_req_mgr_close: 160 release invoked associated userspace process has died
[  880.865699] CAM-REQ-MGR_UTIL cam_req_mgr_util_free_hdls:99 CAM_WARN: CAM-CRM: cam_req_mgr_util_free_hdls: 99 Dev handle = df0102 session_handle = a60200
[  880.871851] ion_buffer_destroy: 1 callbacks suppressed
[  880.871858] ION client likely missing a call to dma_buf_kunmap or dma_buf_vunmap
[  880.871999] ION client likely missing a call to dma_buf_kunmap or dma_buf_vunmap
[  880.873664] ION client likely missing a call to dma_buf_kunmap or dma_buf_vunmap
[  880.873691] ION client likely missing a call to dma_buf_kunmap or dma_buf_vunmap
[  880.873866] ION client likely missing a call to dma_buf_kunmap or dma_buf_vunmap
[  880.873889] ION client likely missing a call to dma_buf_kunmap or dma_buf_vunmap
[  880.873988] ION client likely missing a call to dma_buf_kunmap or dma_buf_vunmap
[  880.874057] ION client likely missing a call to dma_buf_kunmap or dma_buf_vunmap
[  880.879726] ION client likely missing a call to dma_buf_kunmap or dma_buf_vunmap
[  880.879944] ION client likely missing a call to dma_buf_kunmap or dma_buf_vunmap
[  885.149685] RTW: wlan0- hw port(0) mac_addr =00:c0:ca:b3:a3:e9
[  890.823665] qcom_rpmh DRV:apps_rsc TCS Busy, retrying RPMH message send: addr=0x40d00
[  906.997270] RTW: wlan0- hw port(0) mac_addr =00:c0:ca:b3:a3:e9
[  920.855588] CAM_ERR: CAM-MEM: cam_mem_mgr_create_debug_fs: 126 failed to create dentry
[  920.884159] CAM_INFO: CAM-HFI: cam_hfi_init: 878 Init IO1 : [0x10c00000 0xcf300000] IO2 [0xe0200000 0x1ed00000]
[  920.894375] CAM_INFO: CAM-ICP: cam_icp_mgr_hw_open: 3879 FW download done successfully
[  920.894383] CAM_INFO: CAM-ICP: cam_icp_mgr_process_dbg_buf: 2572 FW_DBG:CICP_FW_E : HFI  :QC_IMAGE_VERSION_STRING=CICP.FW.1.0-00079,OEM_IMAGE_VERSION_STRING=CRM,BUILD_TIME: Oct 17 2019 05:49:19,CACHE_ENABLED at icphostinterface.c:636 QC_IMAGE_VERSION_STRING=CICP.FW.1.0-00079 OEM_IMAGE_VERSION_STRING=CRM
[  920.894389] CAM_INFO: CAM-ICP: cam_icp_mgr_process_dbg_buf: 2572 FW_DBG:CICP_FW_E : HFI  :ELF variant: CACHE-ENABLED:T480:API_V2:USE_CDM_1_1: , API version: 0x2000049 at icphostinterface.c:637 QC_IMAGE_VERSION_STRING=CICP.FW.1.0-00079 OEM_IMAGE_VERSION_STRING=CRM
[  921.122596] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Linked as a consumer to regulator.58
[  921.124189] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Linked as a consumer to regulator.55
[  921.126333] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Linked as a consumer to regulator.60
[  921.126411] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Linked as a consumer to regulator.79
[  921.153915] CAM_INFO: CAM-SENSOR: cam_sensor_driver_cmd: 918 Probe success,slot:0,slave_addr:0x7a,sensor_id:0x3d
[  921.154186] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Dropping the link to regulator.79
[  921.159914] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Dropping the link to regulator.60
[  921.160560] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Dropping the link to regulator.55
[  921.160703] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Dropping the link to regulator.58
[  921.188632] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Linked as a consumer to regulator.59
[  921.191511] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Linked as a consumer to regulator.55
[  921.193717] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Linked as a consumer to regulator.60
[  921.193854] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Linked as a consumer to regulator.79
[  921.218411] CAM_INFO: CAM-SENSOR: cam_sensor_set_alt_id: 731 [Applying Alternative Address] : Sensor Index   : 1465270452
[  921.218420] CAM_INFO: CAM-SENSOR: cam_sensor_set_alt_id: 732 [Applying Alternative Address] : Slave Address  : 0xE2
[  921.218427] CAM_INFO: CAM-SENSOR: cam_sensor_set_alt_id: 733 [Applying Alternative Address] : Slave sensor id: 0x7750
[  921.218432] CAM_INFO: CAM-SENSOR: cam_sensor_set_alt_id: 734 [Applying Alternative Address] : cci_client sid : 0x71
[  921.218777] CAM_INFO: CAM-SENSOR: cam_sensor_set_alt_id: 770 [Applying Alternative Address] : Probe : Alternate address updated
[  921.219067] CAM_INFO: CAM-SENSOR: cam_sensor_driver_cmd: 918 Probe success,slot:2,slave_addr:0xe2,sensor_id:0x7750
[  921.219268] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Dropping the link to regulator.79
[  921.223412] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Dropping the link to regulator.60
[  921.223568] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Dropping the link to regulator.55
[  921.223668] qcom,camera ac50000.qcom,cci:qcom,cam-sensor2: Dropping the link to regulator.59
[  921.284538] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Linked as a consumer to regulator.58
[  921.285965] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Linked as a consumer to regulator.55
[  921.288211] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Linked as a consumer to regulator.60
[  921.288298] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Linked as a consumer to regulator.79
[  921.311468] CAM_INFO: CAM-SENSOR: cam_sensor_driver_cmd: 997 CAM_ACQUIRE_DEV Success, sensor_id:0x3d,sensor_slave_addr:0x7a
[  921.859799] CAM_ERR: CAM-ISP: cam_ife_hw_mgr_print_acquire_info: 710 Successfully acquire single IFE[6 -1] with [0 pix] [0 pd] [1 rdi] ports for ctx:1
[  921.861359] CAM_ERR: CAM-CRM: cam_req_mgr_cb_add_req: 2742 req 1 not found in in_q
[  921.861782] CAM_INFO: CAM-CSIPHY: cam_csiphy_core_cfg: 1137 START_DEV: CSIPHY_IDX: 0, Device_slot: 0, Datarate: 288000000, Settletime: 2800000000
[  921.872367] CAM_INFO: CAM-ISP: cam_vfe_bus_ver3_init_hw: 3659 Overriding clock gating at bus input
[  921.872375] CAM_INFO: CAM-ISP: cam_vfe_top_ver3_init_hw: 246 Disable clock gating at IFE top
[  921.872758] CAM_ERR: CAM-ISP: cam_ife_mgr_start_hw: 4510 ->Config HW, 00000000851a209a
[  921.873590] CAM_INFO: CAM-SENSOR: cam_sensor_driver_cmd: 1089 CAM_START_DEV Success, sensor_id:0x3d,sensor_slave_addr:0x7a
[  926.700983] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Dropping the link to regulator.79
[  926.704609] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Dropping the link to regulator.60
[  926.704746] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Dropping the link to regulator.55
[  926.704858] qcom,camera ac4f000.qcom,cci:qcom,cam-sensor0: Dropping the link to regulator.58
[  926.716260] CAM_ERR: CAM-SYNC: cam_sync_signal: 203 Error: accessing an uninitialized sync obj = 2
[  926.716270] CAM_ERR: CAM-SYNC: cam_sync_signal: 203 Error: accessing an uninitialized sync obj = 3
[  926.716278] CAM_ERR: CAM-SYNC: cam_sync_signal: 203 Error: accessing an uninitialized sync obj = 4
[  926.716286] CAM_ERR: CAM-SYNC: cam_sync_signal: 203 Error: accessing an uninitialized sync obj = 5
[  926.716293] CAM_ERR: CAM-SYNC: cam_sync_signal: 203 Error: accessing an uninitialized sync obj = 1
[  926.716302] CAM_ERR: CAM-ISP: __cam_isp_ctx_release_dev_in_top_state: 3789 releasing hw
[  926.716346] CAM_INFO: CAM-ISP: cam_ife_mgr_release_hw: 4755 Release HW success ctx id: 1
[  926.716439] CAM_WARN: CAM-CRM: cam_req_mgr_close: 160 release invoked associated userspace process has died
[  926.716536] CAM-REQ-MGR_UTIL cam_req_mgr_util_free_hdls:99 CAM_WARN: CAM-CRM: cam_req_mgr_util_free_hdls: 99 Dev handle = bb0102 session_handle = a10200
[  926.723046] ion_buffer_destroy: 1 callbacks suppressed
[  926.723050] ION client likely missing a call to dma_buf_kunmap or dma_buf_vunmap
[  926.723184] ION client likely missing a call to dma_buf_kunmap or dma_buf_vunmap
[  926.723315] ION client likely missing a call to dma_buf_kunmap or dma_buf_vunmap
[  926.723534] ION client likely missing a call to dma_buf_kunmap or dma_buf_vunmap
[  926.728238] ION client likely missing a call to dma_buf_kunmap or dma_buf_vunmap
[  926.728261] ION client likely missing a call to dma_buf_kunmap or dma_buf_vunmap
[  926.728311] ION client likely missing a call to dma_buf_kunmap or dma_buf_vunmap
[  926.728332] ION client likely missing a call to dma_buf_kunmap or dma_buf_vunmap
[  926.728425] ION client likely missing a call to dma_buf_kunmap or dma_buf_vunmap
[  926.728641] ION client likely missing a call to dma_buf_kunmap or dma_buf_vunmap
[  928.697163] RTW: wlan0- hw port(0) mac_addr =00:c0:ca:b3:a3:e9

Sometimes, when we tweak the cables slightly or power the system after a while, the camera server including the ToF and tracking work fine, even though CAM ERR message appear on terminal line system call messages.

Through these I'm not able to actually conclude if it's a hardware issue or software issue in the voxl-camera-server while running multiple camera sensors simultanelously through the server.

Would be great if you could help out.

Jetson Nano

@Alex-Kushleyev Is there something we could do to tweak the way the data is being published through the voxl-camera-server pipeline?

Jetson Nano

@Alex-Kushleyev could you look into?

Alex Kushleyev

Hi @Jetson-Nano ,

Thank you for sending the logs. Unfortunately, I do not see anything specific that would indicate an error in the last few logs.

Do you have more than one VOXL2 you could use for testing? also, do you have another TOF sensor you could use? It seems the issue is tied to the TOF sensor but it is not clear whether it is on the VOXL2 side (maybe connector issue) or the TOF sensor side.

What you are describing is not normal behavior - the camera server from any recent official release should be pretty stable when used with these standard configurations (tracking camera, TOF, etc).

Since the tracking camera by itself seems to be working fine, please focus on testing TOF only (you can disconnect the tracking camera) and make sure you can get that to work reliably, testing on a different VOXL2 or a different TOF module if possible (to rule out one of the two being an issue)

Also, which SDK are you using?

Alex

Jetson Nano

@Alex-Kushleyev Thanks for reverting. Yes, I have another VOXL2 board. I'll test and share the results. I'm currently testing with SDKs 1.1.1 and 1.2 releases.

Jetson Nano

@Alex-Kushleyev I tested the ToF with another VOXL2 board I have. Modified the camera config as only for ToF. On ground tests, the camera server seems to work fine. On terminal system logger publishes similar type of CAM_INFO, CAM_ERR dmesg -w messages as earlier.

We used the same interposer as used in earlier VOXL2 compute.