voxl-camera-server crashes and QVIO says CAM_MISSING_STALLED

Aaky

Hello community,
I am using VOXL2 with SDK 0.9.5 and platform image 1.5.5. My voxl-camera-server version is 1.3.5 at present. We are building custom UAV with VOXL2. I am observing weird issue with voxl-camera-server and QVIO. During my VIO based flight on PX4 version 1.12.31 I see QVIO crashing midflight giving error CAM_MISSING_STALLED. When this error occurs, failsafe is trigged on PX4 and UAV crashes. I saw whenever I get this error "CAM_MISSING_STALLED" on "voxl-inspect-qvio", voxl-camera-server has crashed so I checked the "voxl-inspect-cam -a" for all camera streaming output and I saw tracking camera stream had died since voxl-camera-server had also died. Now I tried "systemctl restart voxl-camera-server" but it always get's some segmentation fault and dosent allow me to restart the service successfully. If I perform hard reset for VOXL2 then camera server again starts correctly. I have observed on "voxl-inspect-cpu" my CPU temperature is around 95 degrees on all 8 cores when this above problem occurs but my cpu utilization was fairly low around 42% average of all 8 cores together, also RAM utilization was low. I do not have fan installed on top of CPU but I do have fan acting as exhaust to pull out all hot air from UAV's avoinics compartment which has voxl2 installed.

Can there be any relation of high CPU temperature with voxl-camera-server crashing and QVIO giving CAM_MISSING_STALLED error?
How can we restart camera-server even if it crashes without hard reboot and ensure qvio server never dies due to missing camera streams?

tom

@Aaky It's possible that there is a hardware issue with the tracking sensor or flex cable that is causing it to be inconsistent

Aaky

@tom Thank you for the response.

We verified the connection of flex cable with tracking camera and everything seems to be intact visually.

Does Linux kernel on voxl2 kill any process on high CPU temperature? ( I have also worked on voxl1 and I remember voxl1 stops running cores if cpu temperature goes high). The reason why I am asking this is I saw camera-server crashing always when cpu temperature is high, where voxl-inspect-cpu shows 95 degrees on all cores with CPU overheat warning.

tom

@Aaky Yes, actually, I had missed that comment in your original post. Cores will start to be throttled at around 90 C.

What does your hardware setup look like? We recommend blowing a fan on the VOXL2 when doing desktop development as you aren't receiving the same
cooling factors as you would when in flight

Aaky

@tom Thank you for the info.

We have custom drone designed with Carbon fiber chamber inside which voxl2 resides over a dampner platform and we have camera bay towards front of the UAV which has CSI connection with VOXL2. We do not have fan over VOXL2 but we have kept fan towards backside of this carbon fiber chamber to act as exhaust but I guess it isnt working out well to cooldown CPU.