Hires (M0026) Issue

gauravshukla914

Hello, we have a VOXL flight deck with system-image 3.3.0. We have a hires camera (M0026). We are having issues getting the data from the hires camera.

When we do voxl-inspect-cam hires_preview, it gives data for 2-3 seconds and then stops the data, and then we see "server disconnected" on the screen.

size(bytes)| height | width |exposure(ms)| gain | frame id |latency(ms)|Framerate(hz)| format
| 460800 | 480 | 640 | 33.3 | 126 | 16 | 181.0 | 26.9 | NV21^C

I also run roslaunch voxl_mpa_to_ros voxl_mpa_to_ros.launch and then echo the topic by "rostopic echo /hires_preview" - I get the data on the screen and then it stops.

The weird part is, just a day before I checked the output from the camera data and I was getting the data perfectly. This morning, it doesn't seem to work and nothing's been changed or touched in between.

Also, this issue is with MIPI extension cable only. When I connect the sensor (M00026) directly to the board, I get the data continously and also verified image in rqt viewer. When I connect the extension cable, then I get this issue. I already replaced the extension cable as I had one spare, and I have been extrememly careful with the extension cable - so I'm sure the cable is not damaged since its a new cable and we are only doing testing right now (not flying the drone).

After running roslaunch voxl_mpa_to_ros voxl_mpa_to_ros.launch - As soon as I do, rostopic echo /hires/image_raw, this is what happens.

MPA to ROS app is now running

Found pipe for interface: tracking, now advertising
Found pipe for interface: hires_preview, now advertising
Found pipe for interface: qvio_overlay, now advertising
Found pipe for interface: dfs_disparity, now advertising
Did not find pipe for interface: tflite,
interface will be idle until its pipe appears
Did not find pipe for interface: stereo,
interface will be idle until its pipe appears
Did not find pipe for interface: tof,
interface will be idle until its pipe appears
Found pipe for interface: imu0, now advertising
Found pipe for interface: imu1, now advertising
Found pipe for interface: qvio, now advertising
Found pipe for interface: dfs_point_cloud, now advertising
Interface hires_preview now publishing
Interface: hires_preview's data pipe disconnected, closing until it returns
Interface: tracking's data pipe disconnected, closing until it returns

@modalab @Alex-Gardner @Eric-Katzfey @Matt-Turi

Chad Sweet

Are you sure you are lining up pin 1s on each cable so that it is connected properly? Most likely issue is that you have not made the connections properly

Vinny

Also make sure you are only using 1 extension cable, and not cascading them for longer lengths. MIPI CSI cannot operate at indefinite lengths, so if you try to cascade two cables, the interfaces will likely fail for HIRES.
Thanks!

Vinny

Just to also confirm, this cable may be new, but are you sure it is not creased or folded?
We should clarify the bend radius is limited to about 5mm. We are actively updating our cables section of the website to improve this info.
Do you use the M0036 (150mm extension) or the M0074 (60mm extension)?
I'm curious if you are using the M0036 and then try to use M0074 if your issue goes away. If so, it is indicative of a length issue with this HIRES config.

gauravshukla914

Hi @Chad-Sweet the connections are correct. When I put the other way round, I can't even log into VOXL through ssh.

Hi @Vinny No I'm not cascading two cables on to one another. Its just one extension cable. The cable is not folded or anything. I have attached the current connection image.
I'm using the M0036 cable (14.5 cm one). We also have a offshore team in India who are using the same setup, and its been working for them without any issues since months - So I'm sure length cannot be an issue with HIRES config.
IMG_9151(1).jpg

gauravshukla914

These are the logs I get when I run journalctl -u voxl-camera-server

-- Logs begin at Thu 1970-01-01 00:00:04 UTC, end at Fri 2022-08-26 15:41:19 UTC. --
Jan 01 00:00:06 apq8096 systemd[1]: Started voxl-camera-server.
-- Logs begin at Thu 1970-01-01 00:00:04 UTC, end at Fri 2022-08-26 15:43:09 UTC. --
Jan 01 00:00:06 apq8096 systemd[1]: Started voxl-camera-server.
Jan 01 00:00:08 apq8096 bash[1959]: ERROR: Camera module not opened, attempting again
Jan 01 00:00:08 apq8096 bash[1959]: ERROR: Camera module not opened, 9 attempts remaining
Jan 01 00:00:10 apq8096 bash[1959]: ------ voxl-camera-server: Camera server is now running
Aug 26 15:43:07 apq8096 bash[1959]: voxl-camera-server ERROR: Error sending request 7958, ErrorCode:
-19
Aug 26 15:43:07 apq8096 bash[1959]: voxl-camera-server FATAL: Recieved Fatal error from camera: trac
king
Aug 26 15:43:07 apq8096 bash[1959]: Camera server will be stopped
Aug 26 15:43:07 apq8096 bash[1959]: ------ voxl-camera-server WARNING: Thread: tracking result threa
d recieved ESTOP
Aug 26 15:43:07 apq8096 bash[1959]: ------ voxl-camera-server WARNING: Thread: hires result thread r
ecieved ESTOP
Aug 26 15:43:08 apq8096 bash[1959]: ------ voxl-camera-server INFO: Camera server is now stopping
Aug 26 15:43:08 apq8096 bash[1959]: There is a chance that it may segfault here, this is a mmqcamera
bug, ignore it
Aug 26 15:43:08 apq8096 bash[1959]: ------ voxl-camera-server INFO: Stopping tracking camera
Aug 26 15:43:09 apq8096 bash[1959]: ------ voxl-camera-server INFO: tracking camera stopped successf
ully
Aug 26 15:43:09 apq8096 bash[1959]: ------ voxl-camera-server INFO: Stopping hires camera
Aug 26 15:43:09 apq8096 bash[1959]: ------ voxl-camera-server INFO: hires camera stopped successfull
y
Aug 26 15:43:09 apq8096 bash[1959]: ------ voxl-camera-server INFO: Camera server exited gracefully

Vinny

@gauravshukla914
Hi Gaurav,
I've asked our Product Manager to reach out to offer a free replacement M0036 cable.
Should you continue to have issues after receipt of the replacement cable, we might need to investigate further on the Voxl side.
Hope this helps!
Thanks!

Hey @gauravshukla914 thank you for your patience, I am a Product Manager at ModalAI. We wanted to let you know that we have sent a replacement cable that should arrive in the next day or two.

Thank you,
ModalAI

gauravshukla914

Hi @modalab I received one more email with product has been cancelled notice. Is the replacement cancelled?

gauravshukla914

Hi @Vinny @modalab curious to know whats the update on here. Is a cable sent out already?

gauravshukla914

Hi @Vinny , the problem still exists even after trying with a new cable.

Hey @Gaurav-Shukla we are sending out a replacement cable. Theres was a mixup internally and your order got cancelled. Would you please ship us the cable that is having issues via our RMA process. Your RMA number will be the order number for the invoice you receive. Please let us know when you receive and test the replacement.

Thank you,
ModalAI

gauravshukla914

Hi @modalab Thanks for your message. I have received one new cable from my previous communication with ModalAI (Since I had ordered 2 cables at the first place). At this point, I'm kinda convinced its not a cable issue. I have replaced cable thrice now, but it's still the same issue. What's even more weird is that, I have a “VOXL only” board with me that I sometimes use if for debugging purposes, and it's working great with this board. I’m able to get the data and also see image stream on rqt. So, this confirms that the cable is not faulty. I also compared the system images of the Voxl flight deck and just the voxl board and they are the same as well i.e 3.3.0. We also have an offshore team working in India who are using same setup (VOXL flight deck + extension cable + M0026 sensor ).

I feel this requires deeper investigation on the voxl side of my board as mentioned by @Vinny . As a matter of fact, even the voxl flight deck that I have is a new one. I'd like to have screen share session and look at this issue together and hopefully we might find a solution there. I already sent an email yesterday regarding the same.

Hey @gauravshukla914 will you please send in the VOXL Flight deck via our RMA Process and we will provide a report on our findings.

Thank you,
ModalAI

gauravshukla914

@modalab Okay, but I'm a bit confused here. The Voxl flight deck plus the extension cable plus the hires sensor (M0026) is all new(less than 3 months old). So in the RMA process, do I just need to complete the form and email it and just send the product right? without adding the diagnostic fee in the cart.

Vinny

@gauravshukla914 Hi Gaurav, sorry for your troubles on this.
It is entirely possible that even new products get damaged, especially by an ESD event.
We'll work with you to investigate and help resolve this.
Thanks for working with @modalab to send this back, and I'm sure I'll be one of the ones reviewing the physical hardware so we can report back as soon as we conclude anything.
Thanks!

gauravshukla914

Thanks @Vinny for your response. But do we have to pay the 100$ amount for diagnostic? Since this is under a 3 month product, I believe I just need to fill up the RMA document and just send the board back, right?

gauravshukla914

Hi @Vinny @modalab could you confirm this RMA process would be without any cost? Please confirm so that I can begin this immediately.

modaltb

Please see email thread in progress, if possible let's use one channel for communications.