Posts made by LucaVertiq

LucaVertiq

@James-Strawson just checking, are you interested in this at all?

LucaVertiq

@Alex-Kushleyev said in Voxl 2 mini overheating and rebooting:

@LucaVertiq , did you try disconnecting EVERYTHING from the board and seeing if it boots? Just bare board with power module.

Alex

Yes, the full boot cycles were observed with just the power module and the voxl. We confirmed the voltage at the voxl was 3.8V and not dropping much below that, so no dead short.

@Vinny said in Voxl 2 mini overheating and rebooting:

Hi @LucaVertiq
To me, it seems as if the board went into a power fail condition. Was this in any sort of crash or was it dropped?
LED D2 coming on, then going Off repeatedly is an indication of a continuous reboot cycle (coinciding with your readings of 0.4A, 0A) but there is likely some hardware fault detected by the system preventing it from booting fully.

I had actually just disassembled the vehicle to connect with ADB as we keep the ADB port inaccessible. I had ADB'd in to do some things, then when I reassembled it this began happening. It's quite possible I knocked something loose, or possibly some ESD event.

I've been too busy flying vehicles to get to test the unbricking procedure, but Monday I will.

LucaVertiq

@Moderator Thanks! I'll give that a shot. Hopefully we can salvage it.

LucaVertiq

@Moderator I have a VOXL 2 mini that seems to be constantly rebooting and power cycling based on the LEDs.

When I power it on with only the power module plugged in and powered with 12V D1 lights up, but D2 does not. DS2 flashes at a high rate or alternately stays solid. The power supply shows that the voltage is staying high, but the current cycles between around 0.4A and 0A with the DS2 LED also turning off. We noticed that the silver SOC cover was getting hot so held it in front of a fan and it no longer cycled, just held ~0.4A. We measured the voltage arriving to the VOXL 2 mini from the power module and confirmed that it is 3.8v and doesn't dip when powered on. The USB 5V D2 LED doesn't seem to come on and when plugged into a computer there is no indication that a USB device has been plugged in. Holding the ADB button while booting and then plugging it into the computer also does nothing.

Before this the voxl has worked and nothing has really changed in the setup, but it does seem like something has broken. We are wondering if there is a way for us to fix it somehow. Unfortunately it just has the standard SDK installed, so I don't believe the debug port is enabled so I can't check how far into boot it's getting.

Do you have any ideas on things to check?

Thanks

LucaVertiq

Hello @Moderator

I've been having issues with voxl-px4 taking up 120-130% "CPU Usage" in inspect services and cpu monitor showing overheat warnings. I thought it may have been related to my changes so I reverted to standard voxl-px4, but it still appears to happen. I'm on suite 1.3.4~beta2 according to the MOTD on login.

Any thoughts?

Here's my voxl-version output:

voxl2-mini:~$ voxl-version
--------------------------------------------------------------------------------
system-image: 1.7.8-M0104-14.1a-perf
kernel:       #1 SMP PREEMPT Sat May 18 03:34:36 UTC 2024 4.19.125
--------------------------------------------------------------------------------
hw platform:  M0104
mach.var:     2.0.0
--------------------------------------------------------------------------------
voxl-suite:   1.3.4~beta2
--------------------------------------------------------------------------------
Packages:
Repo:  http://voxl-packages.modalai.com/ ./dists/qrb5165/sdk-1.3/binary-arm64/
Last Updated: 2024-09-13 17:37:13
List:
kernel-module-voxl-fsync-mod-4.19.125     1.0-r0
kernel-module-voxl-gpio-mod-4.19.125      1.0-r0
kernel-module-voxl-platform-mod-4.19.125  1.0-r0
libfc-sensor                              1.0.7
libmodal-cv                               0.5.9
libmodal-exposure                         0.1.1
libmodal-journal                          0.2.2
libmodal-json                             0.4.3
libmodal-pipe                             2.10.1
libqrb5165-io                             0.4.6
libvoxl-cci-direct                        0.2.3
libvoxl-cutils                            0.1.1
modalai-slpi                              1.1.19
mv-voxl                                   0.1-r0
qrb5165-bind                              0.1-r0
qrb5165-dfs-server                        0.2.0
qrb5165-imu-server                        1.0.1
qrb5165-rangefinder-server                0.1.3
qrb5165-slpi-test-sig                     01-r0
qrb5165-system-tweaks                     0.3.0
qrb5165-tflite                            2.8.0-2
voxl-bind-spektrum                        0.1.1
voxl-camera-calibration                   0.5.7
voxl-camera-server                        2.0.0
voxl-ceres-solver                         2:1.14.0-10
voxl-configurator                         0.8.2
voxl-cpu-monitor                          0.4.8
voxl-docker-support                       1.3.1
voxl-elrs                                 0.2.2
voxl-esc                                  1.4.6
voxl-feature-tracker                      0.3.16
voxl-flow-server                          0.3.6
voxl-fsync-mod                            1.0-r0
voxl-gphoto2-server                       0.0.10
voxl-gpio-mod                             1.0-r0
voxl-jpeg-turbo                           2.1.3-5
voxl-lepton-server                        1.2.2
voxl-lepton-tracker                       0.0.1
voxl-libgphoto2                           0.0.4
voxl-libuvc                               1.0.7
voxl-logger                               0.4.5
voxl-mavcam-manager                       0.5.6
voxl-mavlink                              0.1.1
voxl-mavlink-server                       1.4.2
voxl-modem                                1.1.3
voxl-mongoose                             7.7.0-1
voxl-mpa-to-ros                           0.3.9
voxl-mpa-tools                            1.2.3
voxl-neopixel-manager                     0.0.3
voxl-open-vins                            0.4.14
voxl-open-vins-server                     0.2.66
voxl-opencv                               4.5.5-2
voxl-osd                                  0.0.1
voxl-platform-mod                         1.0-r0
voxl-portal                               0.6.9
voxl-px4                                  1.14.0-2.0.79
voxl-px4-imu-server                       0.1.2
voxl-px4-params                           0.4.8
voxl-qvio-server                          1.0.4
voxl-remote-id                            0.0.9
voxl-reset-slpi                           0.0.1
voxl-state-estimator                      0.0.2
voxl-streamer                             0.7.4
voxl-suite                                1.3.4~beta2
voxl-tag-detector                         0.0.4
voxl-tflite-server                        0.3.4
voxl-utils                                1.4.2
voxl-uvc-server                           0.1.7
voxl-vision-hub                           1.8.6
voxl-vtx                                  1.0.5
voxl2-io                                  0.0.3
voxl2-system-image                        1.7.8-r0
voxl2-wlan                                1.0-r0
--------------------------------------------------------------------------------

LucaVertiq

@Eric-Katzfey That worked perfectly! Thank you. I put all the dynamic notch stuff on my github. I know James mentioned he wanted to implement it.

LucaVertiq

@Moderator So I was able to get the notch filter working, but I'm still wondering if there's a way to get the mavlink onboard to publish faster than 10hz.

They dynamic notch filter as really improved the VIO on my vehicle which is very prone to vibrations.

LucaVertiq

@Moderator bump?

LucaVertiq

@James-Strawson
Hey James,

We chatted at AUVSI about getting dynamic notch filtering working. Right now I have all the basic stuff implemented, but am running into two issues.

The first issue is that even in a sandbox I'm having trouble creating an appropriate notch filter. I've looked at how PX4 and Ardupilot do it, but I'm not seeing as much attenuation as I'd expect. Seeing as you wrote the filter library, I was wondering if you had any notch filters already written. My filter knowledge is rather lacking.

The second issue is that ESC telemetry is only reported across the 'mavlink_onboard' topic at 10hz. I think I'd rather have it be a bit faster.

I was wondering if you had any insight into either of these. Here's the repo I've been working off of:
https://github.com/vertiq-luca/qrb5165-imu-server/tree/feature/rpm_filter
It's just mavlink rpm data feeding a low pass filter for now until I can get my notch filter to work, but it does appear to be filtering!

Best,

Luca

LucaVertiq

@Alex-Kushleyev Glad that you were able to get that working. I think I tried updating to SDK 1.2.X and was having issues with getting the camera server working, but I think I may just need to wipe. All of these things would be super useful! I'll be out for the next week, so no rush, but it's very good to know that there's a fix.

LucaVertiq

@Alex-Kushleyev Oh! I'm glad I wasn't misunderstanding how it should work. Thanks for taking the time to check it out. I was actually able to get it to work in both slots 2 and 3, just not 0 (and of course not 1 because I don't have the driver).

LucaVertiq

@Alex-Kushleyev Thanks for taking a look. I think we won't need a driver for the imx214 on port 1, as I want the camera cables to extend outwards from the board instead of lying across the board regardless of whether we have them on J6 or J7. I've done some more testing and remain stuck with this error so if you find anything please let me know! I may try reflashing to 1.1.2 as a clean slate, but I want to make sure I have all of the settings I've changed and drivers I've built saved if they aren't in the /data/ directiory

LucaVertiq

@Moderator
Another update because this is confusing me. If I have the cameras plugged in with ov7251_1.bin and imx214_0.bin if I run voxl-camera-server -l immediately after power on it detects 2 cameras.

voxl2-mini:~$ voxl-camera-server -l
existing instance of voxl-camera-server found, attempting to stop it
DEBUG:   Attempting to open the hal module
DEBUG:   SUCCESS: Camera module opened on attempt 0
DEBUG:   ----------- Number of cameras: 2

DEBUG:   Cam idx: 0, Cam slot: 0, Slave Address: 0x0020, Sensor Id: 0x0214
DEBUG:   Cam idx: 1, Cam slot: 1, Slave Address: 0x00E2, Sensor Id: 0x7750
DEBUG:   Note: This list comes from the HAL module and may not be indicative
DEBUG:          of configurations that have full pipelines

DEBUG:   Number of cameras: 2


====================================
Stats for camera: 0

ANDROID_SCALER_AVAILABLE_RAW_SIZES:
These are likely supported by the sensor
4208 x 3120
3840 x 2160
1920 x 1080
1920 x 1080

ANDROID_SCALER_AVAILABLE_STREAM_CONFIGURATIONS:
These are NOT necessarily supported by the sensor
4208 x 3120 HAL_PIXEL_FORMAT_YCbCr_420_888
4208 x 3120 HAL_PIXEL_FORMAT_BLOB
4160 x 3120 HAL_PIXEL_FORMAT_YCbCr_420_888
4160 x 3120 HAL_PIXEL_FORMAT_BLOB
4096 x 2304 HAL_PIXEL_FORMAT_YCbCr_420_888
4096 x 2304 HAL_PIXEL_FORMAT_BLOB
4096 x 2160 HAL_PIXEL_FORMAT_YCbCr_420_888
4096 x 2160 HAL_PIXEL_FORMAT_BLOB
4056 x 3040 HAL_PIXEL_FORMAT_YCbCr_420_888
4056 x 3040 HAL_PIXEL_FORMAT_BLOB
4000 x 3000 HAL_PIXEL_FORMAT_YCbCr_420_888
4000 x 3000 HAL_PIXEL_FORMAT_BLOB
3840 x 2160 HAL_PIXEL_FORMAT_YCbCr_420_888
3840 x 2160 HAL_PIXEL_FORMAT_BLOB
3264 x 2448 HAL_PIXEL_FORMAT_YCbCr_420_888
3264 x 2448 HAL_PIXEL_FORMAT_BLOB
3200 x 2400 HAL_PIXEL_FORMAT_YCbCr_420_888
3200 x 2400 HAL_PIXEL_FORMAT_BLOB
2976 x 2976 HAL_PIXEL_FORMAT_YCbCr_420_888
2976 x 2976 HAL_PIXEL_FORMAT_BLOB
2688 x 1512 HAL_PIXEL_FORMAT_YCbCr_420_888
2688 x 1512 HAL_PIXEL_FORMAT_BLOB
2592 x 1944 HAL_PIXEL_FORMAT_YCbCr_420_888
2592 x 1944 HAL_PIXEL_FORMAT_BLOB
2048 x 1536 HAL_PIXEL_FORMAT_YCbCr_420_888
2048 x 1536 HAL_PIXEL_FORMAT_BLOB
1920 x 1440 HAL_PIXEL_FORMAT_YCbCr_420_888
1920 x 1440 HAL_PIXEL_FORMAT_BLOB
1928 x 1208 HAL_PIXEL_FORMAT_YCbCr_420_888
1928 x 1208 HAL_PIXEL_FORMAT_BLOB
1920 x 1080 HAL_PIXEL_FORMAT_YCbCr_420_888
1920 x 1080 HAL_PIXEL_FORMAT_BLOB
1600 x 1200 HAL_PIXEL_FORMAT_YCbCr_420_888
1600 x 1200 HAL_PIXEL_FORMAT_BLOB
1440 x 1080 HAL_PIXEL_FORMAT_YCbCr_420_888
1440 x 1080 HAL_PIXEL_FORMAT_BLOB
1280 x  960 HAL_PIXEL_FORMAT_YCbCr_420_888
1280 x  960 HAL_PIXEL_FORMAT_BLOB
1280 x  800 HAL_PIXEL_FORMAT_YCbCr_420_888
1280 x  800 HAL_PIXEL_FORMAT_BLOB
1280 x  768 HAL_PIXEL_FORMAT_YCbCr_420_888
1280 x  768 HAL_PIXEL_FORMAT_BLOB
1280 x  720 HAL_PIXEL_FORMAT_YCbCr_420_888
1280 x  720 HAL_PIXEL_FORMAT_BLOB
1080 x 1080 HAL_PIXEL_FORMAT_YCbCr_420_888
1080 x 1080 HAL_PIXEL_FORMAT_BLOB
1024 x  738 HAL_PIXEL_FORMAT_YCbCr_420_888
1024 x  738 HAL_PIXEL_FORMAT_BLOB
1024 x  768 HAL_PIXEL_FORMAT_YCbCr_420_888
1024 x  768 HAL_PIXEL_FORMAT_BLOB
 864 x  480 HAL_PIXEL_FORMAT_YCbCr_420_888
 864 x  480 HAL_PIXEL_FORMAT_BLOB
 800 x  600 HAL_PIXEL_FORMAT_YCbCr_420_888
 800 x  600 HAL_PIXEL_FORMAT_BLOB
 800 x  480 HAL_PIXEL_FORMAT_YCbCr_420_888
 800 x  480 HAL_PIXEL_FORMAT_BLOB
 720 x 1280 HAL_PIXEL_FORMAT_YCbCr_420_888
 720 x 1280 HAL_PIXEL_FORMAT_BLOB
 720 x  480 HAL_PIXEL_FORMAT_YCbCr_420_888
 720 x  480 HAL_PIXEL_FORMAT_BLOB
 640 x  480 HAL_PIXEL_FORMAT_YCbCr_420_888
 640 x  480 HAL_PIXEL_FORMAT_BLOB
 640 x  400 HAL_PIXEL_FORMAT_YCbCr_420_888
 640 x  400 HAL_PIXEL_FORMAT_BLOB
 640 x  360 HAL_PIXEL_FORMAT_YCbCr_420_888
 640 x  360 HAL_PIXEL_FORMAT_BLOB
 352 x  288 HAL_PIXEL_FORMAT_YCbCr_420_888
 352 x  288 HAL_PIXEL_FORMAT_BLOB
 320 x  240 HAL_PIXEL_FORMAT_YCbCr_420_888
 320 x  240 HAL_PIXEL_FORMAT_BLOB
 240 x  320 HAL_PIXEL_FORMAT_YCbCr_420_888
 240 x  320 HAL_PIXEL_FORMAT_BLOB
 176 x  144 HAL_PIXEL_FORMAT_YCbCr_420_888
 176 x  144 HAL_PIXEL_FORMAT_BLOB
4208 x 3120 HAL_PIXEL_FORMAT_RAW10
4208 x 3120 HAL_PIXEL_FORMAT_RAW12
4208 x 3120 HAL_PIXEL_FORMAT_RAW16
4208 x 3120 HAL_PIXEL_FORMAT_RAW_OPAQUE

ANDROID_SENSOR_INFO_SENSITIVITY_RANGE
        min = 54
        max = 431

ANDROID_SENSOR_MAX_ANALOG_SENSITIVITY
        431

ANDROID_SENSOR_INFO_EXPOSURE_TIME_RANGE
        min = 10449ns
        max = 683714540ns


====================================
Stats for camera: 1

ANDROID_SCALER_AVAILABLE_RAW_SIZES:
These are likely supported by the sensor
 640 x  480
 640 x  480
 640 x  480

ANDROID_SCALER_AVAILABLE_STREAM_CONFIGURATIONS:
These are NOT necessarily supported by the sensor
 640 x  480 HAL_PIXEL_FORMAT_YCbCr_420_888
 640 x  480 HAL_PIXEL_FORMAT_BLOB
 640 x  400 HAL_PIXEL_FORMAT_YCbCr_420_888
 640 x  400 HAL_PIXEL_FORMAT_BLOB
 640 x  360 HAL_PIXEL_FORMAT_YCbCr_420_888
 640 x  360 HAL_PIXEL_FORMAT_BLOB
 352 x  288 HAL_PIXEL_FORMAT_YCbCr_420_888
 352 x  288 HAL_PIXEL_FORMAT_BLOB
 320 x  240 HAL_PIXEL_FORMAT_YCbCr_420_888
 320 x  240 HAL_PIXEL_FORMAT_BLOB
 240 x  320 HAL_PIXEL_FORMAT_YCbCr_420_888
 240 x  320 HAL_PIXEL_FORMAT_BLOB
 176 x  144 HAL_PIXEL_FORMAT_YCbCr_420_888
 176 x  144 HAL_PIXEL_FORMAT_BLOB
 640 x  480 HAL_PIXEL_FORMAT_RAW10
 640 x  480 HAL_PIXEL_FORMAT_RAW12
 640 x  480 HAL_PIXEL_FORMAT_RAW16
 640 x  480 HAL_PIXEL_FORMAT_RAW_OPAQUE

ANDROID_SENSOR_INFO_SENSITIVITY_RANGE
        min = 54
        max = 3451

ANDROID_SENSOR_MAX_ANALOG_SENSITIVITY
        3451

ANDROID_SENSOR_INFO_EXPOSURE_TIME_RANGE
        min = 0ns
        max = 1266732525ns

====================================
Number of cameras detected: 2
====================================

If I instead run voxl-inspect-cam -a it shows that tracking is working, but hires is not.

|   hires_large_color |
| hires_large_encoded |
|    hires_large_grey |
|   hires_small_color |
| hires_small_encoded |
|    hires_small_grey |
|      hires_snapshot |
|        qvio_overlay |  368640 |  640 |  576 |  5.00 |   64 |      694 |     25.3  | 29.5 |   87.0 | RAW8
|            tracking |  307200 |  640 |  480 |  5.00 |   64 |      694 |     11.8  | 30.0 |   73.7 | RAW8

My camera server config is below:

 * voxl-camera-server Configuration File
 *
 * Each camera has configurations for up to 4 HAL3 streams:
 *    - `preview` stream for raw unprocessed images from CV cameras
 *    - `small_video` 720p (ish) h264/h265 compressed for fpv video streaming
 *    - `large_video` 4k (ish) h264/h265 for onboard video recording to disk
 *    - `snapshot` ISP-processed JPG snapshots that get saved to disk
 *
 * on QRB5165 platforms (VOXL2 and VOXL2 mini) you can only have 3 of the 4 enabled
 *
 * This file is generated from default values by voxl-configure-cameras.
 * Do not expect arbitrary resolutions to work, the ISP and video compression
 * pipelines only support very specific resolutions.
 *
 * The default video compression mode is cqp or Constant Quantization Parameter
 *
 *
 *
 */
{
        "version":      0.1,
        "cameras":      [{
                        "type": "imx214",
                        "name": "hires",
                        "enabled":      true,
                        "camera_id":    0,
                        "fps":  30,
                        "en_preview":   false,
                        "preview_width":        640,
                        "preview_height":       480,
                        "en_raw_preview":       false,
                        "en_small_video":       true,
                        "small_video_width":    1024,
                        "small_video_height":   768,
                        "small_venc_mode":      "h265",
                        "small_venc_br_ctrl":   "cqp",
                        "small_venc_Qfixed":    30,
                        "small_venc_Qmin":      15,
                        "small_venc_Qmax":      40,
                        "small_venc_nPframes":  9,
                        "small_venc_mbps":      2,
                        "en_large_video":       true,
                        "large_video_width":    4208,
                        "large_video_height":   3120,
                        "large_venc_mode":      "h265",
                        "large_venc_br_ctrl":   "cqp",
                        "large_venc_Qfixed":    38,
                        "large_venc_Qmin":      15,
                        "large_venc_Qmax":      50,
                        "large_venc_nPframes":  29,
                        "large_venc_mbps":      30,
                        "en_snapshot":  true,
                        "en_snapshot_width":    4208,
                        "en_snapshot_height":   3120,
                        "ae_mode":      "isp"
                },
                {
                        "type": "ov7251",
                        "name": "tracking",
                        "enabled":      true,
                        "camera_id":    1,
                        "fps":  30,
                        "en_rotate":    false,
                        "en_preview":   true,
                        "preview_width":        640,
                        "preview_height":       480,
                        "en_raw_preview":       true,
                        "ae_mode":      "lme_msv",
                        "ae_desired_msv":       60,
                        "exposure_min_us":      20,
                        "exposure_max_us":      33000,
                        "gain_min":     54,
                        "gain_max":     8000,
                        "exposure_soft_min_us": 5000,
                        "ae_filter_alpha":      0.600000023841858,
                        "ae_ignore_fraction":   0.20000000298023224,
                        "ae_slope":     0.05000000074505806,
                        "ae_exposure_period":   1,
                        "ae_gain_period":       1
                }]
}

It appears as if something happens when the stream is accessed. If I run voxl-camera-server and then kill it voxl-camera-server -l will still show 2 cameras, but if I instead run voxl-camera-server and then in another window run voxl-inspect-cam -a the original window will no longer respond to ctrl-C. If I force kill i with kill -9 proccess_num then run voxl-camera-server -l it now shows 1 camera until I reboot.

Any ideas?

LucaVertiq

@Alex-Kushleyev Hey Alex, that worked for that fix. Additionally I attempted to put the same cameras on J6 in the orientation with the OV7251 on the M0135 Upper and the IMX214 on the M0135 Lower.

My understanding is that I should put the com.qti.sensormodule.ov7251_1.bin and com.qti.sensormodule.imx214_0.bin into the /usr/lib/camera folder (and delete the non-used ones).

When I run voxl-camera-server -l this only shows the tracking camera. Plugging the board back into J7 again shows both cameras when switching back to imx214_2 and ov7251_3 drivers.

Am I making a bad assumption here?

Additionally I notice that the IMX214 doesn't have drivers for _1 and _5, but does have a driver for _3. Why is this? We aren't quite sure what orientation will work best for our vehicle layout so we were hoping to be able to use all 4 configurations of double camera mounted on m0135 boards with J6 and J7.

Thanks,

Luca

LucaVertiq

@Alex-Kushleyev Thank you! I'll look through the code a bit harder to make sure I'm not doing anything silly. I'll let you know if I have any more questions.

LucaVertiq

@Alex-Kushleyev So it looks like this message just repeats in dmesg -w

[   85.669815] Fatal error on slpi!
[   85.669887] slpi subsystem failure reason: err_qdi.c:1063:PC=e616ed3c,SP=31782ca0,FP=31782cc8,LR=e616ed40,BADVA=b220fb58,CAUSE=1e01,TASK=Anonymous.
[   85.669921] subsys-restart: subsystem_restart_dev(): Restart sequence requested for slpi, restart_level = RELATED.
[   85.671120] adsprpc: fastrpc_restart_notifier_cb: slpi subsystem is restarting
[   85.671127] subsys-restart: subsystem_shutdown(): [kworker/u19:0:1857]: Shutting down slpi
[   85.689766] adsprpc: fastrpc_rpmsg_remove: closed rpmsg channel of slpi
[   85.690236] coresight-remote-etm soc:ssc_etm0: Connection disconnected between QMI handle and 8 service
[   85.690290] sysmon-qmi: ssctl_del_server: Connection lost between QMI handle and slpi's SSCTL service
[   85.690690] adsprpc: fastrpc_restart_notifier_cb: received RAMDUMP notification for slpi
[   85.692605] subsys-restart: subsystem_powerup(): [kworker/u19:0:1857]: Powering up slpi
[   85.693232] subsys-pil-tz 5c00000.qcom,ssc: slpi: loading from 0x0000000088c00000 to 0x000000008a600000
[   85.768512] subsys-pil-tz 5c00000.qcom,ssc: slpi: Brought out of reset
[   85.824221] subsys-pil-tz 5c00000.qcom,ssc: Subsystem error monitoring/handling services are up
[   85.824375] subsys-pil-tz 5c00000.qcom,ssc: slpi: Power/Clock ready interrupt received
[   85.824481] qcom_rpmh DRV:apps_rsc TCS Busy, retrying RPMH message send: addr=0x30040
[   85.825742] adsprpc: fastrpc_restart_notifier_cb: slpi subsystem is up
[   85.825747] subsys-restart: subsystem_restart_wq_func(): [kworker/u19:0:1857]: Restart sequence for slpi completed.
[   85.827958] -1620030744:Entered
[   85.828116] -1620030744:SMD QRTR driver probed
[   85.831088] sysmon-qmi: ssctl_new_server: Connection established between QMI handle and slpi's SSCTL service
[   85.831100] coresight-remote-etm soc:ssc_etm0: Connection established between QMI handle and 8 service
[   85.834589] adsprpc: fastrpc_rpmsg_probe: opened rpmsg channel for slpi
[   85.882137] diag: In diag_send_peripheral_buffering_mode, buffering flag not set for 3
[   87.173372] Fatal error on slpi!
[   87.173448] slpi subsystem failure reason: err_qdi.c:1063:EF:sensor_process:0x1:SNS_SEE_I_1:0x67:sns_stream_service.c:436:req_found.
[   87.173497] subsys-restart: subsystem_restart_dev(): Restart sequence requested for slpi, restart_level = RELATED.
[   87.175158] adsprpc: fastrpc_restart_notifier_cb: slpi subsystem is restarting
[   87.175170] subsys-restart: subsystem_shutdown(): [kworker/u19:0:1857]: Shutting down slpi
[   87.192557] adsprpc: fastrpc_rpmsg_remove: closed rpmsg channel of slpi
[   87.192709] coresight-remote-etm soc:ssc_etm0: Connection disconnected between QMI handle and 8 service
[   87.192730] sysmon-qmi: ssctl_del_server: Connection lost between QMI handle and slpi's SSCTL service
[   87.194300] adsprpc: fastrpc_restart_notifier_cb: received RAMDUMP notification for slpi
[   87.197243] subsys-restart: subsystem_powerup(): [kworker/u19:0:1857]: Powering up slpi
[   87.197711] subsys-pil-tz 5c00000.qcom,ssc: slpi: loading from 0x0000000088c00000 to 0x000000008a600000
[   87.270116] subsys-pil-tz 5c00000.qcom,ssc: slpi: Brought out of reset
[   87.325674] subsys-pil-tz 5c00000.qcom,ssc: Subsystem error monitoring/handling services are up
[   87.325726] subsys-pil-tz 5c00000.qcom,ssc: slpi: Power/Clock ready interrupt received
[   87.325841] qcom_rpmh DRV:apps_rsc TCS Busy, retrying RPMH message send: addr=0x30040
[   87.331325] adsprpc: fastrpc_restart_notifier_cb: slpi subsystem is up
[   87.331330] subsys-restart: subsystem_restart_wq_func(): [kworker/u19:0:1857]: Restart sequence for slpi completed.
[   87.331891] -1620030744:Entered
[   87.340471] -1620030744:SMD QRTR driver probed
[   87.343180] adsprpc: fastrpc_rpmsg_probe: opened rpmsg channel for slpi
[   87.344830] sysmon-qmi: ssctl_new_server: Connection established between QMI handle and slpi's SSCTL service
[   87.344850] coresight-remote-etm soc:ssc_etm0: Connection established between QMI handle and 8 service
[   87.350149] diag: In diag_send_peripheral_buffering_mode, buffering flag not set for 3
[   88.637779] Fatal error on slpi!

Then everything from Fatal error repeats.

LucaVertiq

@Alex-Kushleyev When you say 'the crash message' do you mean the ">>> Got an exception from send_request <<<"?

That occurs while voxl-px4 is running. It won't crash and will continue to output the semi repeating messages seen after that until I kill voxl-px4. I'll test the dmesg -w on Monday when I get back into work.

LucaVertiq

@Eric-Katzfey @Alex-Kushleyev We determined it was only when we had the telemetry section of our driver working. For now I've disabled it, but it seems like either we're bogging it down, or that the communication through lib sensors is getting bogged down. When I restart the voxl-px4 on the apps side it works again so it seems like the dsp side hasn't actually crashed. I've started looking at work_queue and it seems that we slow down the rate a bit, so we might need to optimize. I may check out how you guys are using reporting esc telemetry. Maybe we should only report it once all escs have been reported on.

LucaVertiq

@Moderator
Hi, I'm adding support for serial connection to vertiq modules for a project and I'm running into a difficult to debug issue with PX4. I have the driver working on normal PX4 boards, and mostly working on the VOXL2 Mini.

When I am disarmed everything works as expected. If I arm in manual mode, acro mode or stabilize mode (probably others that don't require pos/vel as well) everything works normally most of the time and motors respond as expected with telemetry.

If I arm in position mode for example the DSP side of the chip seems to stop responding. The output of PX4 on the app processor becomes this:

INFO  [muorb] SLPI: Armed by external command
INFO  [logger] Start file log (type: full)
INFO  [logger] [logger] /data/px4/log/2024-03-14/19_52_13.ulg
WARN  [mavlink] Event dropped (5, 65526)
WARN  [mavlink] Event dropped (5, 65526)
INFO  [logger] Opened full log file: /data/px4/log/2024-03-14/19_52_13.ulg
WARN  [mavlink] Dropped 65521 events (seq=65526)
WARN  [mavlink] Dropped 65521 events (seq=65526)
INFO  [muorb] SLPI: Advertising remote topic logger_status
INFO  [uORB] Advertising remote topic sensor_gps
INFO  [uORB] Advertising remote topic estimator_gps_status
INFO  [uORB] Advertising remote topic estimator_aid_src_gnss_hgt
INFO  [uORB] Advertising remote topic estimator_aid_src_gnss_pos
INFO  [uORB] Advertising remote topic estimator_aid_src_gnss_vel
WARN  [mavlink] Event dropped (65527, 9)
WARN  [mavlink] Event dropped (65527, 9)
WARN  [mavlink] Dropped 7 events (seq=9)
WARN  [mavlink] Dropped 7 events (seq=9)
>>> Got an exception from send_request <<<
>>> Send succeeded after retries <<<
Sending topic message
--- msg_id: 1033
--- topic name: offboard_control_mode
Sending topic message
--- msg_id: 1033
--- topic name: vehicle_visual_odometry
Sending topic message
--- msg_id: 1033
--- topic name: manual_control_input
Sending topic message
--- msg_id: 1033
--- topic name: offboard_control_mode
Sending topic message
--- msg_id: 1033
--- topic name: vehicle_visual_odometry
Sending topic message
--- msg_id: 1033
--- topic name: manual_control_input
Sending topic message
--- msg_id: 1033
--- topic name: offboard_control_mode
Sending topic message
--- msg_id: 1033
--- topic name: vehicle_visual_odometry
Sending topic message
--- msg_id: 1033
--- topic name: manual_control_input
Sending topic message
--- msg_id: 1033
--- topic name: offboard_control_mode
Sending topic message
--- msg_id: 1033
--- topic name: vehicle_visual_odometry
Sending topic message
--- msg_id: 1033
--- topic name: offboard_control_mode
Sending topic message
--- msg_id: 1033
--- topic name: vehicle_visual_odometry
Sending topic message
--- msg_id: 1033
--- topic name: manual_control_input
Sending topic message
--- msg_id: 1033
--- topic name: offboard_control_mode
Sending topic message
--- msg_id: 1033
--- topic name: vehicle_visual_odometry
Sending topic message
--- msg_id: 1033
--- topic name: manual_control_input
Connection error: connection reset
Sending topic message
--- msg_id: 1033
--- topic name: offboard_control_mode
Got response cb 0
Sending topic message
--- msg_id: 1033
--- topic name: vehicle_visual_odometry
Got response cb 0
Got flight controller event
Received standard error event SNS_STD_ERROR_NOT_SUPPORTED
Sending topic message
--- msg_id: 1033
--- topic name: cpuload
Got flight controller event
Received standard error event SNS_STD_ERROR_NOT_SUPPORTED
Got response cb 0
Sending topic message
--- msg_id: 1033
--- topic name: manual_control_input
Got response cb 0
Got flight controller event
Received standard error event SNS_STD_ERROR_NOT_SUPPORTED
Sending topic message
--- msg_id: 1033
--- topic name: offboard_control_mode
Got response cb 0
Got flight controller event
Received standard error event SNS_STD_ERROR_NOT_SUPPORTED
Sending topic message
--- msg_id: 1033
--- topic name: vehicle_visual_odometry
Got response cb 0
Got flight controller event
Received standard error event SNS_STD_ERROR_NOT_SUPPORTED
Sending topic message
--- msg_id: 1033
--- topic name: manual_control_input
Got flight controller event
Received standard error event SNS_STD_ERROR_NOT_SUPPORTED
Got response cb 0
Sending topic message
--- msg_id: 1033
--- topic name: offboard_control_mode
Got response cb 0
Got flight controller event
Received standard error event SNS_STD_ERROR_NOT_SUPPORTED
Sending topic message
--- msg_id: 1033
--- topic name: vehicle_visual_odometry
Got response cb 0
Got flight controller event
Received standard error event SNS_STD_ERROR_NOT_SUPPORTED
Sending topic message
--- msg_id: 1033
--- topic name: manual_control_input
Got response cb 0
Got flight controller event
Received standard error event SNS_STD_ERROR_NOT_SUPPORTED
Sending topic message
--- msg_id: 1033
--- topic name: offboard_control_mode
Got response cb 0
Got flight controller event
Received standard error event SNS_STD_ERROR_NOT_SUPPORTED
Sending topic message
--- msg_id: 1033
--- topic name: vehicle_visual_odometry
Got response cb 0
Got flight controller event
Received standard error event SNS_STD_ERROR_NOT_SUPPORTED
Sending topic message
--- msg_id: 1033

It seems to be some sort of issue with muorb? Maybe the overall volume of messages is too much? I'm not sure, but would love some help as it seems like it's at the intersection of the two chips possibly.

Thanks,

Luca

LucaVertiq

@Moderator
At some point the camera orientation for the voxl2 mini flipped from:

To:

Is there any way to go back to the original orientation? It makes camera mounting much easier.

Thanks!