@James-Strawson Thanks for your reply! It made me realise what the error must be. We use a locally cloned repository for libmodalpipe because we want to build our software directly on the drone. This is then used in the build and link process of our software. This happened to be on the 'master' branch and not the 'SDK-1.0.0' branch. So the pipe clients use a different version than the pipe servers which is likely the cause of our errors.
Posts made by Tjark
-
RE: Possible bug in libmodal_pipe server.c
-
RE: mpa_to_ros node crashes after subscribing to /hires topics
I have the same issue. If you try it multiple times sometimes it will work but most of the times it will crash like shown.
-
RE: Possible bug in libmodal_pipe server.c
@James-Strawson I still haven't found the root cause but I wanted to share some of my findings. Maybe you can help my understanding. A little bit of background: we don't want to power off the drone but when we are not flying we go into a sleep state where we shut down some of the voxl programs and our own programs. Then when we wake up we start up the programs and everything should work again. It is after about 6 of these cycles where we have this issue (so it also takes a while to reproduce).
Here is some logging of where the problem just kicked in:
Aug 19 20:15:14 QUAD voxl-vision-hub[1524]: write to ch: 0 id: 14 result: -1 errno: 32 Aug 19 20:15:14 QUAD voxl-vision-hub[1524]: write error: Broken pipe Aug 19 20:15:14 QUAD voxl-vision-hub[1524]: previous client state was 2 Aug 19 20:15:14 QUAD voxl-vision-hub[1524]: Client voxl_pipe_handler-232508 (id 14) disconnected from channel 0 Aug 19 20:15:14 QUAD voxl-vision-hub[1524]: write to ch: 0 id: 15 result: -1 errno: 32 Aug 19 20:15:14 QUAD voxl-vision-hub[1524]: write error: Broken pipe Aug 19 20:15:14 QUAD voxl-vision-hub[1524]: previous client state was 1 Aug 19 20:15:14 QUAD voxl-vision-hub[1524]: Client voxl_pipe_handler695511 (id 15) disconnected from channel 0 Aug 19 20:15:14 QUAD voxl-vision-hub[1524]: ERROR in pipe_server_write_to_client, client_id should be between 0 & 15 Aug 19 20:15:14 QUAD voxl-vision-hub[1524]: write to ch: 1 id: 14 result: -1 errno: 32 Aug 19 20:15:14 QUAD voxl-vision-hub[1524]: write error: Broken pipe Aug 19 20:15:14 QUAD voxl-vision-hub[1524]: previous client state was 2 Aug 19 20:15:14 QUAD voxl-vision-hub[1524]: Client voxl_pipe_handler672850 (id 14) disconnected from channel 1 Aug 19 20:15:14 QUAD voxl-vision-hub[1524]: write to ch: 1 id: 15 result: -1 errno: 32 Aug 19 20:15:14 QUAD voxl-vision-hub[1524]: write error: Broken pipe Aug 19 20:15:14 QUAD voxl-vision-hub[1524]: previous client state was 1 Aug 19 20:15:14 QUAD voxl-vision-hub[1524]: Client voxl_pipe_handler507424 (id 15) disconnected from channel 1 Aug 19 20:15:14 QUAD voxl-vision-hub[1524]: ERROR in pipe_server_write_to_client, client_id should be between 0 & 15 Aug 19 20:15:14 QUAD voxl-vision-hub[1524]: ERROR in pipe_server_write_to_client, client_id should be between 0 & 15 Aug 19 20:15:14 QUAD voxl-vision-hub[1524]: ERROR in pipe_server_write_to_client, client_id should be between 0 & 15 Aug 19 20:15:14 QUAD voxl-vision-hub[1524]: ERROR in pipe_server_write_to_client, client_id should be between 0 & 15
voxl_pipe_handler
is our own program. When this was logged we weren't properly closing the pipe when our program was terminated so that could explain the write errors. But after handling that correctly, we still have the issue ofclient_id should be between 0 & 15
.A few questions. The id behind the client name seems to be a random number although the commit which adds the random number is after SDK-1.0.0. (https://gitlab.com/voxl-public/voxl-sdk/core-libs/libmodal-pipe/-/commit/b5fc28c9fc41184e0bbeee09c7f1867f9dbc1121). How is that possible? Did you deploy it well? This is the version we use:
QUAD:~$ apt show voxl-vision-hub Package: voxl-vision-hub Version: 1.6.6 Priority: optional Section: base Maintainer: James Strawson <james@modalai.com> Installed-Size: unknown Provides: voxl-vision-px4 Depends: librc-math,libmodal-pipe(>=2.4.0),libmodal-json,voxl-mpa-tools(>=0.2.5),voxl-mavlink-server,libmodal-cv(>=0.3.1) Conflicts: voxl-vision-px4,voxl-mavlink-server(<<1.0.0) Replaces: voxl-vision-px4 Download-Size: 88.4 kB APT-Manual-Installed: yes APT-Sources: file:/data/voxl-suite-offline-packages ./ Packages Description: main hub managing communication between VOXL MPA services and autopilots
In https://gitlab.com/voxl-public/voxl-sdk/core-libs/libmodal-pipe/-/blob/master/library/src/misc.c?ref_type=heads#L168, I think you're taking the modulo of a possible negative number. The modulo can then also be a negative number (https://stackoverflow.com/questions/7594508/modulo-operator-with-negative-values) so your result is not within the specified range. See also the logging above containing a negative number.
I think that reconnecting a pipe with the same name makes that a new client id is used, possibly due to the random number in the name and then no two client names will be the same. I also saw that you changed something in that part of the code in this commit: https://gitlab.com/voxl-public/voxl-sdk/core-libs/libmodal-pipe/-/commit/773de63cb73000c08df1b04bc4e2f1e78e700816 Do you think that that could be the case? I will try to test that.
-
Possible bug in libmodal_pipe server.c
I was looking through the libmodal_pipe code because we have some issues with an error
ERROR in pipe_server_write_to_client, client_id should be between 0 & 15
and I'm not sure what is causing this. We are restarting voxl programs often and I think something goes wrong with connecting/disconnecting pipes but I haven't figured it out yet.But when I looked through the code I saw that here all file descriptors are closed: https://gitlab.com/voxl-public/voxl-sdk/core-libs/libmodal-pipe/-/blob/master/library/src/server.c?ref_type=heads#L1611. But a few lines above that (https://gitlab.com/voxl-public/voxl-sdk/core-libs/libmodal-pipe/-/blob/master/library/src/server.c?ref_type=heads#L1568),
c[ch].n_clients
is set to0
so I think the code at line 1611 isn't doing anything. I don't know if it gives problems and it is probably not related to my problem but I wanted to point it out to you. -
RE: Connect RC controller with SDK 1.0.0/PX4 1.14
@Eric-Katzfey thanks for the update, although it is a pity since it used to work. Do you have an idea when support will be added again?
-
RE: Most VOXL 2 hires-capable camera ports nonfunctional after SDK 1.0 flash
@brycek That's great!
@Modalai could you still answer the question if this is a good way of handling this?
-
RE: Can't install ros-melodic-tf2-geometry-msgs with voxl-suite 1.0.0
@ModalAI could you answer this question?
-
Connect RC controller with SDK 1.0.0/PX4 1.14
We are trying to get a drone working with SDK 1.0.0 but we are unable to get a RC controller properly connected. It worked with SDK 0.9.5.
We have a RadioLink RC controller connected to the VOXL2 IO board similar to the Graupner setup: https://docs.modalai.com/voxl2-io-user-guide/#using-sbus-graupner-gr-16 but we don't get RC signals through in QGroundControl. This topic seems to suggest that the VOXL2 IO board is not yet supported for SDK 1.0.0 but that is for an ESC: https://forum.modalai.com/topic/2429/voxl2-io-m0065-support-on-px4-1-14-in-sdk-1-0-0. Should it be possible to connect our RC with VOXL2 IO?
-
RE: Most VOXL 2 hires-capable camera ports nonfunctional after SDK 1.0 flash
@brycek We had the same issue but we managed to get it working. We have the following setup with 3 cameras:
Camera name: hires2
Camera type: imx412
Camera id: 0
Camera slot: 2Camera name: hires
Camera type: imx214
Camera id: 1
Camera slot: 3Camera name: tracking
Camera type: ov7251
Camera id: 2
Camera slot: 4We added an extra option to
/usr/bin/qrb5165-configure-cameras
:17) # Hires(imx214) + Hires2(imx412) + Tracking(ov7251) CAM_LIST+=("hires2:imx412:0") CAM_LIST+=("hires:imx214:1") CAM_LIST+=("tracking:ov7251:2") BIN_LIST+=("/usr/share/modalai/chi-cdk/ov7251/com.qti.sensormodule.ov7251_4.bin") BIN_LIST+=("/usr/share/modalai/chi-cdk/imx412/com.qti.sensormodule.imx412_2.bin") BIN_LIST+=("/usr/share/modalai/chi-cdk/imx214/com.qti.sensormodule.imx214_3.bin") ;;
So the camera name, type and id are used in the
CAM_LIST+=
part and the camera type and the camera slot are used in theBIN_LIST+=
part.Then we can execute
voxl-configure-cameras 17
which places the camera drivers in the correct places and creates a defaultvoxl-camera-server.conf
file. Then when we dovoxl-camera-server --list
, all the cameras will appear.Also small note that you need to update the file
/usr/bin/qrb5165-print-camera-configs
if you want the option to be shown when executingvoxl-configure-cameras --help
@ModalAI, is this the correct way to approach this or is this not recommended because problems are expected when using it this way?
@brycek I think you can adjust it to match your situation
-
voxl_vision_hub.h missing
I installed SDK 1.0.0 on a VOXL2 and during compiling of our own code I get the following error:
fatal error: voxl_vision_hub.h: No such file or directory #include <voxl_vision_hub.h> ^~~~~~~~~~~~~~~~~~~ compilation terminated.
I checked the folder
/usr/include/
and the file is missing over there:voxl2:~$ ls /usr/include/voxl_* /usr/include/voxl_common_config.h /usr/include/voxl_imu_server.h /usr/include/voxl_qvio_server.h /usr/include/voxl_vision_px4.h /usr/include/voxl_cutils.h /usr/include/voxl_io.h /usr/include/voxl_slpi_uart_bridge_interface.h /usr/include/voxl_io: i2c.h other.h spi.h /usr/include/voxl_trajectory: setpoint_position.h trajectory_evaluation.h trajectory_interface.h trajectory_protocol.h trajectory_utils.h voxl_trajectory.h
I believe this is caused because the new header file is not set as public header in this file: https://gitlab.com/voxl-public/voxl-sdk/services/voxl-vision-hub/-/blob/master/src/CMakeLists.txt#L32
I would expect that when that is changed to
set_target_properties(voxl-vision-hub PROPERTIES PUBLIC_HEADER "../include/voxl_vision_px4.h;../include/voxl_vision_hub.h")
it will work.
Is that correct?
-
Can't install ros-melodic-tf2-geometry-msgs with voxl-suite 1.0.0
I'm installing the new SDK 1.0.0 which has voxl-suite 1.0.0. When trying to install
ros-melodic-tf2-geometry-msgs
I get the following error:voxl2:~$ apt install -y ros-melodic-tf2-geometry-msgs Reading package lists... Done Building dependency tree Reading state information... Done Some packages could not be installed. This may mean that you have requested an impossible situation or if you are using the unstable distribution that some required packages have not yet been created or been moved out of Incoming. The following information may help to resolve the situation: The following packages have unmet dependencies: ros-melodic-tf2-geometry-msgs : Depends: ros-melodic-orocos-kdl but it is not going to be installed Depends: ros-melodic-python-orocos-kdl but it is not going to be installed E: Unable to correct problems, you have held broken packages.
I dove into it a little bit more and the issue is that
ros-melodic-tf2-geometry-msgs
depends onros-melodic-orocos-kdl
which in turn depends onlibeigen3-dev
.But
voxl-suite
now depends onvoxl-feature-tracker
which depends onvoxl-eigen3
andvoxl-eigen3
conflicts withlibeigen3-dev
:voxl2:~$ apt show voxl-eigen3 Package: voxl-eigen3 Version: 3.4.0 Priority: optional Section: base Maintainer: matt.turi@modalai.com Installed-Size: unknown Conflicts: libeigen3-dev Replaces: libeigen3-dev, lib32-libeigen Download-Size: 1014 kB APT-Manual-Installed: yes APT-Sources: file:/data/voxl-suite-offline-packages ./ Packages Description: Eigen3 Library packaged up for APQ
What is the recommended way to get
ros-melodic-tf2-geometry-msgs
installed? In our case we don't needvoxl-feature-tracker
. Is there a way to combine it withvoxl-eigen3
? Or should we just not usevoxl-suite
? -
RE: Random PX4 Mavlink Disconnected
@tom Yes, thanks! That is a good idea. I will do that.
I was looking at the release page of the system image and saw this line with 1.5.3 internal release:HLOS Added service file that changes SLPI restart level to avoid board crash when SLPI crashes
What is SLPI and could you describe a little bit more what kind of issues you saw when SLPI crashed?
I also sometimes sawpx4io io_reg_get(4,0,15): data error -5
in the logging so maybe that could be related? -
Random PX4 Mavlink Disconnected
Hi,
We have an issue where sometimes the drone drops out of the air. Looking at the
journalctl
logging we see aWARNING: PX4 Mavlink Disconnected
message at the time when it drops. Also, a few seconds before it has disconnected we see the messagekernel: ==> rtl8188e_iol_efuse_patch
. Has anyone else experienced this or has an idea what could be causing this?Also, I saw that there are two ways to get the
WARNING: PX4 Mavlink Disconnect
. One with a disconnect callback and one with a timeout. Would it be an idea to tell in the logging message of the timeout that the disconnect is caused by a timeout?This is the software we have running:
-------------------------------------------------------------------------------- system-image: 1.5.2-M0054-14.1a-perf-nightly-20221219 kernel: #1 SMP PREEMPT Mon Dec 19 02:17:54 UTC 2022 4.19.125 -------------------------------------------------------------------------------- hw version: M0054 -------------------------------------------------------------------------------- voxl-suite: 0.9.5 -------------------------------------------------------------------------------- PHIL:~$ voxl-inspect-services -v Service Name | Version | Enabled | Running | CPU Usage --------------------------------------------------------------------------- docker-autorun | 1.2.4 | Disabled | Not Running | docker-daemon | 1.2.4 | Disabled | Not Running | modallink-relink | 0.16.1 | Disabled | Not Running | voxl-camera-server | 1.3.7 | Enabled | Running | 0.0 voxl-cpu-monitor | 0.3.0 | Enabled | Running | 0.0 voxl-dfs-server | 0.1.0 | Disabled | Not Running | voxl-imu-server | 0.5.0 | Enabled | Running | 0.0 voxl-mavlink-server | 0.3.0 | Enabled | Running | 0.0 voxl-modem | 0.16.1 | Disabled | Not Running | voxl-portal | 0.5.0 | Disabled | Not Running | voxl-px4-imu-server | 0.1.2 | Disabled | Not Running | voxl-px4 | 1.12.31 | Enabled | Running | 0.0 voxl-qvio-server | 0.8.2 | Enabled | Running | 0.0 voxl-remote-id | 0.0.5 | Disabled | Not Running | voxl-softap | 0.1.5 | Disabled | Not Running | voxl-static-ip | 0.1.5 | Disabled | Not Running | voxl-streamer | 0.4.1 | Disabled | Not Running | voxl-tag-detector | 0.0.4 | Disabled | Not Running | voxl-tflite-server | 0.3.1 | Disabled | Not Running | voxl-time-sync | 1.2.2 | Disabled | Not Running | voxl-vision-px4 | 1.4.0 | Enabled | Running | 0.0 voxl-wait-for-fs | 1.2.2 | Enabled | Completed |
-
PX4 firmware version
As I understand we need to use the modified PX4 firmware provided by modalai on https://docs.modalai.com/flight-core-firmware/. That is currently based on version 1.11. Is there already an estimate of when version 1.12 or 1.13 will be released? I need to use a parameter which was introduced in version 1.12 so I'm curious to know.
Also, what is expected if the original PX4 firmware (1.12 or 1.13) is used? I see some people on this forum using it and sometimes it is disregarded but sometimes you seem to be interested in the results. Could it work? Would be great to have a little bit more information on that.
And if you are still in the testing phase of a new firmware version, is it possible to use some beta versions?
-
RE: PX4 v1.12 for VOXL Flight
I'm looking to use a parameter which was introduced in PX4 v1.12. So I'm also interested in when PX4 v1.12 for VOXL Flight will be released. Do you have any estimate on that?
-
RE: 'ERROR failed to set scheduler' after restart of voxl-qvio-server
Is there any update on this issue?
-
RE: 'ERROR failed to set scheduler' after restart of voxl-qvio-server
I was logged in through SSH. But we also have our own blowup handler running on the drone which can execute
systemctl restart voxl-qvio-server
when it has detected a blowup. Then we are not connected through SSH or ADB but it will also fail to set the scheduler to FIFO. And I'm not sure if the voxl-qvio-server is restarted by ModalAI software but then I think it is the same result. -
'ERROR failed to set scheduler' after restart of voxl-qvio-server
When voxl-qvio-server is started, it sets itself to use the FIFO scheduler with high priority. This is done here: https://gitlab.com/voxl-public/voxl-sdk/services/voxl-qvio-server/-/blob/master/server/main.cpp#L973
When the drone boots, this is successful the first time.
journalctl -u voxl-qvio-server
reports:Jan 01 00:00:08 Drone_201 voxl-qvio-server[2552]: setting scheduler Jan 01 00:00:08 Drone_201 voxl-qvio-server[2552]: set FIFO priority successfully!
Now when I execute
systemctl restart voxl-qvio-server
it is unable to set itself to use the FIFO scheduler with high priority.journalctl -u voxl-qvio-server
reports:Jul 01 08:23:52 Drone_201 voxl-qvio-server[11943]: WARNING Failed to set priority, errno = 1 Jul 01 08:23:52 Drone_201 voxl-qvio-server[11943]: This seems to be a problem with ADB, the scheduler Jul 01 08:23:52 Drone_201 voxl-qvio-server[11943]: should work properly when this is a background process Jul 01 08:23:52 Drone_201 voxl-qvio-server[11943]: ERROR failed to set scheduler
It seems that this is similar to the issue reported here: https://bugzilla.redhat.com/show_bug.cgi?id=1467919.
What can we do to make sure that voxl-qvio-server is always running with the FIFO scheduler?
Version information:
yocto:~$ opkg list | grep voxl libvoxl_cutils - 0.0.2 - ModalAI's c utils libvoxl_io - 0.5.4 - ModalAI library allowing apps processor access to accessory serial ports voxl-camera-calibration - 0.0.1 - On-board camera calibration for VOXL voxl-camera-server - 0.8.1 - publishes camera frames over named pipe interface voxl-cpu-monitor - 0.1.7 - publishes CPU Data over MPA pipe and provides fan tools voxl-docker-support - 1.1.3 - tools to improve the usability of docker on VOXL voxl-imu-server - 0.8.1 - VOXL IMU interface for Modal Pipe Architecture voxl-mavlink - 0.0.2 - mavlink headers voxl-mpa-tools - 0.2.7 - misc tools for modal pipe architecture voxl-nodes - 0.1.7 - ROS nodes supported by ModalAI voxl-portal - 0.1.2 voxl-qvio-server - 0.3.1 - publishes QVIO data over named pipe interface voxl-qvio-server - 0.3.4 voxl-streamer - 0.2.3 - Gstreamer-based application to handle RTSP streaming voxl-tag-detector - 0.0.2 - Detect apriltags for MPA voxl-tflite - 0.0.1 - 64-bit tensorflow lite libraries voxl-tflite-server - 0.1.1 - client of voxl-camera-server that does deep learning (object detection, monocular depth estimation) voxl-utils - 0.8.5 voxl-vision-px4 - 0.9.2 - Interface between VOXL's computer vision services and PX4
Things I already have figured out from looking at the link mentioned above.
When the drone starts up the voxl-qvio-server task lives in the root cgroup:
yocto:~$ cat /sys/fs/cgroup/cpu,cpuacct/tasks | grep $(pidof voxl-qvio-server) 2514
Therefore it uses the realtime runtime budget of the root. This is 0.95 seconds per second (default values):
yocto:~$ cat /sys/fs/cgroup/cpu,cpuacct/cpu.rt_runtime_us 950000
It needs a realtime runtime budget bigger than 0 to be able to set the scheduler to FIFO so this is good.
Now when I execute
systemctl restart voxl-qvio-server
things change. The task doesn't live in the root cgroup anymore:yocto:~$ cat /sys/fs/cgroup/cpu,cpuacct/tasks | grep $(pidof voxl-qvio-server) yocto:~$
But now it lives in a new group:
yocto:~$ cat /sys/fs/cgroup/cpu,cpuacct/system.slice/voxl-qvio-server.service/tasks | grep $(pidof voxl-qvio-server) 13562
but this new group doesn't have any realtime runtime budget:
yocto:~$ cat /sys/fs/cgroup/cpu,cpuacct/system.slice/voxl-qvio-server.service/cpu.rt_runtime_us 0
and therefore it is unable to set the scheduler to FIFO with high priority. This is also mentioned in https://www.kernel.org/doc/Documentation/scheduler/sched-rt-group.txt at section 2.2:
By default all bandwidth is assigned to the root group and new groups get the
period from /proc/sys/kernel/sched_rt_period_us and a run time of 0. If you
want to assign bandwidth to another group, reduce the root group's bandwidth
and assign some or all of the difference to another group.If I manually assign bandwidth/realtime runtime budget to the voxl-qvio-server group it is able to set the scheduler to FIFO with high priority
echo 550000 > /sys/fs/cgroup/cpu,cpuacct/cpu.rt_runtime_us echo 200000 > /sys/fs/cgroup/cpu,cpuacct/system.slice/cpu.rt_runtime_us echo 200000 > /sys/fs/cgroup/cpu,cpuacct/system.slice/voxl-qvio-server.service/cpu.rt_runtime_us
I tried to script this and add it to the service but then it fails at the first startup because then
/sys/fs/cgroup/cpu,cpuacct/system.slice/voxl-qvio-server.service/cpu.rt_runtime_us
doesn't exist yet. Maybe it is possible to add it conditionally but I wasn't able to get it working robust yet. This link looks also interesting: https://lists.freedesktop.org/archives/systemd-devel/2017-July/039353.html. But at this point I thought it was better to ask on this forum if you are aware of this problem and maybe already have a solution for this.My service file is this:
yocto:~$ cat /etc/systemd/system/voxl-qvio-server.service # # Copyright (c) 2021 ModalAI, Inc. # [Unit] Description=voxl-qvio-server SourcePath=/usr/bin/voxl-qvio-server After=voxl-wait-for-fs.service Requires=voxl-wait-for-fs.service [Service] User=root Type=simple PIDFile=/run/voxl-qvio-server.pid ExecStart=/usr/bin/voxl-qvio-server [Install] WantedBy=multi-user.target
I hope you can help me out. I think it is important that voxl-qvio-server is always running with the FIFO scheduler and high priority.