"Serial Port Closed!" and other HITL Errors

labkit

I have been attempting to follow this HITL guide in order to perform simulations on my Starling 2 drone. The tutorial requires a USB to JST cable, which I received yesterday. It also uses Modal AI's voxl-gazebo-docker Docker image, which I installed a few weeks ago. Some specifics about the hardware and software I'm using are as follows.

Drone:
SKU: MRB-D0005-4-V3-C6-T8
voxl-suite 1.2.0
system-image: 1.7.6-M0054-14.1a-perf

Cable:
MCBL-00091-1

On PC:
Ubuntu: 22.04
Docker Engine - Community: 26.1.3
voxl-gazebo-docker: 1.2

And here is the output of voxl-inspect-services as requested by the forum guidelines.

I have plugged the JST end of the HITL connection cable to this point on the VOXL2 circuit board, which I believe to be J18. It is the only JST connector on the VOXL2 board. To insert the HITL connection cable, I had to remove another cable that previously connected J18 on VOXL2 to the 4-in-1 ESC.

When I try to follow the instructions in the HITL guide to run Gazebo in headless mode, I experience the following messages, which continue to repeat "Serial Port Closed!" until I use Ctrl-C. This occurs after I enter "docker run --rm -it --net=host --privileged -v /dev/input:/dev/input:rw voxl-gazebo-docker ./run_headless.bash"

I receive a different error when trying to run without headless mode.

The error when running without headless mode has been present since I first starting trying to use HITL, but the error when running headless mode only occurred persistently after I restarted the computer and also allowed the system to update software. Instead, I used to be able to get to the part of the simulation where I ran "voxl-px4-hitl." I received quite a bit of terminal output after running that command including these errors.

The qshell commands did not work properly after HITL had been set up in this way, malfunctioning in various ways. For example, qshell commander arm returned INFO [muorb] SLPI: Takeoff detected and then ceased to respond.

To troubleshoot the serial port error, I have checked the /dev/input directory.

The contents of this directory do not change when the drone is plugged into the computer or unplugged.
In the /dev directory however, ttyUSB0 does appear when the drone is plugged in.

I tried to use the ttyUSB0 instead of input with the command "docker run --rm -it --net=host --privileged -v /dev/input:/dev/input:rw -v /dev/ttyUSB0:/dev/ttyUSB0 voxl-gazebo-docker ./run_headless.bash"

I got a slightly different error this time.

I also ran a simple script to test whether it was able to access the serial port.

It was able to successfully access the port. So the issue does not appear to be that the serial port is inaccessible to all apps.

Thanks in advance if you're able to provide any insight on how to get the HITL simulation up and running.

Eric Katzfey

@labkit For headless mode try docker run --rm -it --net=host --privileged -v /dev/input:/dev/input:rw voxl-gazebo-docker and then instead of trying to launch Gazebo you will just be at the command prompt. You can ls /dev/ttyUSB0 there to make sure it is available. If it looks good then you can run headless gazebo with ./run_headless.bash. Don't worry about the sdf warnings.

Eric Katzfey

@labkit For GUI mode you can do something similar. Start with xhost + and then launch the docker with docker run --rm -it --net=host --privileged -e DISPLAY=$DISPLAY -v /dev/input:/dev/input:rw -v /tmp/.X11-unix:/tmp/.X11-unix:ro voxl-gazebo-docker. Check for /dev/ttyUSB0. Then run gazebo with ./run_gazebo.bash. Then, on VOXL 2, check qshell dsp_hitl status. You are starting px4 with the voxl-px4-hitl command, correct?

labkit

@Eric-Katzfey Yes, I start px4 with voxl-px4-hitl. Here's the results of the qshell dsp_hitl status command called on the PX4 command prompt after SSHing to the Starling. I assume all the values are set to 0 because the "Serial port closed!" error is preventing Gazebo from sending it data.

On the laptop, I used Docker's virtual file sharing to share /tmp/.X11-unix with docker, so the original GUI mode error seems to be resolved. (The one in which it couldn't acess that directory.)

By using your instructions, I can get to the docker command prompt in both GUI and headless modes. However, /dev/ttyUSB0 is not present here, although it can still be found from the Linux command line. Running Gazebo from the Docker command line still results in the serial port closed error.

Here's the attempt to run in GUI mode. Headless mode has the same issue.

Eric Katzfey

@labkit Okay, interesting. Here's what I get on my machine:

modalai@modalai-XPS-15-9570:~$ ls /dev/ttyUSB0 
/dev/ttyUSB0
modalai@modalai-XPS-15-9570:~$ docker run --rm -it --net=host --privileged -v /dev/input:/dev/input:rw voxl-gazebo-docker
root@modalai-XPS-15-9570:/usr/workspace/voxl2_hitl_gazebo# ls /dev/ttyUSB0 
/dev/ttyUSB0

Eric Katzfey

@labkit Maybe a groups / permission issue? Here's some more information from my setup:

root@modalai-XPS-15-9570:/usr/workspace/voxl2_hitl_gazebo# ls -l /dev/ttyUSB0
crw-rw---- 1 root dialout 188, 0 May 31 16:01 /dev/ttyUSB0
root@modalai-XPS-15-9570:/usr/workspace/voxl2_hitl_gazebo# groups
root
root@modalai-XPS-15-9570:/usr/workspace/voxl2_hitl_gazebo# exit
exit
modalai@modalai-XPS-15-9570:~$ ls -l /dev/ttyUSB0
crw-rw---- 1 root dialout 188, 0 May 31 09:00 /dev/ttyUSB0
modalai@modalai-XPS-15-9570:~$ groups
modalai adm dialout cdrom sudo dip plugdev lpadmin lxd sambashare docker
modalai@modalai-XPS-15-9570:~$

Perhaps you need to be in dialout group?

labkit

@Eric-Katzfey Here are my results with the same commands.

I ran docker in the dialout group but it didn't seem to make a difference. I was already in root.

As you can see, much of the contents of /dev is missing in docker compared to what's available from the linux command line. It's not only ttyUSB0.

labkit

I got Docker to avoid the error by doing the steps in the tutorial in a different order with setup on the drone coming first. This must have been why I didn't receive the "serial port closed" error when I first tried to set up the simulation two days ago. However, I don't think the simulation is in shape to run yet, all I've managed is to get back to where I was originally....

labkit

@Eric-Katzfey I have figured out why I thought that the drone wasn't giving me a "Serial port closed!" error a couple of days ago but has been doing so yesterday and today. When the gazebo docker container has not been created, the first attempt to create it will result in a "Serial port closed!" error. However, when the container is already running, future attempts to create it will not display the error, they will simply show the following message rather than creating a new container. At first I thought that this message meant that the connection to the drone had successfully avoided the "serial port closed" message.

The bad news is that this means that there may never have been a successful connection between the laptop and drone at all. A hardware misconfiguration can't be ruled out in this case. As I said, I had to remove a connection to the ESC 4 in 1 in order to connect the cable to the JST port on the VOXL2. If the connection's in the right place, it might be a permissions issue like you mentioned, although docker was run as privileged and I'm already in dialout.

Vinny

Hi @labkit
MCBL-00091-1 is by default meant for ESC tools...i.e.: it will plug into the ESC side of that cable you removed, not the VOXL 2 side:
https://docs.modalai.com/cable-datasheets/#mcbl-00091
To use it for HWIL, pins 2&3 need to be swapped as noted: https://docs.modalai.com/cable-userguides/#m0163-4-pin-jst-to-4-pin-jst-cross-over-example
You do not need to have the M0163 if you are comfortable re-pinning JSTs:
https://docs.modalai.com/cable-userguides/#flightcore-v2-conversion-cables-info

Hope this helps.
If you have already determined that and made the swap, then please ignore my message.
Thanks!

labkit

@Vinny I swapped pins 2&3 so that the former pin 2 is now pin 3 and vice versa. The swap occurred with no damage to the pins as far as I am aware, but the "Serial port closed" issue is still present. This image shows the correct rewiring, right? The RX and TX wires (white and green) are switched and the USB to JST is in J18 on the VOXL 2. The ESC is unplugged from VOXL 2.

Vinny

Hi @labkit
Looks good! At least now the efforts with Eric and team are not in vain since that is the correct MCBL-00091 config for HWIL.
You are in good hands with them now

labkit

@Eric-Katzfey I rewired the cable but the issue hasn't changed at all. It's still reporting "Serial port not found!" and the /dev/ttyUSB0 that you mentioned still isn't available from the Gazebo Docker command line.

Eric Katzfey

@labkit Is there another Linux computer that you can try? Many people use HITL and no one has reported any issues like this so I am suspecting that there is something about the environment on your machine that is causing the problem.

labkit

Also, ignore my earlier post that said I had fixed things by performing the tutorial steps in a different order. I was mistaken in thinking I had resolved the original issue. The next message explained why I made that mistake.

labkit

@Eric-Katzfey I will look into trying this.

labkit

@Eric-Katzfey Issue is mostly resolved. There were two main errors to fix. The first was that the cable that I was using needed to have its RX and TX wires swapped as Vinny pointed out. The second was that Docker Desktop needed to be uninstalled and replaced with only the core Docker CE Engine. After doing this, the HITL seems to be up and running on the alternate Linux computer I tested with. On the original computer, the "Serial port closed error" is no longer present although there does seem to be a graphics issue when attempting to run the Gazebo image with graphics.

Once this is fixed I think it should be all ready to perform simulations.