USB3 Unstable

Nathan Sullivan 0

Good afternoon,

We're using the USB3 port on the LTE I/O expansion board, and have it connected via a hub to 4 devices - one storage and 3 cameras. It sometimes works, but the port is unreliable at best, and totally unusable most of the time. We've tried going through passive hubs, powered hubs, and a $400 optically decoupled hub - all yielding the same results; it appears to sort-of work until you actually go to use it, at which point everything downstream of that port just disappears. It will occasionally sporadically start coming back, but not reliably enough to do anything useful. Any ideas?

A week ago, this was all largely working, though the week before it was not. We never did figure out why it suddenly started working again, and certainly don't know why it stopped.

Thanks!

Vinny

Hi @Nathan-Sullivan-0
I'd like to help you here.
There are a few things I am thinking of off the top of my head, but I need a lot more details, so please bare with me and my questions:

Can you please provide a detailed list of what parts you are using and how you are using them?
(which hub, what devices, what modes (USB2/USB3), and are you doing self powered or bus powered for each item or some variation thereof). For self-powered, where is your power coming from?
How are you powering VOXL 2?
Can you please send a picture of your setup and indicate what cables you are using, including any off-the-shelf ones or ones from ModalAI, or ones you built yourself?
Do these issues persist on each item individually if you do not use the hub, or does this only happen with the downstream hub?
Does the issue with the hub happen when various configurations of the devices are used, or is it only in one specific configuration?
When you say they do not work, do they just not show up in lsusb, do not enumerate or even show some sort of power-on event, low data throughput? What exactly is the failure mode you observe?

My gut tells me a few things to focus on based on your results:

You may be overloading your hub, or the USB3 port we provide. Which is why it is important to know more details about the parts you are using
Your cables may be intermittent
Is it possible you are trying to use the GPIO that controls the Hub on our board and inadvertently setting the hub into reset mode? This is why I also want to know if you can use devices individually.

I do not know if having 3 UVC cameras is a factor here or not. But, before I tag someone from SW team, let's try to rule out the HW first.

Thanks!

Nathan Sullivan 0

@Vinny Sorry for the delay, the weather here has prohibited more flight tests until today, and we were trying a few things that we thought may have been related. I'm nearly 100% confident that this is a hardware issue, fwiw.

We're using a Voxl2, LTE I/O breakout board, USB2 from that board to an Alfa wireless (which is it's own set of problems), and three Sony ILX-LR1's, with custom wiring harnesses connecting their control ports to an ATmel 4809 board for control and feedback, which is in turn connected to the VOXL2 via the I/O board and UART. That piece is extremely reliable, and is the most consistent part of the entire system. I've attempted to use a whole host of different USB hubs, ranging from the mundane (typical Amazon low-profile non-powered unit) to the exotic (SeaLevel's optically isolated $400 USB monstrosity).
The VOXL, and only the VOXL, is powered from the typical VOXL power supply. We were also powering a few accessories (like the GPS) from this, but we've broken them all out onto separate power systems.
I can't send a picture here, but am happy to send one to an email address. The power cables, usb cables, etc., are ModalAI cables. The cables to our GPS receiver, ATmel board, and the entire downstream wiring harness on the cameras are made in-house. The cameras are also connected via USB, which seems to be much more reliable than the storage drive.
I can't actually check this, as multiple components require USB3. When I tried to getting creative with testing, and using dd to test write speed without any of the data generating components connected, it still had issues after a seemingly random amount of time.
The issue persists regardless of the number of cameras plugged in. If the drive is plugged directly in and not through the hub, it never actually connects at all, which I don't understand. Often the ModalAI board will either restart or disconnect the connected ADB client when a drive is plugged in, if that is plugged in while running.
They'll disappear from lsusb, then re-appear one by one, sometimes with all cameras and the drive coming back, sometimes with just one or two cameras coming back before everything drops off again.

When you say overload, do you mean in terms of data or power?

I've tried several sets of cables, and also plugged the entire thing into a raspberry pi, where it works flawlessly.

I'm not sure which GPIO that is, so that's entirely possible, but we're currently only using UART2.

The cameras, fwiw, are not really being used as cameras in the typical fashion that ModalAI systems use cameras, but rather are being used for still image collection on a frame-by-frame basis, and are not being used for any onboard image processing.

Thanks!

Vinny

Hi @Nathan-Sullivan-0
OK, I will try my best to give you some help, but generally speaking we have no experience with the Sony ILX-LR1 cameras and I'm thinking the issue is either driver related or the way you adapt our USB ports to the USB-C on the Sony camera.

I will contact you to see if we can help a little more, but I am not sure we can fully resolve an issue with an unsupported camera system. I would like to see a photo so I better understand your connections, so stay tuned and I will be in touch.

However,
One other tidbit I can offer (and this is all speculation based on experience, so please do not take this as anything other than advice) is to really check the source code of your Atmel. Most Arduino code I have seen misses the mark on all standard interfaces such as USB or UART, and introduces a ton of errors when paired with a validated system. For example, most Arduinos do not enumerate over USB correctly since they do not indicate the correct power modes, thus throwing off many Linux hosts, such as ours. This working on an Rpi does not have me concerned, since again, I think most open source implementations that are done out there start on Rpis or Arduinos, but when trying to work with a formal Ubuntu system, several inefficiencies are exposed and need to be corrected. I'm not sure if that is happening here, but there is a tool called usbview that can help you diagnose that: https://learn.microsoft.com/en-us/windows-hardware/drivers/debugger/usbview
There are linux versions as well: https://community.linuxmint.com/software/view/usbview

In that tool tool you can see what each device is reporting for its capabilities and needs, and if the sum of the needs end up being greater than the capabilities of the up-stream device, enumeration will fail or at best be flakey.

EDIT: To add, the reason why USBView is important is that you may find some devices are being reported as bus-powered when they are actually self-powered. Our Linux host actually does care about that nuance and will try to limit the number of bus-powered nodes to be <500mA if there is a path on the same bus-powered tree. If a hub correctly reports as "high power capable", and devices all report as "self powered", this will help to know.

Vinny

Hi @Nathan-Sullivan-0
I did reach out, but for the benefit of the other forum users, I did a little digging into the modes of the Sony ILX-LR1.

I'm wondering if there is a nuance between the camera being enabled as UVC device, but then if you are in still frame mode, sending it (very high resolution) JPEG/RAW/HEIF formats.

Maybe someone from SW can confirm, but UVC devices may only expect data in MPEG or other video formats. So, I wonder if the fact that your devices enumerate at first, but then the moment you try to use them in still image mode, things go wrong. This may be a data formatting issue.

Another thing to note: can you make sure these images come across USB at controlled ways at different times. You are trying to push a LOT of data into this one port, even those stills are at 15MB minimum. Not sure what frame rate you are using, but you may be saturating the USB link. Have you done a bandwidth analysis?
Our MCBL-00022-2 is not qualified for 10Gbps speeds, and for 5Gbps rates, we only qualified it for prototype use. Sustained very high throughput rates may require you to source a proper shielded USB3 to USB-C cable. We try to make that disclaimer clear on our Tech Docs page:

And the last point I want to highlight is the USB-C port on the Sony camera may be orientation dependent when not using it in USB-C modes. We are only USB-3 modes on our Hub, and I'm not sure what cable you are using to convert our USB-3 to the USB-C, but we have found that a lot of USB-C to "less-than-USB-C" cables (i.e.: USBC-to-USB3, USBC-to-USB2) actually have orientation specific insertions at the USB-C side if done cheaply. We have some in our own HQ that work like that (I'm trying to find them all and trash them )

Look forward to your email response.
Thanks!