@jklre , one way to achieve what you need is to set up a VPN client on VOXL2 (and VPN server on your linux server) and create a adb TCP bridge via adb forward or reverse (which you already found). Your VPN client would connect to the VPN server on your linux server via the ADB TCP tunnel.
The VPN client / server would then take care of proper data routing (just like regular VPN).
I believe guys from Ascend Engineering were able to to achieve this : https://ascend-engineering.gitbook.io/modalai/sharing-internet-over-adb but it requires initial internet connection to set up.
Your best bet may be to get all the dependencies downloaded to your linux server, then push them to voxl2 via ADB, and add a local package source for apt on voxl2 (so that it can find new packages) and install.
Similarly for python modules, you should be able to do an offline install, i think..
Alex