I tried swapping the tof camera with a spare one I had around. The spare tof worked immediately without issue. I then disconnected the spare tof, configured the camera server with only hires and tracking (with no connected tof camera, which worked fine). I then tried to plug back in the original tof and it started working again. So nothing changed, just unplugged and plugged back the tof camera and reflected the changes by configuring the camera server accordingly, weird...