Ok, in a stroke of epipahny I realised that when switched to offboard mode through ROS2, the drone attempts to do both the modalAI "figure eight" that was configured in /etc/modalai/voxl-vision-hub.conf and satisfy the TrajectorySetpoint given to it by ROS2, resulting in this very jerky motion. Switching offboard_mode parameter to "off" solved this issue!! Works flawlessly now.
Any ideas how I can switch off passing the local odometry to PX4 and pass /vvhub_body_wrt_fixed instead? If I just publish to /fmu/in/visual_vehicle_odometry does it only take input from the publisher or will it publish from both sources similar to the offboard mode example? :)))