@Zachary-Lowell-0 as mentioned I have set offboard-mode in voxl-vision-hub.conf to "off".

I did a flight test shifting to offboard-mode,

The drone was able to execute the script in a stable way without any jittery motions or trying to come back to home position.

These are the flight logs,
https://review.px4.io/plot_app?log=e32a28df-cbba-4765-8499-6fe46e2ff890