Hi @masa-chau,
It seems you are on the right track. A few pointers from my side:
First, what are you actually trying to achieve? If you can avoid tracking in 3D and perform the task in 2D, I would strongly recommend doing that. Recovering 3D position from 2D observations is quite hard and can be very finicky/unreliable. For many tracking tasks, you can get away with tracking entirely in image space.
If you really need to track in 3D:
Use the PX4 estimator output. VIO is fused into EKF2 in PX4, so I would probably use the PX4 estimated poses/velocities as the state of the drone rather than trying to fuse the raw VIO yourself. I believe these should be available in ROS 2 through voxl-microdds-agent, although someone from ModalAI should confirm the exact topics/data being exposed.
Your 2D observations from the object detector are fine, but pay close attention to latency. Also, I would not expect reliable detection at very long distances with the Starling 2 Max fisheye camera; in practice, the useful detection range may be significantly below what you might initially expect.
I would not rely on the ToF camera for this.
For the actual tracking algorithm, I would look at an EKF or MAP estimator that fuses the 2D observations (e.g. bearing and potentially bounding-box size) with the drone's 3D pose/velocity. The important point is that a single 2D observation does not give you reliable depth. To estimate the object's distance over time, you need additional constraints, typically through a motion model or assumptions about how the tracked object moves.
Whether the object is flying or driving, you will probably need to make some approximation about its motion. I would look into the existing literature on monocular 3D object tracking / bearing-only tracking for this part.
Good luck!
