Best Strategy for Injecting Photorealistic Rendering Outputs into VOXL2 Camera Pipeline in Gazebo HITL
-
Hi ModalAI community,
I'm working on a Gazebo HITL setup with VOXL2 and need to inject photorealistic rendering (GSplat, NeRF) outputs into the camera pipeline. I'm trying to figure out the best approach and would appreciate your input. I have followed the guide https://docs.modalai.com/voxl2-PX4-hitl/#hitl. To modify the existing docker contents, I copy a part of the file directory (/usr/workspace/voxl2_hitl_gazebo) outside the container. I do the necessary file edits there and initialize the docker with that directory mounted to the container and also compile/build the code changes within the docker.
My Setup:
- VOXL2 running HITL with Gazebo
- Current world: modalai.world with iris_hitl model
- Photorealistic rendering environment where you give a pose and get an image back
Goal: VOXL2 should use these rendered images as its camera output for VIO and other services
Current Understanding:
-VOXL2 has built-in VIO that processes camera images
-Current modalai.world has no camera sensor
-I want to bypass camera rendering and inject photorealistic images directly
-Need images to be available for multiple services (VIO, voxl-tflite-service, etc.)My Questions:
- Should I add a camera sensor to the world and attach a custom plugin? Or is there a cleaner way to inject images directly into the camera pipeline?
- If I inject rendered images directly into a camera plugin (modifying or creating a custom plugin), will VOXL2 still perform proper VIO? Or does it need actual camera sensor data?
- What's the minimal change needed? I want to avoid dealing with camera intrinsics, distortion models, and rendering pipelines.
- How do I make the injected images available to multiple VOXL2 services? I need VIO, voxl-tflite-service, and potentially other services to access the same camera feed.
- What is the specific plugin at /usr/workspace/voxl2_hitl_gazebo/build/ I need to modify for this?
-
Hi ModalAI community,
I'm working on a Gazebo HITL setup with VOXL2 and need to inject photorealistic rendering (GSplat, NeRF) outputs into the camera pipeline. I'm trying to figure out the best approach and would appreciate your input. I have followed the guide https://docs.modalai.com/voxl2-PX4-hitl/#hitl. To modify the existing docker contents, I copy a part of the file directory (/usr/workspace/voxl2_hitl_gazebo) outside the container. I do the necessary file edits there and initialize the docker with that directory mounted to the container and also compile/build the code changes within the docker.
My Setup:
- VOXL2 running HITL with Gazebo
- Current world: modalai.world with iris_hitl model
- Photorealistic rendering environment where you give a pose and get an image back
Goal: VOXL2 should use these rendered images as its camera output for VIO and other services
Current Understanding:
-VOXL2 has built-in VIO that processes camera images
-Current modalai.world has no camera sensor
-I want to bypass camera rendering and inject photorealistic images directly
-Need images to be available for multiple services (VIO, voxl-tflite-service, etc.)My Questions:
- Should I add a camera sensor to the world and attach a custom plugin? Or is there a cleaner way to inject images directly into the camera pipeline?
- If I inject rendered images directly into a camera plugin (modifying or creating a custom plugin), will VOXL2 still perform proper VIO? Or does it need actual camera sensor data?
- What's the minimal change needed? I want to avoid dealing with camera intrinsics, distortion models, and rendering pipelines.
- How do I make the injected images available to multiple VOXL2 services? I need VIO, voxl-tflite-service, and potentially other services to access the same camera feed.
- What is the specific plugin at /usr/workspace/voxl2_hitl_gazebo/build/ I need to modify for this?
@Ege-Yüceel Wow, this is a very ambitious plan! Keep in mind that VIO needs not only camera images but also IMU input. Camera images are normally supplied by the voxl-camera-server on MPA pipes. So, you would need to stop voxl-camera-server and start your own application that takes the images from Gazebo and places them into correctly named MPA pipes. The images would have to be formatted properly, etc. Likewise for IMU samples that normally come from voxl-imu-server. I think this will be quite difficult to get working properly but theoretically possible.
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login