ModalAI Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Firmware Structure (Adding VTOL support) - Clarifications

    Ask your questions right here!
    2
    16
    120
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • G
      garret @garret
      last edited by garret

      @garret quick update from me. i was able to get "vtol_att_control" module to start along with all the fw_* and mc_* controllers. i now seem to have access to the necessary vtol parameters through QGC. However, the connecting to QGC is super slow now and also I am getting mag0: Timeout and baro0: Timeouterrors after running voxl-px4-hitl.

      here is the full boot up log just in case:

      voxl2:/$ voxl-px4-hitl
Found DSP signature file
INFO  [px4] mlockall() enabled. PX4's virtual address space is locked into RAM.
INFO  [px4] assuming working directory is rootfs, no symlinks needed.
INFO  [muorb] Got muorb init command
Sending initialization request
Got topic data before configuration complete
Got topic data before configuration complete
Got topic data before configuration complete
Got topic data before configuration complete
Got topic data before configuration complete
Got topic data before configuration complete
Got topic data before configuration complete
Got topic data before configuration complete
Got topic data before configuration complete
Got topic data before configuration complete
Got topic data before configuration complete
Got topic data before configuration complete
Got topic data before configuration complete
INFO  [muorb] SLPI: muorb aggregator thread running
INFO  [muorb] muorb protobuf initalize method succeeded
INFO  [muorb] succesfully did ADVERTISE_TEST_TYPE
INFO  [muorb] SLPI: Creating pthread test_MUORB

INFO  [muorb] SLPI: Successfully created px4 task PX4_test_MUORB with tid 2097656
INFO  [muorb] succesfully did SUBSCRIBE_TEST_TYPE
INFO  [muorb] SLPI: Creating pthread test_MUORB

INFO  [muorb] SLPI: Successfully created px4 task PX4_test_MUORB with tid 2097655
INFO  [muorb] succesfully did TOPIC_TEST_TYPE
INFO  [muorb] SLPI: Creating pthread test_MUORB

INFO  [muorb] SLPI: Successfully created px4 task PX4_test_MUORB with tid 2097654
INFO  [muorb] succesfully did UNSUBSCRIBE_TEST_TYPE
INFO  [muorb] SLPI: Creating pthread test_MUORB

INFO  [muorb] SLPI: Successfully created px4 task PX4_test_MUORB with tid 2097653
INFO  [muorb] muorb test passed

______  __   __    ___ 
| ___ \ \ \ / /   /   |
| |_/ /  \ V /   / /| |
|  __/   /   \  / /_| |
| |     / /^\ \ \___  |
\_|     \/   \/     |_/

px4 starting.

INFO  [parameters] Starting param sync THREAD
INFO  [muorb] SLPI: Advertising remote topic log_message
INFO  [px4] startup script: /bin/sh /usr/bin/voxl-px4-hitl-start 0
INFO  [muorb] SLPI: Starting param sync THREAD
Running on M0054
INFO  [muorb] SLPI: before starting the qshell_entry task
INFO  [muorb] SLPI: Creating pthread qshell

INFO  [muorb] SLPI: qshell entry.....
INFO  [muorb] SLPI: Successfully created px4 task PX4_qshell with tid 2097652
INFO  [muorb] SLPI: Init app map initialized
INFO  [muorb] SLPI: after starting the qshell_entry task
INFO  [param] selected parameter default file /data/px4/param/hitl_parameters
INFO  [muorb] SLPI: Marking DeviceNode(parameter_client_reset_request) as advertised in process_remote_topic
INFO  [uORB] Marking DeviceNode(parameter_client_reset_response) as advertised in process_remote_topic
INFO  [muorb] SLPI: Advertising remote topic parameter_update
INFO  [muorb] SLPI: Marking DeviceNode(parameter_client_set_value_request) as advertised in process_remote_to
INFO  [uORB] Marking DeviceNode(parameter_server_set_used_request) as advertised in process_remote_topic
INFO  [muorb] SLPI: Marking DeviceNode(parameter_server_set_used_response) as advertised in process_remote_to
INFO  [uORB] Marking DeviceNode(parameter_client_set_value_response) as advertised in process_remote_topic
INFO  [parameters] BSON document size 2018 bytes, decoded 2018 bytes (INT32:53, FLOAT:43)
INFO  [logger] logger started (mode=all)
INFO  [logger] Start file log (type: full)
INFO  [logger] [logger] /data/px4/log/2025-07-02/19_36_21.ulg	
INFO  [muorb] SLPI: Advertising remote topic mavlink_log
INFO  [muorb] SLPI: Advertising remote topic event
INFO  [logger] Opened full log file: /data/px4/log/2025-07-02/19_36_21.ulg
INFO  [muorb] SLPI: Advertising remote topic logger_status
INFO  [qshell] Send cmd: 'sensors start -h'
INFO  [muorb] SLPI: Marking DeviceNode(qshell_req) as advertised in process_remote_topic
INFO  [muorb] SLPI: qshell gotten: sensors start -h
INFO  [muorb] SLPI:   arg0 = 'sensors'

INFO  [muorb] SLPI:   arg1 = 'start'

INFO  [muorb] SLPI:   arg2 = '-h'

INFO  [uORB] Advertising remote topic sensor_selection
INFO  [uORB] Advertising remote topic sensors_status_imu
INFO  [uORB] Advertising remote topic vehicle_acceleration
INFO  [uORB] Advertising remote topic vehicle_angular_velocity
INFO  [uORB] Advertising remote topic sensor_combined
INFO  [uORB] Advertising remote topic vehicle_air_data
INFO  [uORB] Advertising remote topic vehicle_gps_position
INFO  [uORB] Advertising remote topic vehicle_magnetometer
INFO  [uORB] Advertising remote topic sensor_preflight_mag
INFO  [muorb] SLPI: Ok executing command: sensors start -h
INFO  [uORB] Advertising remote topic qshell_retval
INFO  [qshell] qshell return value timestamp: 70283703, local time: 70289506
INFO  [qshell] Send cmd: 'ekf2 start'
INFO  [muorb] SLPI: Marking DeviceNode(qshell_req) as advertised in process_remote_topic
INFO  [muorb] SLPI: qshell gotten: ekf2 start
INFO  [muorb] SLPI:   arg0 = 'ekf2'

INFO  [muorb] SLPI:   arg1 = 'start'

INFO  [uORB] Advertising remote topic vehicle_attitude
INFO  [uORB] Advertising remote topic vehicle_local_position
INFO  [uORB] Advertising remote topic estimator_event_flags
INFO  [uORB] Advertising remote topic estimator_innovation_test_ratios
INFO  [uORB] Advertising remote topic estimator_innovation_variances
INFO  [uORB] Advertising remote topic estimator_innovations
INFO  [uORB] Advertising remote topic estimator_sensor_bias
INFO  [uORB] Advertising remote topic estimator_states
INFO  [uORB] Advertising remote topic estimator_status_flags
INFO  [uORB] Advertising remote topic estimator_status
INFO  [muorb] SLPI: Ok executing command: ekf2 start
INFO  [qshell] qshell return value timestamp: 70697060, local time: 70701423
INFO  [qshell] Send cmd: 'mc_pos_control start'
INFO  [muorb] SLPI: Marking DeviceNode(qshell_req) as advertised in process_remote_topic
INFO  [muorb] SLPI: qshell gotten: mc_pos_control start
INFO  [muorb] SLPI:   arg0 = 'mc_pos_control'

INFO  [muorb] SLPI:   arg1 = 'start'

INFO  [uORB] Advertising remote topic takeoff_status
INFO  [muorb] SLPI: Ok executing command: mc_pos_control start
INFO  [qshell] qshell return value timestamp: 70881803, local time: 70884861
INFO  [qshell] Send cmd: 'fw_pos_control start'
INFO  [muorb] SLPI: Marking DeviceNode(qshell_req) as advertised in process_remote_topic
INFO  [muorb] SLPI: qshell gotten: fw_pos_control start
INFO  [muorb] SLPI:   arg0 = 'fw_pos_control'

INFO  [muorb] SLPI:   arg1 = 'start'

INFO  [uORB] Advertising remote topic position_controller_status
INFO  [uORB] Advertising remote topic position_controller_landing_status
INFO  [uORB] Advertising remote topic tecs_status
INFO  [uORB] Advertising remote topic launch_detection_status
INFO  [uORB] Advertising remote topic landing_gear
INFO  [uORB] Advertising remote topic flaps_setpoint
INFO  [uORB] Advertising remote topic spoilers_setpoint
INFO  [muorb] SLPI: Ok executing command: fw_pos_control start
INFO  [qshell] qshell return value timestamp: 71194603, local time: 71197800
INFO  [qshell] Send cmd: 'mc_att_control start'
INFO  [muorb] SLPI: Marking DeviceNode(qshell_req) as advertised in process_remote_topic
INFO  [muorb] SLPI: qshell gotten: mc_att_control start
INFO  [muorb] SLPI:   arg0 = 'mc_att_control'

INFO  [muorb] SLPI:   arg1 = 'start'

INFO  [muorb] SLPI: Ok executing command: mc_att_control start
INFO  [qshell] qshell return value timestamp: 71291921, local time: 71293922
INFO  [qshell] Send cmd: 'fw_att_control start'
INFO  [muorb] SLPI: Marking DeviceNode(qshell_req) as advertised in process_remote_topic
INFO  [muorb] SLPI: qshell gotten: fw_att_control start
INFO  [muorb] SLPI:   arg0 = 'fw_att_control'

INFO  [muorb] SLPI:   arg1 = 'start'

INFO  [muorb] SLPI: Ok executing command: fw_att_control start
INFO  [qshell] qshell return value timestamp: 71376777, local time: 71378810
INFO  [qshell] Send cmd: 'mc_rate_control start'
INFO  [muorb] SLPI: Marking DeviceNode(qshell_req) as advertised in process_remote_topic
INFO  [muorb] SLPI: qshell gotten: mc_rate_control start
INFO  [muorb] SLPI:   arg0 = 'mc_rate_control'

INFO  [muorb] SLPI:   arg1 = 'start'

INFO  [uORB] Advertising remote topic rate_ctrl_status
INFO  [muorb] SLPI: Ok executing command: mc_rate_control start
INFO  [qshell] qshell return value timestamp: 71542510, local time: 71544910
INFO  [qshell] Send cmd: 'fw_rate_control start'
INFO  [muorb] SLPI: Marking DeviceNode(qshell_req) as advertised in process_remote_topic
INFO  [muorb] SLPI: qshell gotten: fw_rate_control start
INFO  [muorb] SLPI:   arg0 = 'fw_rate_control'

INFO  [muorb] SLPI:   arg1 = 'start'

INFO  [uORB] Marking DeviceNode(rate_ctrl_status) as advertised in process_remote_topic
INFO  [muorb] SLPI: Ok executing command: fw_rate_control start
INFO  [qshell] qshell return value timestamp: 71685112, local time: 71688487
INFO  [qshell] Send cmd: 'vtol_att_control start'
INFO  [muorb] SLPI: Marking DeviceNode(qshell_req) as advertised in process_remote_topic
INFO  [muorb] SLPI: qshell gotten: vtol_att_control start
INFO  [muorb] SLPI:   arg0 = 'vtol_att_control'

INFO  [muorb] SLPI:   arg1 = 'start'

INFO  [uORB] Marking DeviceNode(flaps_setpoint) as advertised in process_remote_topic
INFO  [uORB] Marking DeviceNode(spoilers_setpoint) as advertised in process_remote_topic
INFO  [uORB] Advertising remote topic vtol_vehicle_status
INFO  [uORB] Advertising remote topic vehicle_thrust_setpoint
INFO  [uORB] Advertising remote topic vehicle_torque_setpoint
INFO  [uORB] Marking DeviceNode(vehicle_thrust_setpoint) as advertised in process_remote_topic
INFO  [uORB] Marking DeviceNode(vehicle_torque_setpoint) as advertised in process_remote_topic
INFO  [muorb] SLPI: Ok executing command: vtol_att_control start
INFO  [qshell] qshell return value timestamp: 71820328, local time: 71823028
INFO  [qshell] Send cmd: 'mc_hover_thrust_estimator start'
INFO  [muorb] SLPI: Marking DeviceNode(qshell_req) as advertised in process_remote_topic
INFO  [muorb] SLPI: qshell gotten: mc_hover_thrust_estimator start
INFO  [muorb] SLPI:   arg0 = 'mc_hover_thrust_estimator'

INFO  [muorb] SLPI:   arg1 = 'start'

INFO  [muorb] SLPI: Ok executing command: mc_hover_thrust_estimator start
INFO  [qshell] qshell return value timestamp: 71866787, local time: 71869655
INFO  [qshell] Send cmd: 'mc_autotune_attitude_control start'
INFO  [muorb] SLPI: Marking DeviceNode(qshell_req) as advertised in process_remote_topic
INFO  [muorb] SLPI: qshell gotten: mc_autotune_attitude_control start
INFO  [muorb] SLPI:   arg0 = 'mc_autotune_attitude_control'

INFO  [muorb] SLPI:   arg1 = 'start'

INFO  [uORB] Advertising remote topic autotune_attitude_control_status
INFO  [muorb] SLPI: Ok executing command: mc_autotune_attitude_control start
INFO  [qshell] qshell return value timestamp: 71922119, local time: 71924972
INFO  [qshell] Send cmd: 'land_detector start multicopter'
INFO  [muorb] SLPI: Marking DeviceNode(qshell_req) as advertised in process_remote_topic
INFO  [muorb] SLPI: qshell gotten: land_detector start multicopter
INFO  [muorb] SLPI:   arg0 = 'land_detector'

INFO  [muorb] SLPI:   arg1 = 'start'

INFO  [muorb] SLPI:   arg2 = 'multicopter'

INFO  [muorb] SLPI: Ok executing command: land_detector start multicopter
INFO  [qshell] qshell return value timestamp: 71981835, local time: 71985165
INFO  [qshell] Send cmd: 'manual_control start'
INFO  [muorb] SLPI: Marking DeviceNode(qshell_req) as advertised in process_remote_topic
INFO  [muorb] SLPI: qshell gotten: manual_control start
INFO  [muorb] SLPI:   arg0 = 'manual_control'

INFO  [muorb] SLPI:   arg1 = 'start'

INFO  [uORB] Advertising remote topic vehicle_land_detected
INFO  [muorb] SLPI: Ok executing command: manual_control start
INFO  [qshell] qshell return value timestamp: 72060721, local time: 72063250
INFO  [uORB] Advertising remote topic manual_control_setpoint
INFO  [qshell] Send cmd: 'control_allocator start'
INFO  [muorb] SLPI: Marking DeviceNode(qshell_req) as advertised in process_remote_topic
INFO  [muorb] SLPI: qshell gotten: control_allocator start
INFO  [muorb] SLPI:   arg0 = 'control_allocator'

INFO  [muorb] SLPI:   arg1 = 'start'

INFO  [uORB] Advertising remote topic control_allocator_status
INFO  [uORB] Marking DeviceNode(control_allocator_status) as advertised in process_remote_topic
INFO  [uORB] Advertising remote topic actuator_motors
INFO  [uORB] Advertising remote topic actuator_servos
INFO  [uORB] Advertising remote topic actuator_servos_trim
INFO  [muorb] SLPI: Ok executing command: control_allocator start
INFO  [qshell] qshell return value timestamp: 72577791, local time: 72580927
INFO  [qshell] Send cmd: 'rc_update start'
INFO  [muorb] SLPI: Marking DeviceNode(qshell_req) as advertised in process_remote_topic
INFO  [muorb] SLPI: qshell gotten: rc_update start
INFO  [muorb] SLPI:   arg0 = 'rc_update'

INFO  [muorb] SLPI:   arg1 = 'start'

INFO  [muorb] SLPI: Ok executing command: rc_update start
INFO  [qshell] qshell return value timestamp: 73009950, local time: 73012100
INFO  [qshell] Send cmd: 'commander start -h'
INFO  [muorb] SLPI: Marking DeviceNode(qshell_req) as advertised in process_remote_topic
INFO  [muorb] SLPI: qshell gotten: commander start -h
INFO  [muorb] SLPI:   arg0 = 'commander'

INFO  [muorb] SLPI:   arg1 = 'start'

INFO  [muorb] SLPI:   arg2 = '-h'

INFO  [muorb] SLPI: Creating pthread commander

INFO  [muorb] SLPI: Successfully created px4 task PX4_commander with tid 2097647
INFO  [uORB] Advertising remote topic led_control
INFO  [muorb] SLPI: LED: open /dev/led0 failed (22)
INFO  [uORB] Advertising remote topic tune_control
INFO  [muorb] SLPI: Ok executing command: commander start -h
INFO  [qshell] qshell return value timestamp: 73282818, local time: 73285620
INFO  [muorb] SLPI: Preflight Fail: Accel Sensor 0 missing
INFO  [muorb] SLPI: Preflight Fail: barometer 0 missing
INFO  [muorb] SLPI: Preflight Fail: ekf2 missing data
INFO  [muorb] SLPI: Preflight Fail: Gyro Sensor 0 missing
INFO  [muorb] SLPI: Preflight Fail: Compass Sensor 0 missing
INFO  [uORB] Advertising remote topic health_report
INFO  [uORB] Advertising remote topic failsafe_flags
INFO  [uORB] Advertising remote topic actuator_armed
INFO  [uORB] Advertising remote topic vehicle_control_mode
INFO  [uORB] Advertising remote topic vehicle_status
INFO  [uORB] Advertising remote topic failure_detector_status
INFO  [qshell] Send cmd: 'commander mode posctl'
INFO  [muorb] SLPI: Marking DeviceNode(qshell_req) as advertised in process_remote_topic
INFO  [muorb] SLPI: qshell gotten: commander mode posctl
INFO  [muorb] SLPI:   arg0 = 'commander'

INFO  [muorb] SLPI:   arg1 = 'mode'

INFO  [muorb] SLPI:   arg2 = 'posctl'

INFO  [uORB] Advertising remote topic vehicle_command
INFO  [muorb] SLPI: Ok executing command: commander mode posctl
INFO  [uORB] Advertising remote topic vehicle_command_ack
INFO  [qshell] qshell return value timestamp: 73321888, local time: 73324809
INFO  [qshell] Send cmd: 'load_mon start'
INFO  [muorb] SLPI: Marking DeviceNode(qshell_req) as advertised in process_remote_topic
INFO  [muorb] SLPI: qshell gotten: load_mon start
INFO  [muorb] SLPI:   arg0 = 'load_mon'

INFO  [muorb] SLPI:   arg1 = 'start'

INFO  [muorb] SLPI: Ok executing command: load_mon start
INFO  [uORB] Advertising remote topic cpuload
INFO  [qshell] qshell return value timestamp: 73355849, local time: 73358005
INFO  [qshell] Send cmd: 'flight_mode_manager start'
INFO  [muorb] SLPI: Marking DeviceNode(qshell_req) as advertised in process_remote_topic
INFO  [muorb] SLPI: qshell gotten: flight_mode_manager start
INFO  [muorb] SLPI:   arg0 = 'flight_mode_manager'

INFO  [muorb] SLPI:   arg1 = 'start'

INFO  [uORB] Marking DeviceNode(vehicle_command) as advertised in process_remote_topic
INFO  [muorb] SLPI: Ok executing command: flight_mode_manager start
INFO  [qshell] qshell return value timestamp: 73487516, local time: 73488991
INFO  [dataman] data manager file '/data/px4/dataman' size is 7866640 bytes
INFO  [muorb] SLPI: Advertising remote topic transponder_report
INFO  [muorb] SLPI: Advertising remote topic rtl_time_estimate
INFO  [muorb] SLPI: Advertising remote topic position_setpoint_triplet
INFO  [qshell] Send cmd: 'pwm_out_sim start -m hil'
INFO  [muorb] SLPI: Marking DeviceNode(qshell_req) as advertised in process_remote_topic
INFO  [muorb] SLPI: qshell gotten: pwm_out_sim start -m hil
INFO  [muorb] SLPI:   arg0 = 'pwm_out_sim'

INFO  [muorb] SLPI:   arg1 = 'start'

INFO  [muorb] SLPI:   arg2 = '-m'

INFO  [muorb] SLPI:   arg3 = 'hil'

INFO  [uORB] Advertising remote topic actuator_outputs
INFO  [muorb] SLPI: Ok executing command: pwm_out_sim start -m hil
INFO  [qshell] qshell return value timestamp: 74688971, local time: 74692095
INFO  [uORB] Advertising remote topic actuator_outputs_sim
INFO  [qshell] Send cmd: 'dsp_hitl start -g -m -o'
INFO  [muorb] SLPI: Marking DeviceNode(qshell_req) as advertised in process_remote_topic
INFO  [muorb] SLPI: qshell gotten: dsp_hitl start -g -m -o
INFO  [muorb] SLPI:   arg0 = 'dsp_hitl'

INFO  [muorb] SLPI:   arg1 = 'start'

INFO  [muorb] SLPI:   arg2 = '-g'

INFO  [muorb] SLPI:   arg3 = '-m'

INFO  [muorb] SLPI:   arg4 = '-o'

INFO  [muorb] SLPI: Creating pthread dsp_hitl__main

INFO  [muorb] SLPI: Successfully created px4 task PX4_dsp_hitl__main with tid 2097645
INFO  [muorb] SLPI: Ok executing command: dsp_hitl start -g -m -o
INFO  [muorb] SLPI: DSP HITL Serial Port is open: 0
INFO  [uORB] Advertising remote topic sensor_accel
INFO  [uORB] Advertising remote topic sensor_gyro
INFO  [muorb] SLPI: Got 5 from orb_subscribe
INFO  [qshell] qshell return value timestamp: 74734798, local time: 74736644
INFO  [uORB] Advertising remote topic esc_status
INFO  [mavlink] mode: Onboard, data rate: 100000 B/s on udp port 14556 remote port 14557
INFO  [muorb] SLPI: Advertising remote topic telemetry_status
INFO  [uORB] Advertising remote topic sensor_mag
INFO  [uORB] Advertising remote topic battery_status
ERROR [muorb] SLPI: Sending updates at 74781185, delta 45889
INFO  [uORB] Advertising remote topic sensor_baro
INFO  [uORB] Advertising remote topic sensor_gps
INFO  [uORB] Advertising remote topic vehicle_imu
INFO  [uORB] Advertising remote topic vehicle_imu_status
INFO  [uORB] Advertising remote topic home_position
INFO  [mavlink] partner IP: 127.0.0.1
INFO  [muorb] SLPI: Advertising remote topic obstacle_distance
INFO  [uORB] Advertising remote topic vehicle_odometry
INFO  [uORB] Advertising remote topic estimator_aid_src_fake_pos
INFO  [uORB] Advertising remote topic estimator_aid_src_fake_hgt
INFO  [uORB] Advertising remote topic estimator_aid_src_gravity
INFO  [uORB] Advertising remote topic ekf2_timestamps
INFO  [uORB] Marking DeviceNode(vehicle_thrust_setpoint) as advertised in process_remote_topic
INFO  [uORB] Marking DeviceNode(vehicle_thrust_setpoint) as advertised in process_remote_topic
INFO  [uORB] Marking DeviceNode(vehicle_torque_setpoint) as advertised in process_remote_topic
INFO  [uORB] Marking DeviceNode(vehicle_torque_setpoint) as advertised in process_remote_topic
INFO  [uORB] Marking DeviceNode(flaps_setpoint) as advertised in process_remote_topic
INFO  [uORB] Marking DeviceNode(spoilers_setpoint) as advertised in process_remote_topic
INFO  [uORB] Advertising remote topic vehicle_local_position_setpoint
INFO  [uORB] Advertising remote topic vehicle_attitude_setpoint
INFO  [uORB] Advertising remote topic trajectory_setpoint
INFO  [uORB] Advertising remote topic vehicle_constraints
INFO  [uORB] Marking DeviceNode(landing_gear) as advertised in process_remote_topic
INFO  [muorb] SLPI: Advertising remote topic offboard_control_mode
INFO  [muorb] SLPI: Advertising remote topic mission_result
INFO  [muorb] SLPI: Advertising remote topic timesync_status
INFO  [mavlink] mode: Normal, data rate: 100000 B/s on udp port 14558 remote port 14559
INFO  [muorb] SLPI: Marking DeviceNode(telemetry_status) as advertised in process_remote_topic
INFO  [uORB] Advertising remote topic vehicle_rates_setpoint
INFO  [uORB] Advertising remote topic landing_gear_wheel
INFO  [uORB] Advertising remote topic actuator_controls_status_0
INFO  [uORB] Marking DeviceNode(actuator_controls_status_0) as advertised in process_remote_topic
INFO  [uORB] Advertising remote topic sensors_status_mag
INFO  [uORB] Advertising remote topic sensors_status_baro
INFO  [uORB] Advertising remote topic estimator_aid_src_mag_heading
INFO  [uORB] Advertising remote topic estimator_aid_src_mag
INFO  [uORB] Advertising remote topic estimator_aid_src_baro_hgt
INFO  [uORB] Advertising remote topic estimator_gps_status
INFO  [uORB] Advertising remote topic estimator_aid_src_gnss_hgt
INFO  [uORB] Advertising remote topic estimator_aid_src_gnss_pos
INFO  [uORB] Advertising remote topic estimator_aid_src_gnss_vel
INFO  [mavlink] MAVLink only on localhost (set param MAV_{i}_BROADCAST = 1 to enable network)
INFO  [px4] Startup script returned successfully
pxh> ERROR [muorb] SLPI: MAG #0 failed:  TIMEOUT!	
ERROR [muorb] SLPI: BARO #0 failed:  TIMEOUT!	
pxh> qsheERROR [muorb] SLPI: BARO #0 failed:  TIMEOUT!
pxh> qshell dsp_hitl statERROR [muorb] SLPI: MAG #0 failed:  TIMEOUT!
ERROR [muorb] SLPI: BARO #0 failed:  TIMEOUT!	
pxh> qshell dsp_hitl statusINFO  [uORB] Advertising remote topic vehicle_global_position
INFO  [uORB] Advertising remote topic estimator_baro_bias
INFO  [muorb] SLPI: Advertising remote topic geofence_result

INFO  [qshell] Send cmd: 'dsp_hitl status'
INFO  [muorb] SLPI: Marking DeviceNode(qshell_req) as advertised in process_remote_topic
INFO  [muorb] SLPI: qshell gotten: dsp_hitl status
INFO  [muorb] SLPI:   arg0 = 'dsp_hitl'

INFO  [muorb] SLPI:   arg1 = 'status'

INFO  [muorb] SLPI: Running: yes
INFO  [muorb] SLPI: Messages received from simulator:
INFO  [muorb] SLPI: 	Heartbeat received: 150
INFO  [muorb] SLPI: 	HIL Sensor received: 1003
INFO  [muorb] SLPI: 	Odometry received: 0
INFO  [muorb] SLPI: 	GPS received: 12
INFO  [muorb] SLPI: Outputs to PX4:
INFO  [muorb] SLPI: 	IMU updates: 2412
INFO  [muorb] SLPI: 		Current accel x, y, z: -0.135196, 0.199197, -9.980605
INFO  [muorb] SLPI: 		Current gyro x, y, z: 0.002064, 0.002343, -0.004121
INFO  [muorb] SLPI: 	Magnetometer sent: 850
INFO  [muorb] SLPI: 	Barometer sent: 500
INFO  [muorb] SLPI: 	GPS sent: 12
INFO  [muorb] SLPI: 	Odometry sent: 0
INFO  [muorb] SLPI: Outputs to simulator:
INFO  [muorb] SLPI: 	Heartbeats sent: 12
INFO  [muorb] SLPI: 	Actuator updates sent: 4828
INFO  [muorb] SLPI: Unknown messages received: 0
INFO  [muorb] SLPI: Ok executing command: dsp_hitl status
INFO  [qshell] qshell return value timestamp: 86887060, local time: 86898155
pxh> ERROR [muorb] SLPI: BARO #0 failed:  TIMEOUT!	
ERROR [muorb] SLPI: MAG #0 failed:  TIMEOUT!	
ERROR [muorb] SLPI: BARO #0 failed:  TIMEOUT!	
ERROR [muorb] SLPI: MAG #0 failed:  TIMEOUT!	
ERROR [muorb] SLPI: MAG #0 failed:  TIMEOUT!	
ERROR [muorb] SLPI: BARO #0 failed:  TIMEOUT!	
ERROR [muorb] SLPI: MAG #0 failed:  TIMEOUT!	
ERROR [muorb] SLPI: BARO #0 failed:  TIMEOUT!	
ERROR [muorb] SLPI: MAG #0 failed:  TIMEOUT!	
ERROR [muorb] SLPI: BARO #0 failed:  TIMEOUT!	
ERROR [muorb] SLPI: MAG #0 failed:  TIMEOUT!	
ERROR [muorb] SLPI: BARO #0 failed:  TIMEOUT!	
ERROR [muorb] SLPI: BARO #0 failed:  TIMEOUT!	
ERROR [muorb] SLPI: MAG #0 failed:  TIMEOUT!	
ERROR [muorb] SLPI: MAG #0 failed:  TIMEOUT!	
ERROR [muorb] SLPI: BARO #0 failed:  TIMEOUT!	
ERROR [muorb] SLPI: MAG #0 failed:  TIMEOUT!

      I am wondering if slow connection to QGC and sensor timeouts are because of chocking communication channel with all the extra data. Do you think voxl2 can handle increasing mavlink stream of data? If so, where can I take a look to increase it?

      1 Reply Last reply Reply Quote 0
      • Eric KatzfeyE
        Eric Katzfey ModalAI Team @garret
        last edited by

        @garret The install-voxl.sh and related scripts are only there for developer use. They can help speed things up when making and testing many changes. Normally the package should be built and deployed. The default.px4board file controls what gets built. So, if you have new modules you first have to make sure they are built by adding them into that file. There is one for the Linux side (aka voxl2) and one for the DSP side (aka voxl2-slpi). High level stuff like Mavlink and logging go on the Linux side, all real time flight control functionality goes on the DSP side. Once it has been built it needs to be started. That is done by adding the start command into the startup file boards/modalai/voxl2/target/voxl-px4-start. If the driver / module is on the DSP side it has to be started with the qshell command.

        Eric KatzfeyE 1 Reply Last reply Reply Quote 1
        • Eric KatzfeyE
          Eric Katzfey ModalAI Team @Eric Katzfey
          last edited by

          @Eric-Katzfey Those timeouts indicate that the processor is overloaded. Perhaps try enabling the new modules one by one to see if you can figure out which one is taking up too much time.

          G 1 Reply Last reply Reply Quote 0
          • G
            garret @Eric Katzfey
            last edited by garret

            @Eric-Katzfey surprisingly after leaving voxl2 overnight the QGC connection stopped being an issue. However, baro/mag timeout still occurs.

            However, I wonder how I can check if this is indeed processor overload. When I am running voxl-px4-hitl + connect voxl2 to QGC the total CPU usage is only slightly over 70%:
            91eef573-9e02-43ff-a81a-0b144a01952b-image.png
            387aaa62-a50c-4290-9390-68b08236dc86-image.png

            Similarly, when I looked at mavlink messages, nothing really stood out as too heavy:

            • onboard services:
              5c88c560-5844-4f0e-a422-6903a85694f9-image.png

            • from gcs:
              da6b9cb9-1c99-4f31-8333-3a4ad4354013-image.png

            Some of the things I tried:

            • adjusting -r parameter in voxl-px4-hitl-start script. I went as high as 500000. That affected neither the cpu use nor the frequency of timeout error for baro/mag. Specifically, this is line I changed
            # start the onboard fast link to connect to voxl-mavlink-server
mavlink start -x -u 14556 -o 14557 -r 100000 -n lo -m onboard
            • I also tried adjusting upd_mtu in /etc/modalai/voxl-mavlink-server.conf. I went as high as 300, but it had no effect either.
            • adjusted dsp_hitl baudrate to 2000000 in voxl-px4-hitl-start. Also with little to no effect
            qshell dsp_hitl start -g -m -o -b 2000000

            Correct me if I am wrong, but as far as I understand the issue must be DSP communicating with Linux side. Where could I look to try and speed it up if possible?

            i also may be able to get ARK V6 Flight Controller to run with voxl2 - would that be able to solve my issue?

            Thank you in advance for your response

            G 1 Reply Last reply Reply Quote 0
            • G
              garret @garret
              last edited by

              This post is deleted!
              Eric KatzfeyE 2 Replies Last reply Reply Quote 0
              • Eric KatzfeyE
                Eric Katzfey ModalAI Team @garret
                last edited by

                @garret You need to look at the CPU utilization in the DSP where the majority of the PX4 code is running. top will show you only what's happening on the applications processor (Linux) side. The PX4 log contains this information.

                1 Reply Last reply Reply Quote 1
                • Eric KatzfeyE
                  Eric Katzfey ModalAI Team @garret
                  last edited by

                  @garret If the Iris configuration doesn't produce timeouts then possibly it's Gazebo not sending sensor updates fast enough. But I'm just guessing at this point. We've never tried a VTOL configuration so you are in new territory here. I'd first take a PX4 log and verify that the DSP CPU load is not exceeding 65 - 70%. Then perhaps start putting some diagnostic prints around the reception of the sensor messages from Gazebo and see how often they are being sent. Just have to start isolating the root cause.

                  G 2 Replies Last reply Reply Quote 1
                  • G
                    garret @Eric Katzfey
                    last edited by garret

                    @Eric-Katzfey I tried examining the logs for default modal AI px4 firmware IRIS configuration and my customized firmware with vtol modules enabled. Strangely, the CPU usage seems identical and above 75% even in the default case.

                    Log plots here: https://drive.google.com/file/d/1UZfAYMRQ-VfYnyf1u__l9NAUHnM8Q0PJ/view?usp=sharing.

                    I assume one of the cpus is the apps one and the other is the dsp one. This is still strange, since according to my comment above the total cpu usage of the apps was slightly over 70%.

                    @garret said in Firmware Structure (Adding VTOL support) - Clarifications:

                    total CPU usage is only slightly over 70%

                    As for the gazebo sending messages, still looking into it. However, I did some quick diagnostic and found that gazebo's publication rate seems sufficient:

                    • IRIS case
                    root@nikita-ThinkPad-P16s-Gen-2:/usr/workspace/voxl2_hitl_gazebo# gz topic -d 1 -z /gazebo/modalai_world/modalai/mag
Hz:  89.91
Hz:  88.87
Hz:  76.84
Hz: 105.65
Hz:  87.52
Hz:  90.77
Hz:  83.98
Hz:  85.79
Hz:  57.25
Hz: 134.63
Hz:  82.31
Hz:  88.04
Hz:  48.27
Hz: 227.83
Hz:  87.30
Hz:  75.15
Hz:  60.36
Hz: 105.97
Hz: 101.91
Hz:  92.13
Hz:  89.28
Hz:  83.36
Hz:  88.95
Hz:  88.49
Hz:  55.99
Hz: 156.46
Hz:  85.59
Hz:  90.70
Hz:  50.35
Hz: 129.66
Hz:  92.53
Hz:  87.62
Hz:  84.50
Hz:  66.56
Hz:  90.34
Hz:  89.11
Hz:  73.71
Hz:  58.14
Hz: 117.21
Hz:  88.86
Hz:  49.12
Hz: 179.37
Hz:  91.40
Hz:  89.39
Hz:  62.59
Hz: 108.00
Hz:  87.86
Hz:  75.88
Hz:  84.17
Hz:  77.79
Hz:  72.78
Hz:  87.25
Hz:  93.21
Hz:  64.89
Hz:  73.10
Hz:  92.97
Hz:  53.24
Hz: 196.72
Hz:  90.09
Hz:  89.55
Hz:  48.60
Hz: 121.67
Hz:  68.07
Hz:  96.66
Hz:  91.59
Hz:  89.99
Hz:  91.62
Hz:  72.11
Hz:  91.43
Hz:  78.47
Hz:  91.06
Hz:  81.71
Hz:  72.23
Hz:  93.85
Hz:  82.83
Hz:  84.47
Hz:  73.80
Hz: 114.91
Hz:  85.34
Hz:  95.42
Hz:  79.51
root@nikita-ThinkPad-P16s-Gen-2:/usr/workspace/voxl2_hitl_gazebo# gz topic -d 1 -z /gazebo/modalai_world/modalai/baro
Hz:  45.30
Hz:  33.52
Hz:  40.19
Hz:  38.64
Hz:  39.93
Hz:  45.78
Hz:  41.65
Hz:  42.63
Hz:  44.85
Hz:  39.09
Hz:  45.65
Hz:  36.76
Hz:  41.51
Hz:  43.36
Hz:  34.07
Hz:  47.10
Hz:  45.87
Hz:  39.17
Hz:  53.24
Hz:  40.17
Hz:  37.47
Hz:  37.41
Hz:  50.90
Hz:  37.27
Hz:  53.60
Hz:  38.04
Hz:  45.86
Hz:  43.55
Hz:  43.98
Hz:  44.47
Hz:  40.48
Hz:  39.65
Hz:  44.65
Hz:  43.66
Hz:  41.32
Hz:  42.98
Hz:  40.23
Hz:  44.23
Hz:  42.25
Hz:  30.15
                    • VTOL case
                    root@nikita-ThinkPad-P16s-Gen-2:/usr/workspace/voxl2_hitl_gazebo# gz topic -d 1 -z /gazebo/standard_vtol_hitl_world/standard_vtol_hitl/mag
Hz: 261.34
Hz: 268.29
Hz: 165.38
Hz: 301.10
Hz: 219.42
Hz: 318.94
Hz: 258.53
Hz: 260.81
Hz: 259.69
Hz: 261.20
Hz: 230.35
Hz: 246.97
Hz: 105.02
Hz: 27011.70
Hz: 584.93
Hz: 245.19
Hz: 210.23
Hz: 137.50
Hz: 317.01
Hz: 217.37
Hz: 213.23
Hz: 245.10
Hz: 102.70
Hz: 7085.62
Hz: 255.86
Hz: 214.26
Hz: 358.80
Hz: 135.56
Hz: 601.92
Hz: 197.99
Hz: 289.66
Hz: 146.01
Hz: 710.52
Hz: 285.12
Hz: 183.18
Hz: 283.64
Hz: 267.36
Hz: 196.66
Hz: 293.54
Hz: 261.75
Hz: 189.79
Hz: 285.15
Hz: 269.74
Hz: 210.62
Hz: 247.68
Hz: 278.95
Hz:  92.07
Hz:   9.93
Hz: 39858.11
Hz: 249.81
Hz: 2192.14
Hz: 263.26
Hz: 255.81
Hz: 268.84
Hz: 204.21
Hz: 119.26
Hz:   9.63
Hz:   9.72
Hz:   9.71
Hz:   9.96
Hz: 2776.08
Hz: 133.24
Hz: 274.92
Hz: 823.84
Hz: 202.26
Hz: 276.13
Hz: 276.10
Hz: 216.36
Hz: 339.81
Hz: 270.43
Hz: 266.76
Hz: 261.38
Hz: 265.82
Hz: 268.36
Hz: 256.52
Hz: 276.77
Hz:  93.41
Hz: 1257.61
Hz: 263.08
Hz: 263.41
Hz: 146.94
Hz: 270.98
Hz: 243.40
Hz: 283.23
Hz: 150.90
Hz: 1511.17
Hz: 274.21
Hz: 258.02
Hz: 144.53
Hz: 2885.13
Hz: 247.98
Hz: 137.59
Hz: 829.43
Hz: 247.74
Hz: 254.85
Hz: 264.31
Hz: 209.19
Hz: 275.45
Hz: 267.43
Hz: 270.33
Hz: 245.57
Hz: 228.72
Hz: 232.26
Hz: 264.85
Hz:  93.60
Hz: 49265.94
Hz: 499.46
Hz: 233.43
Hz: 273.17
Hz: 234.92
Hz: 233.14
Hz: 258.48
Hz: 278.42
Hz: 225.91
Hz: 280.44
Hz: 135.68
Hz: 276.36
Hz: 134.54
Hz: 282.50
root@nikita-ThinkPad-P16s-Gen-2:/usr/workspace/voxl2_hitl_gazebo# gz topic -d 1 -z /gazebo/standard_vtol_hitl_world/standard_vtol_hitl/baro
Hz: 132.49
Hz: 129.18
Hz: 129.00
Hz: 130.49
Hz: 111.43
Hz: 151.83
Hz:  97.87
Hz: 133.37
Hz: 108.81
Hz: 145.29
Hz:  88.34
Hz: 257.54
Hz: 127.70
Hz: 102.37
Hz: 131.12
Hz: 124.66
Hz: 104.16
Hz: 135.17
Hz: 117.07
Hz: 123.88
Hz: 134.75
Hz: 130.16
Hz: 134.76
Hz:  70.86
Hz: 577.87
Hz: 123.25
Hz: 120.28
Hz: 104.71
Hz: 179.62
Hz: 125.18
Hz: 119.61
Hz: 121.50
Hz: 148.17
Hz: 126.87
Hz: 103.36
Hz: 132.89
Hz: 145.11
Hz: 114.83
Hz: 113.25
Hz: 148.53
Hz: 132.36
Hz: 128.08
Hz: 124.23
Hz:  72.73
Hz: 282.04
Hz: 127.52
Hz: 128.52
Hz: 124.72
Hz: 121.43
Hz: 112.05
Hz: 126.82
Hz: 108.68
Hz: 165.72
Hz: 128.03
Hz: 124.98
Hz: 100.47
Hz:   9.51
Hz: 137.65
Hz:  93.98
Hz:  88.69
Hz: 135.25
Hz: 129.01
Hz: 130.18
Hz:  89.62
Hz: 143.84
Hz: 126.38
Hz: 138.17
Hz: 133.42
Hz: 122.09
Hz: 136.52
Hz: 136.22
Hz: 117.27
Hz:  54.04
Hz: 39575.75
Hz: 160.93
Hz: 127.02
Hz:  87.96
Hz: 260.10
Hz:  93.48
Hz: 110.70
Hz: 144.94
Hz: 129.48
Hz: 132.85
Hz: 119.29
Hz: 123.63
Hz: 143.99
Hz: 103.66
Hz: 136.10
Hz:   9.17

                    Weirdly there is some frequency jumps with VTOL world. Could this be the cause of the error? If you have an idea where these come from , I would appreciate the hint.

                    Although, when looking at the logs again, this is the magnetometer plots:https://drive.google.com/file/d/1yxWfiHGunIUJAk8WNnSOku5CxXks31so/view?usp=sharing

                    Apologies for small dot thickness, but it can be noted that magnetometer coming from the vehicle is at twice the frequency that it is coming from the sensor. Ironically, these rates are also off from the frequencies pasted above. For IRIS default the vehicle's magnetometer is at about 2Hz, and same for VTOL. Sensor frequency for IRIS is about 1Hz and same for VTOL. I wonder how this is possible, since for EKF2 to run smoothly it requires at least 5Hz update rate on sensors. I assume I am misinterpreting the data, so if you spot a mistake in my analysis please let me know.

                    1 Reply Last reply Reply Quote 0
                    • G
                      garret @Eric Katzfey
                      last edited by

                      @Eric-Katzfey An update from me:

                      I have looked at the cpu usage from voxl-portal and got even more confused. When comparing with my previous posts, the results differ again. To summarize:

                      • after adb shell into voxl2, "top" suggested total linux cpu usage of ~70% when running px4
                      • voxl-inspect-services gave about ~15%
                      • PX4 logs give varying cpu usage from 0% to 88% depending on the core
                      • voxl portal outputs about 10% cpu usage on every core when running px4 hitl sim.

                      Is there a reason all these sources differ so much?

                      I also looked further into potential reasons for timeout errors on vtol. Keeping px4 parameters identical, the error shows up only when loading a vtol frame in gazebo docker. I wonder if there is anything special that you can think of about how gazebo docker is treating a vtol vs multicopter? I looked into both .world and .sdf files line by line, but nothing stood out. Wondering if there is something I am missing somewhere else.

                      Lastly, to follow up on super low rate for "sensor_mag" and "vehicle_magnetometer" from the px4 log.
                      @garret said in Firmware Structure (Adding VTOL support) - Clarifications:

                      https://drive.google.com/file/d/1yxWfiHGunIUJAk8WNnSOku5CxXks31so/view?usp=sharing

                      Identical frequency in both multicopter and vtol case makes me wonder if there is a hardcoded bottleneck somewhere for this specific data - could you point it out to me if that is the case?

                      Hoping to hear from you and thank you for your cooperation.

                      Eric KatzfeyE 3 Replies Last reply Reply Quote 0
                      • Eric KatzfeyE
                        Eric Katzfey ModalAI Team @garret
                        last edited by

                        @garret You have to be careful looking at CPU utilization on the Linux side. Sometimes the number reflects usage on a single core or the usage of a single process and sometimes it is total usage across all cores. The utilization on the DSP is what usually matters more for PX4 since most of the code runs on the DSP.

                        1 Reply Last reply Reply Quote 0
                        • Eric KatzfeyE
                          Eric Katzfey ModalAI Team @garret
                          last edited by

                          @garret Unfortunately, I only have experience with the IRIS model in Gazebo so cannot comment on why it would act differently with the VTOL.

                          1 Reply Last reply Reply Quote 0
                          • Eric KatzfeyE
                            Eric Katzfey ModalAI Team @garret
                            last edited by

                            @garret This is the code that deals with the Mavlink messages coming in from Gazebo and the sending of Mavlink messages back to Gazebo: https://github.com/modalai/px4-firmware/blob/voxl-dev/boards/modalai/voxl2-slpi/src/drivers/dsp_hitl/dsp_hitl.cpp

                            G 1 Reply Last reply Reply Quote 0
                            • G
                              garret @Eric Katzfey
                              last edited by

                              @Eric-Katzfey I am 99% confident I found the issue. When running gazebo for IRIS, the real time factor is 1. When running vtol the real time factor drops to ~0.5 or lower. I just tried running gazebo for iris and manually lower the real time factor. i was able to replicate the issue. I think it is because in the mavlink communication script you shared the update for mag and baro happens only in cases where HIL_SENSOR messege is receiver. If the message is delayed, the sensors timeout.

                              I will start looking into troubleshooting, but if you know what can be done to speed up gazebo's computation or to ease the load of VTOL model, I would appreciate the hint

                              G 1 Reply Last reply Reply Quote 0
                              • G
                                garret @garret
                                last edited by

                                @garret I also wonder if there is a reason for running gazebo provided in docker except for iris comparability? Could i theoretically run voxl2 with a newer version of gazebo? It seems like those might be more efficient with modelling physics

                                1 Reply Last reply Reply Quote 0
                                • First post
                                  Last post
                                Powered by NodeBB | Contributors