voxl2 is not booting

Jetson Nano

@Eric-Katzfey @Alex-Kushleyev @tom @Vinny
Hello guys,
I had a crash after which the voxl2 is not booting up. The board is powered sensors and fans are getting powered but the voxl 2 is not completing the boot. I tried unbricking but did not resolve the issue. there is no visible damage to the board, i had checked it.
could any help me out here.

Jetson Nano

@Eric-Katzfey @Alex-Kushleyev @tom @Vinny

Hey guys, do you think my board is damaged.

Alex Kushleyev

@Jetson-Nano , can you please elaborate on what happened during the attempt to unbrick the voxl2 board?

Jetson Nano

@Alex-Kushleyev Thank you for reverting back.
While I attempt to unbrick the process is displaying as completed but after the reboot and waiting for almost 10 - 15 minutes the board is not booting.
I have turned on and off the Switch 2 for the process. I followed every step mentioned in the website.

Jetson Nano

successfully\n[PROGRAM] flashed \"PrimaryGPT\" successfully\n[PROGRAM] flashed \"BackupGPT\" successfully\n[PROGRAM] flashed \"PrimaryGPT\" successfully\n[PROGRAM] flashed \"BackupGPT\" successfully\nqdl: firehose operation timed out\n\nFlash flat build passed\n"}

this is full log
this is the end line

Jetson Nano

@Alex-Kushleyev

Could you verify that this issue is due to missing sensor at U18 port

Alex Kushleyev

@Jetson-Nano , the boot issue is not related to the missing U18 (which is a secondary barometer, which was removed from the revised VOXL2 design a while ago (in M0154 revision of the VOXL2 board). The sensor was not used. https://docs.modalai.com/voxl2-onboard-sensors/#barometers

I will double check with the team how to proceed in your case.

Alex

Jetson Nano

@Alex-Kushleyev Thank you for your reply. I have a old voxl2 as well as new revised voxl 2, the one not booting is the new one, When I compared it to the old design I didn't find the barometer, but I also had noticed that the new design do not have them, i asked just in case whether it would be the cause. Thank you for clearing the query.

Please discuss with your team and let me know how can I proceed, If you need anymore data from me I can provide it.

Alex Kushleyev

Hi @Jetson-Nano,

You have an option to send the board back to use for diagnostics for a fee (if you would like to do that, please fill out the RMA form : www.modalai.com/rma). However, the likelihood for a positive result is not very high because the board and the SoC is so complex and finding micro fractures can be very difficult.

Also, let me clarify why you are able to flash the board in EDL mode, but not boot. When in EDL mode, the majority of VOXL2 SoC is not actually running. You basically have direct USB to FLASH storage access in order to write the low-level partitions. So it might look like the board flashes fine, but when you try to boot into the OS, some issue may prevent the board from booting completely.

There is a way to perform some more diagnostics on your own as well. VOXL2 has a debug console UART port, which is accessible via J3 : https://docs.modalai.com/voxl2-connectors/#j3---legacy-board-to-board-connector-b2b (pins 27, 29, which are unprotected, 1.8V levels).

Even though normally we use a perf kernel which disables prints to the console, when the VOXL2 boots, there are debug messages printed to console before the kernel starts. See example below. It also possible to enable the Kernel messages to console witih a debug kernel, which we could provide if needed (or you could also build it yourself, i can tell you how).

Do you have any add-on boards (like M0130, which has an unpopulated connector with console debug 4-pin J6, which is level shifted to 3.3V). I will see if we have any other options to tap into the debug console.

https://docs.modalai.com/lte-io-breakout-usb-hub-datasheet/#connector-callouts (look for J6 in the diagram, but it's not populated on the actual board)
the pinout of J6 on M0130, starting witih pin1 is (3.3V, DBG_RX (in), DBG_TX (out), GND)

We also used to make M0017 but it's EOL. https://docs.modalai.com/m0017/

Here is an example of debug console output on a VOXL2 (M0154) with perf kernel (no prints from kernel itself). You should see what you see on your damaged board, perhaps it will actually tell you what the issue is.

Log Type: B - Since Boot(Power On Reset),  D - Delta,  S - Statistic
S - QC_IMAGE_VERSION_STRING=BOOT.XF.3.2-00354-SM8250-1
S - IMAGE_VARIANT_STRING=Soc8250LAA
S - OEM_IMAGE_VERSION_STRING=4ff32558f5a2
S - Boot Interface: UFS
S - Secure Boot: Off
S - Boot Config @ 0x00786070 = 0x00000001
S - JTAG ID @ 0x00786130 = 0x0015a0e1
S - OEM ID @ 0x00786138 = 0x00000000
S - Serial Number @ 0x00786134 = 0xb211f0e6
S - OEM Config Row 0 @ 0x007841e0 = 0x0000000000000000
S - OEM Config Row 1 @ 0x007841e8 = 0x0000000000000000
S - Feature Config Row 0 @ 0x007841f8 = 0x0040200000000400
S - Feature Config Row 1 @ 0x00784200 = 0xc000000000000000
S - Core 0 Frequency, 1516 MHz
S - PBL Patch Ver: 5
S - PBL freq: 600 MHZ
D -      6208 - pbl_apps_init_timestamp
D -     39432 - bootable_media_detect_timestamp
D -       970 - bl_elf_metadata_loading_timestamp
D -       709 - bl_hash_seg_auth_timestamp
D -      6316 - bl_elf_loadable_segment_loading_timestamp
D -      5492 - bl_elf_segs_hash_verify_timestamp
D -      6999 - bl_sec_hash_seg_auth_timestamp
D -       819 - bl_sec_segs_hash_verify_timestamp
D -        35 - pbl_populate_shared_data_and_exit_timestamp
S -     66980 - PBL, End
B -     80795 - SBL1, Start
B -    196023 - SBL1 BUILD @ 20:33:25 on Mar 24 2022
B -    200171 - usb: usb_shared_hs_phy_init: hs phy cfg size , 0xc
D -     10584 - sbl1_hw_init
D -        30 - boot_flash_init
B -    615002 - UFS INQUIRY ID: KIOXIA  THGJFAT0T44BAILB1001
B -    616710 - UFS Boot LUN: 1
D -       884 - Auth Metadata
D -    419406 - sbl1_xblconfig_init
D -         0 - boot_config_data_table_default_init
D -       336 - boot_config_data_table_init
B -    641506 - CDT Version:3,Platform ID:11,Major ID:1,Minor ID:0,Subtype:3
D -     17324 - sbl1_hw_platform_pre_ddr
D -         0 - devcfg init
B -    670573 - PM: PM 0=0x4000008000000040:0x40 
B -    670664 - PM: PM 1=0x4000008000000040:0x800 
B -    675209 - PM: PM 2=0x4000008000000020:0x800 
B -    679875 - PM: PM 5=0x400000000000000:0x0 
B -    684511 - PM: POWER ON by CBLPWR, POWER OFF due to FAULT UVLO
B -    697504 - PM: PMA_2 PON:CBL ON:PON OFF:FLT FAULT1:UVLO
B -    697504 - PM: PMA_3 PON:CBL ON:PON OFF:FLT FAULT1:UVLO
B -    804346 - PM: SET_VAL:Skip
B -    804895 - PM: PSI: b0x00_v0x53
B -    810964 - PM: Device Init # SPMI Transn: 5397
B -    819962 - PM: Driver Init # SPMI Transn: 531
B -    832772 - PM: battery Id: 7574
B -    832772 - PM: Debug Board detected
D -    173576 - pmic XBL init
D -     28548 - vsense_railway_cpr init
D -    209657 - sbl1_hw_pre_ddr_init
D -         0 - boot_dload_handle_forced_dload_timeout
D -      1494 - sbl1_load_ddr_training_data
D -      5124 - sbl1_ddr_set_params
B -    901519 - eCDT MRR - Data Starting Address: 0x09066D00

B -    903715 - DSF version = 156.8.18
B -    907161 - Manufacturer ID = ff, Device Type = 8
B -    910760 - Rank 0 size = 4096 MB, Rank 1 size = 4096 MB
D -     31354 - sbl1_ddr_init
D -         0 - boot_pre_ddi_entry
B -    923967 - do_ddr_training, Start
B -    954406 - DDR: Start of DDR Training Restore
B -    963830 - Frequency = 1805 MHz
B -    963830 - DDR: End of DDR Training Restore
D -     40168 - do_ddr_training, Delta
D -     47793 - sbl1_do_ddr_training
D -         0 - sbl1_hand_control_to_devprog_ddr
D -      1006 - boot_ddi_entry
B -    983167 - Pimem init cmd, entry
D -     10736 - Pimem init cmd, exit
D -     13878 - sbl1_post_ddr_init
D -         0 - sbl1_hw_init_secondary
B -   1003694 - APDP Image Load, Start
D -       824 - Auth Metadata
D -       488 - Segments hash check
D -      9638 - APDP Image Loaded, Delta - (8204 Bytes)
D -         0 - boot_dload_dump_security_regions
B -   1021780 - usb: UFS Serial - e53d3d65
B -   1026264 - usb: chgr -  SDP_CHARGER
D -      8387 - boot_dload_check
D -         0 - boot_cache_set_memory_barrier
D -         0 - boot_smem_debug_init
D -       366 - boot_smem_init
D -        31 - boot_smem_alloc_for_minidump
B -   1047492 - PM: SMEM Chgr Info Write Success
D -      4118 - boot_smem_store_pon_status
D -        30 - sbl1_hw_platform_smem
D -        92 - boot_ddr_share_data_to_aop
D -       518 - boot_clock_init_rpm
D -         0 - boot_vsense_copy_to_smem
D -         0 - boot_share_flash_data
D -        30 - boot_populate_ram_partition_table
D -        31 - boot_populate_ddr_details_shared_table
D -         0 - sbl1_tlmm_init
D -         0 - sbl1_efs_handle_cookies
D -         0 - boot_apt_test
B -   1094218 - OEM_MISC Image Load, Start
D -       732 - Auth Metadata
D -       305 - Segments hash check
D -      9272 - OEM_MISC Image Loaded, Delta - (8192 Bytes)
B -   1106875 - QTI_MISC Image Load, Start
D -      5368 - QTI_MISC Image Loaded, Delta - (0 Bytes)
B -   1123833 - PM: PM Total Mem Allocated: 2964 
D -      7686 - sbl1_pm_aop_pre_init_wrapper
B -   1128408 - AOP Image Load, Start
D -       885 - Auth Metadata
D -      1921 - Segments hash check
D -     13908 - AOP Image Loaded, Delta - (198224 Bytes)
B -   1145641 - QSEE Dev Config Image Load, Start
D -       976 - Auth Metadata
D -       549 - Segments hash check
D -     12505 - QSEE Dev Config Image Loaded, Delta - (45520 Bytes)
B -   1167174 - QSEE Image Load, Start
D -      4331 - Auth Metadata
D -     21685 - Segments hash check
D -     70760 - QSEE Image Loaded, Delta - (3260452 Bytes)
D -       152 - sbl1_hw_play_vibr
B -   1246718 - SEC Image Load, Start
D -      2989 - SEC Image Loaded, Delta - (0 Bytes)
B -   1253214 - CPUCPFW Image Load, Start
D -      4728 - CPUCPFW Image Loaded, Delta - (0 Bytes)
B -   1261785 - QHEE Image Load, Start
D -       854 - Auth Metadata
D -      2715 - Segments hash check
D -     12596 - QHEE Image Loaded, Delta - (448728 Bytes)
B -   1277675 - OEM_HYP Image Load, Start
D -      5277 - OEM_HYP Image Loaded, Delta - (0 Bytes)
B -   1286795 - STI Image Load, Start
D -      5093 - STI Image Loaded, Delta - (0 Bytes)
B -   1295914 - APPSBL Image Load, Start
D -      1037 - Auth Metadata
D -     13024 - Segments hash check
D -     23851 - APPSBL Image Loaded, Delta - (2510848 Bytes)
D -         0 - sbl1_appsbl_arch_determination
B -   1328946 - SBL1, End
D -   1252085 - SBL1, Delta
S - Flash Throughput, 202000 KB/s  (6556580 Bytes,  32309 us)
S - DDR Frequency, 1555 MHz


UEFI Start     [ 1548]


ModalAI v1.0
 - 0x09FC01000 [ 1552] Sec.efi
ASLR        : ON
DEP         : ON (RTB)
Timer Delta : +1 mS
RAM Entry 0 : Base 0x080000000  Size 0x03B800000
RAM Entry 1 : Base 0x180000000  Size 0x100000000
RAM Entry 2 : Base 0x0C0000000  Size 0x0C0000000
Total RAM   : 0x1FB800000
UEFI Ver    : 5.0.220324.BOOT.XF.3.2-00354-SM8250-1
Build Info  : 64b Mar 24 2022 20:33:48
Boot Device : UFS
PROD Mode   : TRUE
Retail      : TRUE
PM0: 30, PM1: 32, PM2: 39, PM3: 33, PM5: 36, 
UFS INQUIRY ID: KIOXIA  THGJFAT0T44BAILB1001
UFS Boot LUN: 1
 Valid primary and !Valid backup partition table
 Restore backup partition table by the primary
 Restore backup partition table success
HW Wdog Setting from PCD : Disabled
DisplayDxe: SW renderer mode enabled!
DisplayDxe: Resolution 640x480 (1 intf)
SPSSDxe:: SPSSDxe_PreLoadChecks SPSS loading in UEFI not supported for this chipID = 455 
UsbConfigLibOpenProtocols: PMI version (0x20)
UsbConfigInit: UsbConfigInit, not start on port: 0, mode 0
UsbConfigInit: UsbConfigInit, not start on port: 1, mode 0
UsbConfigPortsQueryConnectionChange: usbport->connectstate: ATT
APC1 IDDQ WC 0 , L2:0, CPU[0]:175, CPU[1]:175, CPU[2]:175, CPU[3]:175
 APC1 Total 700
LoadSys  TIME 0ms
tsens  TIME 1ms
scm  TIME 2ms
Load CPU 0 Slp: 0x6C, Int: 0x16
Load CPU 1 Slp: 0xE0, Int: 0xFFFFFF65
Load CPU 2 Slp: 0x94, Int: 0xFFFFFFF1
Load CPU 3 Slp: 0x107, Int: 0xFFFFFFB1
Load GPU 0 Slp: 0x99, Int: 0x13B adc_min:0x19E adc_max:0x15C
Load NPU 0 Slp: 0x2C, Int: 0x4F
LmhIsenseInit Pre CPU  TIME 0ms
GLD IDDQ WC 0
LmhIsenseSubSysEntry Post SubSysEntryCb  TIME 0ms
LmhIsenseSubSysEntry Post LmhIsenseSubSysInit  TIME 0ms
LmhIsenseInitSubSys Post SubSysEntry  TIME 0ms
LmhIsenseInitSubSys Post SubSysTrim  TIME 0ms
LmhIsenseInitSubSys Post SubSysExit  TIME 0ms
LmhIsenseInit Post CPU  TIME 0ms
GPU IDDQ 60
LmhIsenseSubSysEntry Post SubSysEntryCb  TIME 0ms
LmhIsenseSubSysEntry Post LmhIsenseSubSysInit  TIME 0ms
LmhIsenseInitSubSys Post SubSysEntry  TIME 0ms
LmhIsenseInitSubSys Post SubSysTrim  TIME 0ms
LmhIsenseInitSubSys Post SubSysExit  TIME 1ms
LmhIsenseInit Post GPU  TIME 2ms
Entering in func1 of NPU  TIME 2ms
NPU IDDQ 70
LmhIsenseSubSysEntry Post SubSysEntryCb  TIME 0ms
LmhIsenseSubSysEntry Post LmhIsenseSubSysInit  TIME 0ms
LmhIsenseInitSubSys Post SubSysEntry  TIME 0ms
LmhIsenseInitSubSys Post SubSysTrim  TIME 0ms
LmhIsenseInitSubSys Post SubSysExit  TIME 0ms
LmhIsenseInit Post NPU  TIME 2ms
isense  TIME 5ms
ISENSE TOTAL TIME 5ms
Disp init wait [ 2031] 
-----------------------------
Platform Init  [ 2050] BDS
UEFI Ver   : 5.0.220324.BOOT.XF.3.2-00354-SM8250-1
Platform           : QRD
Chip Name          : QRB5165
Chip Ver           : 2.1
Chip Serial Number : 0xB211F0E6
Silver cluster Core 0 Freq: 1516 MHz
-----------------------------
UEFI Total : 542 ms
POST Time      [ 2090] OS Loader
Loader Build Info: May 17 2024 23:45:26
VB: Non-secure device: Security State: (0xF7F)
VB: RWDeviceState: Succeed using devinfo!
KeyPress:0, BootReason:0
Fastboot=0, Recovery:0
GetVmData: making ScmCall to get HypInfo
GetVmData: No Vm data present! Status = (0x3)
VM Hyp calls not present
Booting from slot (_a)
Booting Into Mission Mode
Loading Image Start : 2109 ms
Loading Image Done : 2109 ms
Total Image Read size : 4096 Bytes
Loading Image Start : 2109 ms
Loading Image Done : 2135 ms
Total Image Read size : 17981440 Bytes
Loading Image Start : 2141 ms
Loading Image Done : 2166 ms
Total Image Read size : 25165824 Bytes
In Ubuntu ab ota platform, we don't use the system_a / _b slotGetVmData: making ScmCall to get HypInfo
GetVmData: No Vm data present! Status = (0x3)
No Ffbm cookie found, ignore: Not Found
Memory Base Address: 0x80000000
Decompressing kernel image start: 2169 ms
Decompressing kernel image done: 2537 ms
Dtbo hdr magic mismatch 0, with D7B7AB1E
DTB offset is incorrect, kernel image does not have appended DTB
Cmdline: noinitrd  earlycon=msm_geni_serial,0xa90000 androidboot.hardware=qcom androidboot.console=ttyMSM0 androidboot.memcg=1 lpm_levels.sleep_disabled=1 video=vfb:640x400,bpp=32,memsize=3072000 msm_rtb.filter=0x237 service_locator.enable=1 androidboou
RAM Partitions
Add Base: 0x0000000080000000 Available Length: 0x000000003B800000 
Add Base: 0x0000000180000000 Available Length: 0x0000000100000000 
Add Base: 0x00000000C0000000 Available Length: 0x00000000C0000000 
ERROR: Could not get splash memory region node
kaslr-Seed is added to chosen node

Shutting Down UEFI Boot Services: 2607 ms
Start EBS        [ 2607] 
BDS: LogFs sync skipped, Unsupported
App Log Flush : 23 ms
Exit EBS        [ 2645] UEFI End