# Starling fan attachment and optimization Source: https://forum.modalai.com/topic/3228/starling-fan-attachment-and-optimization Category: General Questions (https://forum.modalai.com/category/2/general-questions) Posted: 2024-03-18 05:34:54 UTC by Darshit Desai Replies: 35 · Views: 9724 ## Darshit Desai · 2024-03-18 05:34:54 UTC Hi I have been running a profiling test of the code developed offline on starling with the processes like voxl_mpa_to_ros, voxl_tflite_server and a couple of other custom built ros nodes, I have found that the CPU overheats and I want to install fans on the voxl2 board, is there a guide for that and how to install and power the fans? Also is the CPU overheating concerning for the lifetime of the CPU? ```BASH Name Freq (MHz) Temp (C) Util (%) ----------------------------------- cpu0 1804.8 94.6 77.50 cpu1 1804.8 94.6 70.54 cpu2 1804.8 93.4 71.08 cpu3 1804.8 93.4 68.16 cpu4 1670.4 91.5 17.82 cpu5 1670.4 95.0 44.33 cpu6 1670.4 92.7 22.84 cpu7 844.8 93.8 9.61 Total 95.0 47.73 10s avg 43.86 ----------------------------------- GPU 587.0 95.0 42.38 GPU 10s avg 44.45 ----------------------------------- memory temp: 93.4 C memory used: 3004/7671 MB ----------------------------------- Flags CPU freq scaling mode: auto Standby Not Active CPU OVERHEAT WARNING ----------------------------------- ^C received SIGINT Ctrl-C closing ``` ## Reply by Eric Katzfey (ModalAI staff) · 2024-03-18 15:37:19 UTC @Darshit-Desai Do you see it getting too hot when it is flying or just when it is sitting on the ground? ## Reply by Darshit Desai · 2024-03-18 16:04:00 UTC (in reply to Eric Katzfey) @Eric-Katzfey This happens when I am testing it on the desktop (not flying). But I guess more processes like flying and moving around the drone would only increase the CPU load ## Reply by Alex Kushleyev (ModalAI staff) · 2024-03-18 19:07:28 UTC (in reply to Darshit Desai) @Darshit-Desai , When the drone is flying, there should be plenty of air flow to cool down the board, however it does depend on the overall system load, of course. Your CPU and GPU are both loaded to about 45% each, which is not too much, but without any cooling at all, can certainly reach high temperatures. The CPU will throttle itself (above 95C) to avoid damage, but it is definitely not great to run the board so hot for extended periods of time and many cycles. The simplest thing you can do, if you want to avoid modifying the vehicle, is use a small desktop fan that will blow air onto the board while you are testing in non-flying state. Mounting the fan on the vehicle would require modifications of the frame component design. Alex ## Reply by Darshit Desai · 2024-03-18 19:17:35 UTC (in reply to Alex Kushleyev) @Alex-Kushleyev Ok thanks I will try out the desktop cooling fan recommendation, and I did see a significant throttling in performance where my own algorithm which worked fine on my PC wasn't even able to do a simple O(1) lookup. But about the mounting of the fan, let's assume I can design some temporary 3d printed mount for the fan, may I know how to electrically connect to / solder to a power source like a power module / GPIO of the voxl2? ## Reply by Alex Kushleyev (ModalAI staff) · 2024-03-18 20:14:05 UTC (in reply to Darshit Desai) @Darshit-Desai VOXL2 has a fan connector on both VOXL2 and VOXL2 mini: https://docs.modalai.com/voxl2-connectors/#j2---5vdc-fan-control https://docs.modalai.com/voxl2-mini-connectors/#j2---5vdc-fan-control We also sell a cooling fan that works with VOXL1/2 : https://www.modalai.com/products/voxl-cooling-fan . on VOXL2, the fan output is always on and we (currently) do not have a way to control it. ## Reply by Darshit Desai · 2024-03-20 21:33:02 UTC (in reply to Alex Kushleyev) @Alex-Kushleyev We have the MODAL AI PCB fans in our inventory already, I see that on starling right where the fan connetor is present for the voxl2 board there is screw hole through which a allen head bolt goes in, this is kind of a bad design because even if I use a flat head screw it would still obstruct the connector. ![6356b097-4620-4a0f-bc70-fa92f68046e6-image.png](https://forum.modalai.com/assets/uploads/files/1710970173229-6356b097-4620-4a0f-bc70-fa92f68046e6-image.png) Also for the desktop testing I am using a 12 V 0.5 Amp fan, would that emulate a similar amount of airflow as it would be in flight when the thrust motors rotate, (Considering I don't use any cooling on PCB). The fan diameter is around 120 mm and is the same size as the ones which are present in desktop CPUs ## Reply by Alex Kushleyev (ModalAI staff) · 2024-03-20 21:42:09 UTC (in reply to Darshit Desai) @Darshit-Desai , I agree, the J2 location is not ideal.. Regarding the external fan, you should try it out and see what works for you on the bench test. In flight we don't use a fan because the board typically does not overheat, but it is hard to say how much airflow is coming from the propellers. ## Reply by Darshit Desai · 2024-03-21 22:04:26 UTC (in reply to Alex Kushleyev) @Alex-Kushleyev Sure I will try it out, but given the size of the external fan I want to simulate the exact run of the drone with the airflow to check if it's able to handle the load or do I need to do more optimization (like modify voxl_mpa_to_ros service or remove other services) ## Reply by Alex Kushleyev (ModalAI staff) · 2024-03-23 20:07:53 UTC (in reply to Darshit Desai) @Darshit-Desai , i would say it is not possible to simulate the exact airflow from the drone with an external fan, so my suggestion is set up an external fan for any bench-top development you need to do (to avoid overheating while not flying) and as soon as you are ready to test things out, just test fly. Keep in mind that you can do a manual flight (thrust + attitude) while your CPU / GPU is loaded up with processing as needed. Don't wait too long before you test in flight. ## Reply by Alex Kushleyev (ModalAI staff) · 2024-03-23 20:10:30 UTC (in reply to Alex Kushleyev) Also, it is very typical for software on drones to operate in "idle" mode while not flying to avoid overheating due to cpu load and lack of air flow from propellers. There is usually no need to run the full processing stack at max power while the drone is just sitting on the ground. And if you need to test on bench while not flying (during development), just use external fan. ## Reply by Darshit Desai · 2024-03-24 17:12:18 UTC (in reply to Alex Kushleyev) @Alex-Kushleyev Ok I have tested it with a desktopfan it does make a dent in the rise in temperature, I still want to install the Voxl Fan on the drone before flying, I saw the drawings the hole near the J2 connector has 33.5 mm diameter. Is it safe to remove that bolt to install a flat head bolt and what size should it be in inches/mm? ## Reply by Alex Kushleyev (ModalAI staff) · 2024-03-25 15:04:24 UTC (in reply to Darshit Desai) @Darshit-Desai , do you mean the hole has 3.5mm (not 33.5 mm)? i will check if we have a recommended screw for this. ## Reply by Alex Kushleyev (ModalAI staff) · 2024-03-25 17:32:04 UTC (in reply to Alex Kushleyev) The guidance regarding the fan connector J2 being close to the mounting hole on VOXL2 is the following: we recommend plugging the fan connector into J2 prior to inserting the mounting screw into the VOXL2 mounting hole. After the fan has been connected to J2, the fan wire can be carefully manipulated around the mounting screw during the screw installation to avoid pinching of the wire. The thickness of the fan wire permits a tight bend. Please try it out! ## Reply by Darshit Desai · 2024-03-25 21:43:52 UTC (in reply to Alex Kushleyev) @Alex-Kushleyev That worked after some effort,(almost broke the connector) I seem to now have a good grasp because with the fan and the flight propellers running the temperatures seem to still reach 75 deg C and there is also throttling happening at some 100-500 millisecond differences, the only other way seems to be to cannibalise mpa-to-ros package and profile my own code ## Reply by Alex Kushleyev (ModalAI staff) · 2024-03-26 02:34:55 UTC (in reply to Darshit Desai) @Darshit-Desai 75C is normal, the CPU will not start throttling itself until about 95C. If you look at output of `voxl-inspect-cpu`, it will tell you what frequency each core is running at. If you set the cpu into performance mode using `voxl-set-cpu-mode perf`, all cores will be fixed to max frequency and will stay at max unless the temperature is too high (above 95C) and the thermal management will kick in. If CPU is in auto mode, the core frequencies will jump up and down depending on the required load. ## Reply by Darshit Desai · 2024-03-26 18:13:28 UTC (in reply to Alex Kushleyev) @Alex-Kushleyev Yes I have been checking using that command. The fact is that the moment the core temperatures hit more then 75C the position mode starts to turn off automatically showing it isn't ready to fly even when the starling is in flight, this also gives the rest of the algorithm a throttling effect like the ros messages which are delivered to the algorithm are slower ## Reply by Darshit Desai · 2024-03-26 21:09:41 UTC (in reply to Alex Kushleyev) @Alex-Kushleyev Here are some screenshots of the QGC, CPU Monitor and my code running in parallel in the terminal, The following services were running: 1)Modified MPAtoROS launch node, with topics like /tof_pc, /voa_pc and /tflite_data only being published 2)Tflite server 3)Couple of ros nodes which use the data from above services to find the position of objects in the environment First photo when my code starts up and the cpu core temperature is low: https://drive.google.com/file/d/1AS1crU9FcIAUmhwG3nD1MG9CElVbwTiu/view?usp=sharing Second screenshot is when the core temperature crosses >70 deg C, note how the position mode turns to red showing not ready https://drive.google.com/file/d/1fzKAZKkbLDWjiuCCvKxyHo7UnE8GWeNK/view?usp=sharing Third screenshot: Here I found a peculiar warning which was not being sent to QGC in the voxl portal where it showed high accelerometer bias warning? Could that be the cause, can higher CPU core temperature cause that? https://drive.google.com/file/d/1E8s7nQja1ijlcgFzkGI80TtRCWk703D7/view?usp=sharing This led me to believe that my fan placement might be wrong so I am putting a photo of my starling drone with the fan placement, Is it correct or am I facing some other issues? Here are the photos of the fan on the starling drone: https://drive.google.com/file/d/1ApMiFDQItF9ZbxI8yD-_GXnhKaqCu3yo/view?usp=sharing, https://drive.google.com/file/d/1Axz_itT0f9L1AVpDvHCaoIwfVr3JRWFt/view?usp=sharing, https://drive.google.com/file/d/1B0UbtqbfJIjkOFo1PaPIW3eRHECIo5d1/view?usp=sharing, https://drive.google.com/file/d/1B141Pc6Q6DCoFynJykV3PbuiFIsjCvco/view?usp=sharing ## Reply by Alex Kushleyev (ModalAI staff) · 2024-03-27 13:56:02 UTC (in reply to Darshit Desai) @Darshit-Desai , Please avoid mounting the cpu fan in a way that adds stress to the board. In your particular case, it seems the fan is wedged between the wifi dongle and the actual CPU, which will actually put pressure and can bend the board. IMU is very sensitive to stresses inside the PCB and slight bending can affect the IMU bias. Additionally, direct contact of the fan to the VOXL2 PCB can add some small vibrations (which can potentially throw off any detector in PX4 that is looking for a perfectly still IMU for initialization). To confirm the IMU bias issue, you can inspect the IMU data using QGC (mavlink inspector) and see if the XYZ accelerometer (while sitting still) changes significantly as the board warms up. Then you can remove the wedged fan (and hold it close to the board) and test again and see if the unusual accel bias is gone (when warmed up). My strong recommendation is to remove the fan from its current location. You may want to design + 3D print an plastic mount, perhaps integrated with the GPS mount, but also having extra attachment points so that it does not oscillate / vibrate due to being cantilevered. If you want to go that route, i can see if we can share the GPS mount CAD file with you. Alex ## Reply by Darshit Desai · 2024-03-27 16:54:02 UTC (in reply to Alex Kushleyev) @Alex-Kushleyev said in [Starling fan attachment and optimization](/post/15743): > To confirm the IMU bias issue, you can inspect the IMU data using QGC (mavlink inspector) and see if the XYZ accelerometer (while sitting still) Which parameter would it be? Position NED? ![e15805bb-21c5-4800-9d56-8770254d9fe2-image.png](https://forum.modalai.com/assets/uploads/files/1711558447673-e15805bb-21c5-4800-9d56-8770254d9fe2-image.png) ## Reply by Darshit Desai · 2024-03-27 17:26:03 UTC (in reply to Alex Kushleyev) @Alex-Kushleyev Also I have consistently observed that cpu0-cpu3 have 1.8-2.0 GHz frequency and on an average 45-65% utilization even when the ros nodes are not running while, cpu7 when the ros nodes are running has 1.9-2.8 Ghz average frequency with 70-85% utilization while cpu4-6 are relatively lighter with only 0.6-0.7 Ghz frequency and ~20% utilization at maximum even when I run my complete code stack, is there a specific reason for such a scenario? ## Reply by Darshit Desai · 2024-03-27 18:40:55 UTC (in reply to Alex Kushleyev) @Alex-Kushleyev I tried this recommendation by removing the fan and holding it up and running my code it didn't make any difference as soon as the temperatures go above 75 the accelerometer bias flag is active. Also I don't think fan placement is an issue because the fan is placed right above the heat sink of the cpus and not anywhere near the imus, there is sufficient space between the wifi dongle and the board to move around a little. @Alex-Kushleyev said in [Starling fan attachment and optimization](/post/15743): > Then you can remove the wedged fan (and hold it close to the board) and test again and see if the unusual accel bias is gone (when warmed up). The bias issues only come when I run the object detection and my own sensor fusion module, without that code running and the fan installed the drone is able to fly in position mode. This is more of a cpu heating and load distribution issue, somehow I think cpu0-3 are pinned for some MPA services and pipes and the rest of the 4 cpus are not being utilized equally, I am looking into multi threading for load distribution in my code, let me know if there are any more recommendations ## Reply by Alex Kushleyev (ModalAI staff) · 2024-03-27 19:35:25 UTC (in reply to Darshit Desai) @Darshit-Desai , it is not a good idea to have any external components touching any components of the VOXL2 board. The reason is that if there is even a minor crash, the movement of the external components (fan in this case), can put mechanical stress on the processor itself and cause internal damage. There are some exceptions, such as if you put VOXL2 inside a metal enclosure, you could have a metal heatsink make contact with the cpu or something like that. In your case, the fan is touching the CPU and the wifi dongle, which puts mechanical constraints such that if there is impact, the fan can be jammed between the cpu and wifi dongle, potentially causing damage to VOXL2 components. Here is how a fan was integrated into VOXL1/2 flight deck: - https://www.modalai.com/products/voxl-flight-deck - https://www.modalai.com/products/voxl-2-flight-deck Although it is harder to see it on voxl2 flight dec, but voxl1 flight deck pictures clearly show a FR4 material that is used to separate the fan from main board and is also used for mounting. ## Reply by Alex Kushleyev (ModalAI staff) · 2024-03-27 19:45:27 UTC (in reply to Darshit Desai) @Darshit-Desai , Back to the accelerometer, you can use the following command to print out the raw accel data: ``` px4-listener sensor_accel TOPIC: sensor_accel sensor_accel timestamp: 306076285 (0.437680 seconds ago) timestamp_sample: 306076069 (216 us before timestamp) device_id: 2490378 (Type: 0x26, SPI:1 (0x00)) x: -0.27078 y: 7.87261 z: 5.88561 temperature: 24.30556 error_count: 1 clip_counter: [0, 0, 0] samples: 10 ``` So you should make sure the board is level and check this message periodically as you are running processing as the board heats up. (in my case the board is not flat, so you are not seeing (0,0,9.8). I am curious what the accel reading is at the start and then when you get the accel bias warning. Worth taking a look at px4 imu calibration. I have not done this myself, but it looks like this is the right resource : https://docs.px4.io/main/en/advanced_config/sensor_thermal_calibration.html Regarding CPU frequencies, when cpu governor is in auto mode, it will try to scale down cpu frequencies to save power. but if you want maximum performance, you can set to to performance mode: ``` voxl-set-cpu-mode perf ``` Note that this does not persist after reboot, if you want permanent change, you can change `more /etc/modalai/voxl-cpu-monitor.conf ` and set normal cpu mode to `perf` ## Reply by Darshit Desai · 2024-03-27 19:48:25 UTC (in reply to Alex Kushleyev) @Alex-Kushleyev said in [Starling fan attachment and optimization](/post/15757): > Regarding CPU frequencies, when cpu governor is in auto mode, it will try to scale down cpu frequencies to save power. but if you want maximum performance, you can set to to performance mode: > > voxl-set-cpu-mode perf > > Note that this does not persist after reboot, if you want permanent change, you can change more /etc/modalai/voxl-cpu-monitor.conf and set normal cpu mode to perf This is definitely useful, but is it right that cpu0-3 are pinned for MPA services, if that is the case I can explicitly assign cpu's for my ros nodes to run cpu 4-7? ## Reply by Alex Kushleyev (ModalAI staff) · 2024-03-27 21:20:42 UTC (in reply to Darshit Desai) @Darshit-Desai I do not think that cpu0-3 are pinned for MPA services. The cpu governor typically assigns task to slower cores when possible (0-3 are slowest, 4-6 are medium, and core 7 is the fastest one) in auto / powersave mode. In performance mode, the distribution of load will probably look different. max frequencies for the cores: 0-3: 1800Mhz 4-6: 2420Mhz 7: 2840Mhz ## Reply by Darshit Desai · 2024-04-02 22:04:23 UTC (in reply to Alex Kushleyev) Hi @Alex-Kushleyev, I wanted to ask one more question regarding cpu utilization while running the tflite server. It shows that it uses cores 4, 5 and 6 for processing and connects itself to the camera server. What is the tflite server using cpu for? Publishing images to libmodal-pipe? like bbox drawn on images? What if I want to disable that and zero out any utilization of cpus by the tflite server? By that I mean this line here: https://gitlab.com/voxl-public/voxl-sdk/services/voxl-tflite-server/-/blob/master/src/main.cpp?ref_type=heads#L247 What else is the tflite server using on cpus which can be removed? As in my system I am only concerned with the bbox detection message. @thomas ## Reply by Guest · 2024-04-03 15:17:53 UTC (in reply to Darshit Desai) @Darshit-Desai Yeah, tflite-server uses the CPU to load in images, draw frames, and other things like the link you posted. If you want some part of tflite-server to run more efficiently, you should fork the repository and make changes. We have an in-depth README in the [repository](https://gitlab.com/voxl-public/voxl-sdk/services/voxl-tflite-server) showing how to build and create a custom fork of tflite-server. Let me know if you have any questions! Thomas ## Reply by Darshit Desai · 2024-04-03 15:21:17 UTC (in reply to Guest) @thomas Thank you, actually the purpose of the question was to identify all components which use CPU in the tflite server. So if it's just image publishing and drawing of bboxes or seg maps on images then I can just comment out the relevant function calls to publishing the image with bbox and the part where it actual makes an image with bbox and writes parameters on the image like fps and other details. Is there any other part which is being done on the CPU other then the ones highlighted? ## Reply by Guest · 2024-04-03 15:27:20 UTC (in reply to Darshit Desai) @Darshit-Desai I mean, there's really only one line of tfliter-server that **isn't** on the CPU and it's [this](https://gitlab.com/voxl-public/voxl-sdk/services/voxl-tflite-server/-/blob/master/src/main.cpp?ref_type=heads#L200) one which actually does the inference (assuming GPU or NPU delegate was selected). Everything else uses the CPU. Thomas ## Reply by Darshit Desai · 2024-04-04 19:25:51 UTC (in reply to Guest) @thomas @Alex-Kushleyev I have bottle necks in my sensor fusion pipeline because of which the temperature of the cpus go very high even with the fan cooling and the propeller throwing airflow in flight. I am doing a rigid body transformation of the incoming points from tof frame to rgb camera frame and that transformation of 38528x3 vector takes a lot of cpu capacity and overheats the cpu. Now I have tried every trick in my toolbox ranging from multi threading to removing unnecessary pipes from voxl_mpa_to_ros but none of them work. I see one of the options is, I somehow filter out the points which are irrelevant to me (i.e. I am only looking for a certain depth range between 10 cms to 1.5 mtrs) before it is published on mpa to ros and then do a rigid body transformation. Another option is to use the raw data by somehow tapping into one of the camera server pipes and filter out the points. Any thoughts on how to optimize the below pipeline for performance would be helpful ![28d5f456-a23c-4777-b716-38b822427570-image.png](https://forum.modalai.com/assets/uploads/files/1712258745750-28d5f456-a23c-4777-b716-38b822427570-image.png) ## Reply by Alex Kushleyev (ModalAI staff) · 2024-04-04 19:56:48 UTC (in reply to Darshit Desai) @Darshit-Desai , can you specify how you are doing the 3D vector transformation? Are you using any math library to help with that? Voxl2 has a powerful NEON instruction set, but the code has to be written correctly to utilize it. Eigen library provides a lot of optimized vector functions. ## Reply by Darshit Desai · 2024-04-04 20:08:20 UTC (in reply to Alex Kushleyev) @Alex-Kushleyev Yes I use Eigen3 for doing the rigid body transformation, Method0: Use the tf2 sensor msgs::do transform cloud function directly on the large 38528x3 pointcloud, which didn't work so I used Eigen 3 from the next method onwards Method1: I have tried statically assigning eigen matrixxd variables, I also tried bifurcating the large 38528x3 pointcloud matrix which i get into 5 parts and parallelizing the mutliplication by doing the multiplication of 5 different parts of matrices on different pinned cpus and then combining them. None of the methods worked because all of them end up heating up the cpu when ran in combination of the voxl-tflite-server. Method2: Other method I tried was filtering the points by depth which reduced the number of points to 15000 points for rigid body transformation and then doing the same parallelized multiplication of rotation matrix and pointcloud but that also ends up heating the cpu too much when run in combination of voxl-tflite-server. All of the code was written in eigen3. This is the code that I used for method 2 https://github.com/darshit-desai/Project_LegionAir/blob/master/your_pointcloud_package/src/pc_transform.cpp Note all of this methods above run fine on my desktop cpu which is understandable because my desktop cpu has literally a million times more compute power and better cooling then what's onboard the voxl Edit: Also in case of voxl I use the CPUs in perf mode ## Reply by Alex Kushleyev (ModalAI staff) · 2024-04-04 20:24:35 UTC (in reply to Darshit Desai) @Darshit-Desai , i suspect you are running into an issue discussed in this topic : https://stackoverflow.com/questions/61140594/why-eigen-use-column-major-by-default-instead-of-row-major I did not confirm this is the issue in your case but you should check. Basically because you have a super long array of vectors, if your memory storage order is not correct, you can be missing cpu cache for every single float that you (eigen3) are loading to get the x, y, z of each vector during the matrix multiplication. This is also true when you are populating the array of vectors into eigen data structure (writes will take a long time if memory locations are jumping around) For best results, x, y and z components of each vector have to be stored in consecutive memory locations and vector N+1 should be right after vector N (in memory) ## Reply by Darshit Desai · 2024-04-04 20:59:08 UTC (in reply to Alex Kushleyev) @Alex-Kushleyev said in [Starling fan attachment and optimization](/post/15993): > https://stackoverflow.com/questions/61140594/why-eigen-use-column-major-by-default-instead-of-row-major As far as I understand from the article here and some of my own search online, https://libeigen.gitlab.io/docs/group__TopicStorageOrders.html It seems that eigen default uses column major order if the options aren't specified. For the point cloud data we have column major would be better right? @Alex-Kushleyev said in [Starling fan attachment and optimization](/post/15993): > For best results, x, y and z components of each vector have to be stored in consecutive memory locations and vector N+1 should be right after vector N (in memory) I am also not sure about the memory locations here so if the matrix is column major and the incoming pointcloud is of shape 3, 38528 a column major matrix should be optimal for consecutive memory allocations ## Reply by Alex Kushleyev (ModalAI staff) · 2024-04-04 21:26:39 UTC (in reply to Darshit Desai) @Darshit-Desai , i would say the best way to confirm storage order would be to print out the memory location of the vector’s 0th, 1st and 2nd element after you put it into eigen structure. You should be able to get the pointer using & operator and print out the pointer using printf(“%p”, ptr). If the 3 pointer values are separated by 4 or 8 (single or double precision), then storage order is optimal