I suspected probably it is because of the fused local position by vision and searched in parameter. It sounds EKF2_GPS_CTRL is probably relevant. So, I set it to be 0 from 7. After trying that, my outdoor missions were all smooth without any accident.
I then set it back to 7 and tried flying in position mode. This error happened again and with altitude mode was forced.
So, anybody here know how to fix the problematic local position estimate? This is an AI drone and it shouldn't perform worse than drones without any vision.