While tuning the motor, how does the VESC measure the magnetic flux?
If it calculates from voltage produced on each phase, then does it counts IR term with back-EMF?
Or does it assume that back-EMF = total voltage produced on each phase?
Because for most motors which is laminated, they need very small current to rotate with un-loaded condition so IR term could be assumed 0.
But the motor I'm testing is not laminated so it needs high current to rotate, which means IR term isn't negligible.
I believe it spins the motor at a certain duty cycle and measures the speed. since a constant duty cycle is basically a constant voltage the motor should draw as much current as it needs to spin at that speed.
That should make the ir value negligible unless the motor has really high losses and lots of drag. in your case, you have lots of drag (eddy losses) because of the unlaminated stator, but that is a different loss mechanism than ir losses.
I am quite concerned by an unlamented stator though. That would make a motor virtually unworkable because of the extreme eddy losses.
You can refer to the mcpwmfoc.c source file and observer_update() function to see how it works. Not exactly that simple. The underlying science is available here:
http://cas.ensmp.fr/~praly/Telechargement/Journaux/2010-IEEE_TPEL-Lee-Hong-Nam-Ortega-Praly-Astolfi.pdf
FOC forces the 3 phase voltages and measures currents (not BEFM) at high frequency (10s of kHz). From there (and mostly from the phase shift between voltage and currents - to keep it simple), it will determine the actual rotor position vs the field position.