Problem Summary: Using Trampa VESC75/300 in a heavy robot that needs to be able to operate at temperatures down to -20C. Below about +20C the VESC will not run the motor smoothly at high torque/low speeds for some period of time after the VESC is turned on. For 20C this lasts about 1 minute. For lower temperatures this can be many minutes of poor performance making the robot drive very badly. Once the system is warmed up it functions smoothly with great torque and power.
Symptoms Details: During this "cold start" period the motor's motion is jerky. The motor repeatedly loses torque for periods of 0.5 to 10 seconds and during those times makes a high-pitched whining/grinding noise. On the Realtime Data, "current motor" gets stuck at some low value (10 to 30 amps) and the motor does not move despite "duty cycle" increasing. The motor is effectively cogging like a poorly tuned sensorless motor, despite having a properly set-up encoder, and running in sensored mode.
Setup Details:
- VESC is in Duty Cycle mode, controlled by PPM. For testing this problem it is manually controlled via a Futaba transmitter/reciever.
- VESC is in FOC mode.
- Motor is instrumented with AS5047 Encoder, VESC is set up to use AS5047 encoder. Encoder is configured to 3.3V. VESC is set to 3.3V. According to terminal, no encoder faults or spi transaction failures occur. Setup is mechanically solid and works smoothly once VESC is warm.
- The motor is a large in-runner. Kv = 69rpm/volt. Continuous output 3100watts.
- The battery is 14S7P made of LG HG2 18650 Cells. BMS is rated for 300A for at least a minute.
- Symptoms occur well below the erpm for sensored to sensorless transition.
Testing: I have isolated this problem to the VESC specifically. If the VESC is cold (10C) and all other system components are warm (30C), jerkiness, whining and torque loss will occur. If the VESC is warm (30C) and all other system components are cold (10C), jerkiness, whining and torque loss will not occur.
Questions:
- Is poor performance when cold a known issue on the VESC75/300? Has anyone else had similar experiences?
- Is there any known reason that this behavior would occur? Perhaps there are specific parts whose performance is degraded at low temperature like an oscillator or a DCDC converter?
- Can anyone with some VESC experience suggest tests that would help me further isolate and eliminate this problem?
Typically that has to do with the changed resistance of the motor. A cold motor has a lower resistance compared to a warm motor. If the resistance value is 5% off, it can cause issues. The VESC detects the resistance at a certain time with a certain base temperature of all components. That also includes the motor cables and motor windings. If you then go up or down in temperature, the resistance changes and the original parameters are off. To avoid that you can use a temp probe inside the motor (10K NTC) and use temperature compensation. Then the VESC controller knows the motor temp and can adjust accordingly. A good way to get an overview of parameter changes is making detections at different temperatures and note down the parameters.
Thanks Frank. I've connected a KTY84/130 inside the motor windings, plugged in to the VESC, set "Motor Temperature Sensor Type" to KTY84/130 in VESC TOOL, and set "Temp Comp" to true. Temperature readings are now correct in Realtime Data plus or minus 2 degrees C. I then redid the motor setup at the same temperature that I am testing at today (about 15C) meaning that there shouldn't even be any temperature change to compensate for. The system still demonstrates jerkiness, whining and torque loss for a few minutes after the VESC is turned on in this 15C temperature.
In previous investigations, I disassembled the robot and tested heating and cooling each component individually. Cooling only the VESC causes the cold start glitching to occur always. Heating only the VESC causes the system to work properly always. Heating or cooling the motors, encoders, batteries, or cables has no effect on system performance. These tests provide some evidence that the root of this problem may be something in the VESC itself rather than other system components like the motor.
At this point I'm unsure even what to test next to move towards a resolution. I've replicated the problem using firmware 5.1, 5.2 & 5.3. I've independently adjusted some of the FOC parameters without seeing any results. Increasing switching frequency seems to decrease the time duration of the glitching after cold start but does not eliminate it. Leaving the VESC turned on for a long time prior to trying to spin the motor does eliminate the jerky motion, but this an inconvenient workaround.
We need to look into this ourselves. Not sure what the cause could be. The unit should not care so much about these temperature fluctuations.
Have you tried run it without sensors ( AS5047P) Maybe the problem is located there. How long is the cable to teh AS5047P and do you run it from 3.3V or 5V. 3.3V gives the better performance.
This could be sensor related.
I think you are absolutely on to something. Basically no-one else has this problem, and very few people use SPI encoders with the VESC. I found some evidence elsewhere that SPI will hang up on the STM32 in cold temperature startup situations. So I will be setting up to run hall sensors instead and see if the problem is eliminated. I will be back.
Additional Info: The encoder is 3.3V. Very well shielded, short cable. No encoder faults. No spi checksum failures (found by typing "encoder" in the terminal). I've taken great pains to make it 100% reliable. The encoder temperature does not effect the results. The encoder position appears correct on realtime data even as the glitching is occurring (no jumps, no getting stuck).
Check your setup. We made a small manual to use AS5047P sensors. https://trampaboards.com/resources/manuals/207.pdf
Double checked the AS5047P encoder setup. All good. It works perfectly once the VESC itself has warmed up for some time, so incorrect wiring or mechanical setup is unlikely. Also have set up about 20 of these so far, further reducing the likelihood that a mistake would go unnoticed.
Haven't gotten the halls set up yet, but did configure the VESC to run sensorless. Results were better than expected, but still unusable for our purposes. The electrical whining and jerky operation during cold start is still present. This indicates that the problem is not related to the use of an encoder.
To Recap: If the VESC has been turned off in cold temperatures for at least 30 min, it's performance upon startup is very rough for the first few minutes. Specifically, this occurs when trying to move the motor with high torque at low speed. The problem starts becoming evident below 20 degrees C and becomes severe at 0 degrees C and below. The rough performance consists of an electrical whining noise and sudden torque loss that causes the motor to lock up repeatedly as I attempt to drive it. Through controlled experiments I have found that this problem is related to the temperature of the VESC only and not related to the temperature of any other system components. Once the VESC has been allowed to warm up for several minutes, or if it is artificially warmed up with a small heater, the problem goes away entirely and system performance is flawless.
Below is an image showing realtime data when the problem is occurring at around 10 degrees C VESC temperature. Notice that Duty Cycle rises as I attempt to drive the motor, but current gets stuck at some fairly constant value. During this time, the motor is not developing enough torque to move and the system makes a high pitched electrical whining noise. Each huge current spike is when the motor starts moving again.
Below is a second image showing realtime data when the VESC has been warmed up to 30C with a heater and working correctly. Notice that when duty cycle rises, current increases too. The motor moves silently with great torque at low speeds.
Our application demands immediate high-torque, low speed motor performance in temperatures down to -20 degrees C. This seems like a pretty reasonable goal. All electronic parts on the VESC75/300 that I've researched so far are rated to below this temperature (usually -40 degrees C). Any ideas why this problem is happening or how to resolve it?
could it be related with the battery?
My testing does not currently support the battery being a cause of the problem. That said, I've seen enough to know that quite a lot is still unknown.
Summary of Thermal Testing:
It seems to me that somehow the increasing duty cycle doesn't really go to the motor. Do you have the possibility to measure a (or better all) phases with a scope when the issue occurs? I think you will find the root cause there. Because if the duty cycle is really going up like your traces show, the current would go up for sure. But since it doesn't, probably something is going wrong with the mosfets or gate drivers.
Why does the current need to go up as the duty increases? If it's an inertial load a constant current would still give a change in velocity.
It looks more like the mosfets aren't turning on properly, or at least that's the only way I can think that you would be stuck at some constant current and that current be lower than the normal state current.
Best bet is to look at the outputs with a scope and see what is actually going on. Better would be to look at the gate drive signals as that will tell you a lot more.
Can you also try cooling the unit down, then in the console using foc_openloop 50 1000 (<current> <erpm>) to have a constant current irrespective of the duty cycle and use the data logging to record what happens. I am expecting running the command will cause it to slowly heat up, then at some point it will recover.
Another possibility is the 12v supply is not at 12v when it's cold like this. So you can try monitoring that on the AUX pin while it is heating up.
Threshold voltage goes up quite a lot with decreasing temperature.
At the same time the on resistance goes down, which could mess up detection parameters.
Gate drivers say the VDD decoupling cap should be larger for low temp.
Under voltage lockout could trigger if the 12v is dropping.
Possibly tying into this, the deadtime for the 75/300 was changed in 5.3 from 660ns to 1000ns.
https://github.com/vedderb/bldc/commit/5e02197459e64cdb4b05f65d3a496fffb...
Ah, you're right. I thought it was not speeding up, but the graphs don't show the speed at all, so this assumption is probably wrong.
Lots of good info & testing ideas. We have a scope available and will try scoping the motor phase leads while this problem is occurring. I think this is a great idea, though I may not be able to understand the results. Images to come. Also definitely going to check on that 12V supply during the warmup process. Hope to do this testing tomorrow.
More Clues:
I hooked up the scope to all phases and grounded it properly. The results are disturbing and seem to show a problem with one of the phases. This would explain why the whining and locking-up seems to happen only in certain rotor positions. I took some videos of the scope during the tests.
Cold Start: https://youtu.be/xh1S2zOZW18
Warmed up: https://youtu.be/cONJx3N-AZ0
I've also scoped the 12V line on the AUX plug during a cold start condition and once warmed up. It's extremely noisy but there is not too much difference between when it is warm and when it is cold. Voltage will oscillate between 8V and 16V presumably when the mosfets are switching. The 3.3V and 5V lines also show a lot of noise. Very hard for me to know if the electrical noise on the 12V, 5V, and 3.3V lines causes problems or is just an unrelated non-ideality in the system.
Here's the phase voltages with the 12V line scoped as well: https://youtu.be/JWdmr9L9OeA
This vid is of a different 75/300, with different firmware. Worth noting that this cold start problem occurs on most/all of the 75/300 hardware I've used, even brand new products.
Ok, so basically you found the issue. When cold one (or more) phases doesn't switch correctly. That would explain why the performance drops significantly.
Now we just need somebody like Tech to have a look at it and see if he understands the root cause.
Just ordered a 75/300 so I can test this out.
What voltage are you running at?
50 to 60 volts. Thank you for taking an interest in this. I'm really curious to hear your findings.
Used the scope to watch cold start on 4 more VESC 75/300s today. So far 6 out of 6 have serious cutting in and out on Phase B. One has the same problem on Phase A too.
@canoeman, over on the VESC Discord @TechAumNu opened up a VESC 75/300 and found some great results looking at the components and their temperature sensitivity. Check out https://discord.com/channels/904830990319485030/940683566122156073/94106....
Does your 75/300 have a black case or the silver one?
@kubark42: Thanks. Struggling to find my way around discord but I'll figure it out. Looks like a lot more is going on over there.
Happens on both the bare aluminum and the black case for me.
Have been adding flyouts and scoping lots of different pins on the gate drivers. Scoping the HO vs HS pins showed cutting in&out on phase B just like the phase voltages themselves. Experiment was derailed because after adding flyouts to just about all the pins on the phase B gate driver the problem became almost nonexistent for the one VESC I was testing. Still have 5 that squeal and stutter when cold so testing will go on. I can't help but wonder about bad solder joints.
Conclusion: (for now)
Here are some more videos to show symptoms: https://youtu.be/JPnf0ZBPd2Y https://youtu.be/Y3ltNkb0-rA
After extensive investigation by many very smart, helpful people on the discord, Vedder told me to swap out these 6X 0 ohm jumpers (circled in blue) for 4.7ohm resistors. Problem solved. No more squealing, glitching, torque loss or lousy running when cold.
Just got another batch of VESC75/300s from Trampa. They still have this problem right out of the bag on the first run. I have to open and fix every single one myself to stand any chance of good startup performance on a cold day.
Steps to generate cold start problem:
As of my most recent order in July 2022, Trampa seems to be shipping out new VESC75/300 units with the 4.7ohm gate resistors installed! This completely resolves the cold start issue without requiring any modifications by the customer. Here is a picture of the gate resistor block on a brand new 75/300:
Considering I first started posting about this problem in January, I'd say that's a pretty commendable timeline for a company to fix a hardware bug found by a customer!