I'm using VESC 6 mk III, and other parts follow MIT RACECAR: https://www.jetsonhacks.com/racecar-j/
After uploading fermware 5.2 using VESC TOOL 3.0 for motor setting, we successfully detected it in the FOC menu as shown in the following figure.
However, when I tried to turn the motor using the lower left button after motor configuration, both the DC motor and the servo motor did not move properly as shown in the following video.
DC motor
Servo motor
How can I drive the motor properly?
Please use the wizards for configuration. You can also use the latest BETA VESC-Tool. Motor detection got a lot better. https://vesc-project.com/node/2859
First thing you should try is connecting nothing to the VESC 6, other than battery, motor and Laptop via USB.
When you use the arrows, left/right and up/down do a different thing. One is using the values specified in the lower menu bar, the other is spinning up the motor to full duty cycle, using the values in the motor settings/general/current.
How are the VESC 6 connected to the other controllers? Did you check for GND loops?