Is there a complete documentation for the communication protocol between the vesc tool and the board?
The sourcecode is the best documentation you can have, but be aware that the firmware and the PC ui / phone apps are closely coupled protocol wise. New config options in new firmware result in an extension of the protocol that you have to cope with else it won't work.
Source code seems to be very poorly commented. A lot of functions are hard to understand.