VESC 75/300 suited for a 4.5 tonne sailboat?

You can try this, but I would go up in voltage if you can. For best performance higher voltage and lower Amp flow is the way to go.

5KW is not a lot, sot you will get away with a 50V setup.

A 5KW BLDC is way better than a DC motor. DC motor support is not perfect yet.

Is 5KW enough to drive the boat?

Some critical information is missing to give you a good advice.

What is the expected gross weight of the boat? What was the original ICE motor horse power rating? What was the original's motor top RPM and reduction ratio to the prop? What is targeted speed and range for the EV conversion?

These things will define the approach. Higher voltage is generally good for better efficiency, faster charging but you have to take extra caution handling the battery packs and the wiring as 100v+ DC can be deadly. Remember this is also used in the water, so proper water proofing and corrosion protection should be considered.

I would totally forgo any lead acid packs though. Good maybe for a couple of years before needing replacement, terrible capacity to weight/size ratio. Li-Ion cylinder cells only, at least 3Ah per cell. Properly/safely built packs with good fusing at the cell level, emergency disconnect, BMS with overcurrent, overcharge, overdischarge and other protections. 

Are you sure about your 5kW mechanical power requirement for 4.5 tonnes boat ??  Once I worked on a 3 tonnes / 10meters sailboat, originally with  30bhp diesel to be electrified. We used 10kW dc brushed (that was 15 years ago), and it appears it was not enough - even with properly dimensioned and efficient propeller. You need some power margin to be able to react quickly sometimes.

Following, 

I'd recommend not going past 50 V. 

What you're proposing isn't crazy, but it all depends on your application.

Putting the power question in perspective, a 3.5 ton 30' sailboat will move at about 6kts with about 5-10 effective kW made by the sails. So if you're looking only to get to hull speed in calm water 5kW properly geared with a highly efficient (i.e. large diameter) propeller is an okay approach.

In my 1.5tonne sailboat (F-27 trimaran), my engine can't get me past 4kts. It's a high-thrust propeller so it will provide a bunch of forward and backwards thrust, it just won't get me going fast. But my sails take me to 18kts, so the engine is really there just to get me on and off the mooring ball.

I will echo what others said earlier about the SLA batteries. You really want nothing to do with them in this application. SLAs are terrible at high current efficiency. Go for LiFePO4 only (because you really don't want to mess with a battery fire, either).
 

BTW, I don't see the connection between the VESC and a brushed DC motor (unless I didn't realize it can control DC motors in addition to BLDC?). DC and BLDC are pretty far apart in powere electronics control scheme. If you go DC brushed, the controllers are simpler, but the brushed motor likely won't be anywhere near as efficient and the brushes might corrode in the marine atmosphere. A BLDC is probably the better choice, and they're cheap enough these days that there's no real advantage in using a brushed motor.

At the moment I use two 400w trolling motors(32lbs) on a very light 22ft ex-sailboat hull. Boat weight is around 600kg(no keel) and I manage 3.8knots at 460W. My biggest problem was that the dc motors pulled more amps at part throttle than my vescs could handle. In dc mode it seemed motor current were not correctly measured. I'm still waiting for some questions to be answered regarding DC mode myself here on this forum.

 If you have a big enough propeller with low pitch and high voltage it could work for you. I think your biggest problem is to create enough torque to spin such a large prop that a boat that size demands. Amps create torque, so try to find some documentation on how many amps that motor will pull at 30-100 percent power and make sure your vesc is able to deliver that continous. At 30 percent power the the transistors in on(conducting )about 30% of the time so there will be a lot of heat build up in the vesc.The motor will then get low voltage but draw many amps at part throttle from the vesc. If the vesc can handle that current you can increase battery voltage to get more speed. A larger prop will give less slippage so better overall efficiency and range. I think you will find your setup a bit underpowered at times but it should at least work at low speed.