Would you mind sharing how you solved this? (In case anyone else has the same issue)

I had to clean my mesh by merging by distance..

I flipped my normals while I was there at the same time and came back and it was all fine.

Thanks a lot, for anyone who has the following issue :

You baked you curved meshes and end up with weird UVs.

Select all the mesh in Edit Mode > Merge By Distance (basically you want to remove the bevel you added when doing the vertex painting) and bake again, it fixed the issue for me :)