Blender to Zbrush workflow

Hey folks! Not sure if I'm allowed to ask this here, since Zbrush isn't covered. But maybe there are some Zbrush users in here!

So I like to export from Blender with FBX, because its handy when I have a lot of assets which I want to export at the same time. And as far as I know Zbrush prefers it's Tools to be around a size of 2, so that the brush size ect is decent. But this means I sometimes have to rescale/ unify my objects in Zbrush, but of course If I export back to blender the scale is totally off.  This is problematic when I want to bake or change stuff, then I have to eyeball it.

Am I missing something or is there a trick to solve this issue?