I think it all depends on whether you can make it a smooth user experience. For example, if you can give them something to look at while the reload takes place it can lessen the impact and even improve the experience.
Having said that, if you have physics objects you might want to recreate them from scratch - though I expect you’ve solved that problem, if it’s there.
I don’t reload scenes, personally. I have the ‘scene’ in a module (ie: wrapped up within a custom group) and I simply destroy that group and recreate a new instance of it within the same scene. This negates the need to leave the scene at all. The extra work required to provide a smooth transition off screen etc is minimal, tbh.