Since option sets are loaded onto every page (regardless of whether they are being used on that page), yes technically they slow down the page load. But most option sets are light text fields, so any page load slowdown is incredibly small. However, if your option set is 50 largish images, then I could see that causing a noticeable slowdown. Best way is to check google dev tools, in the network tag, where you should be able to see the option set and the time it takes to load.
Also, I think option sets may be cached (only loaded on initial page loads), but I can’t remember. The answer is probably somewhere in the forum or documents.
Additional thought… I wonder if an image is part of an option set, if the image file is being downloaded itself, or just the link to where the image is stored in AWS. Again, google dev tools would show you.