@getfelix welcome to the community!
Searches in RGs are a no go if you want performance
Pre-load lists on your entries and even so it is not advisable to have lists in objects that you need to manage via RGs . Even worse if these lists are beyond 100 entries.
Re-think your dB structure and logic. Build it as inversely related as possible. There are many tricks for the logic as well but cannot go into specifics as I do not know how your app functions.
Check out this insightful resource (paid):