Generally speaking, the more constraints you can put in the search, the better. This will limit the number of results that have to travel from server to client.
Do as much filtering in the search constraints as you can. (Not all constraints are possible in the search constraints dialog. Like the Advanced condition is only available [usually] in :filter … though you do have :filter in API workflows and those are server side.)
What happens is everything returned by Search comes to the browser. :filter then operates on that list in memory/in the browser.
Constraints happen on the server and the :filter element at client side.
So the best is to use as many constraints possible and limit the number of :filter usage.