CTRL+Click wrong selection

have 4 groups with 4 buttons, one in each one of them (also some text in each group)

I click a button, then ctrl+click 2nd button and it selects the groups too instead of just do multi-selection of those two buttons