Add sprite to a collision group but retain full collision

Maybe this is a limitation of how collision groups work, but I'm trying to use them for just some of my sprites, and can't see how to make it work.

For example I want my player sprite to collide with everything, but my enemy sprite to collide with walls and the player, but ignore collisions with other enemies. (The overlap when there are a bunch of enemies really hurts the performance.)

If I assign my player sprite, walls, and enemy to three separate groups using setGroups, they immediately stop colliding with anything. I was hoping the normal collisions would be reported if a sprite didn't also have groups assigned by collideWithGroups.

Anyone know if there's a way to do this? Thanks!

Also using playdate.graphics.sprite:setCollidesWithGroups(groups) should do.