Sounds like a cool idea! Maybe the invert function is too much for the hardware, or has a memory leak. I bet there are ways to make it work though! Your limited-area method might do the trick. (And you can't hurt a Playdate by running a game... not to worry.)
Re the character being behind things... is the character a sprite? You should be able to set that in front of other sprites (and of course the background) pretty easily.
You might need to mark it as sprite:markDirty() and/or do a sprite.update() in the update() loop if you're not already? Usually sprites mark themselves dirty in my experience, but maybe if the player hasn't moved on the screen that could be an issue?