I've made a sprite table with 10 frames in it. I'd like frame 1 and 5 to show for 300 ms and the others for 100, then loop around.
Is there a "built in" way to do this that I'm missing, or do I need to keep track of the frame and set the frameTime accordingly? i.e. can I pass a table of delays in as an argument when creating the animation timer?
No built-in way.
But try this https://devforum.play.date/t/lightweight-animatedimage-library/4500/25
And do note the link to Sequence part way down the page, which is a second option.
The way that link's structured looks like I authored the AnimatedImage library, which isn't the case 
I've had great experiences with it, though I have not tried to do the specific thing you're wanting to do. It takes an array of options, and one of the options is delay, I'm not sure it supports specifying different delays depending on frame. (it'd be nice if it did, and could pull this directly out of the GIF information in future)
Looks like the .delay part of AnimatedImage is just built on top of the SDK's imagetable delay option, so I'd expect that making this easier would require rewriting that part, or implementing a workaround like you mentioned in the original post.
Good spot I've put the link between < > to prevent it displaying like that