I think you might be looking at lib3d the wrong way. It's an example of C code, not intended to be a fully usable Lua library.
It's part of the Playdate SDK. Why wouldn't it then be intended to be fully usable? (puzzled - is there a disclaimer I missed along the way?)
... effective for educational purposes, even with some bugs. (This isn't even the only one.)
We may have some disagreements about what a SDK is for 
.
If you're interested in solving the bug, I would encourage delving into the C code.
Oh, I have been. But I haven't done any linear algebra or 3D math since 35 years or so, so I'm more than a but rusty. Consider me a 'consumer' of lib3d, not a contributor.
I would appreciate a PR to mini3d-plus if you have the bandwidth...
Sorry, I'm old (soon to be 65), I don't understand what 'PR' means in this context ('Public Reminder'? 'Puerto Rico'? - no idea).
...or you could fork mini3d/lib3d into your own library. I think that might be more pragmatic than a bug report.
As someone who has written a few (both internal and external) SDKs of their own, IMO if there's a bug it's better to either fix, post a workaround for, or deprecate an API rather than fork.
Anyhow, it's too late, I already did 
.