Windows, SDK 1.13.2, same behavior on actual hardware.
Using simple wireframe via lib3d with Dave's render.c fix from this earlier post, the strange camera behavior is the same whether ENABLE_Z_BUFFER and ENABLE_ORDERING_TABLE are both set to on or off.
I'd initially posted on strange 'setCameraTarget' behavior in the 'get help' section, but NaOH replied there with an observation that the SDK lib3d camera code had one or more bugs, so I did some more research and am posting more information and a test case here.
I tried starting over again using NaOH's 'kart' camera adjustment approach (using the plain-vanilla lib3d that ships with the SDK, not the modified one out on github) in my own somewhat hamfisted fashion (required pretty much a rewrite of my test module)... meaning, I made a 'player node' and added it to the scene, and moved it about in world by applying translations and rotations, then setting the camera position/target based on the player node position more or less like NaOH did in 'kart' with his modified lib3d... but after setting the camera target that way on the SDK lib3d I wound up with the same crazy results I'd gotten in the first place. At that point, I went back to my original, simpler approach of just moving the camera around in the scene in an attempt to isolate things to just the camera behavior.
In the hope that it will be useful, I put together a test case for this particular issue using the standard SDK lib3d. The idea is to provide a very simple example of how to move the camera through a scene with three cubes in it, including rotating the camera... that is, it would be if it worked - which it doesn't 
.
(lib3dTestCase2 - source and .pdx attached).
lib3dTestCase2.zip (1.5 MB)
Walkthrough: I start with the camera at 0, 1, 0 (just hovering above the origin) pointing along the positive Z axis, and three cubes are 10 units off in the positive Z distance. Without moving the camera location, just spinning the camera left or right doesn't move the cubes out of frame (as one would expect), instead the cubes spin on their individual Y axis, which is very strange. Adding some camera translation (forward/back) to the mix increases the weirdness (capture from the test case on actual hardware via 'Mirror', converted to GIF):
Code here:
import "CoreLibs/graphics"
-- Tengu Yamabushi, 4/2/2023
-- Test case to show my complete misunderstanding of lib3d camera
local gfx = playdate.graphics
local worldLoaded = false
local cameraX = 0
local cameraY = 1
local cameraZ = 0
local cameraAngle = 270 -- init: looking up the Z axis
local moveLimiter = 0.05 -- move this fraction of a unit per frame
-- intermediate 'look at' calcs
local lookX = 0 -- x on a unit circle
local lookZ = 0 -- y on a unit circle
local lookAtX = 0 -- world coords to point the camera at, based on current camera position and unit circle adjust
local lookAtY = 0
local lookAtZ = 0
local scene = nil
local rootNode = nil
local function CreateCube(x,z)
if rootNode == nil then return end
local node = rootNode:addChildNode()
local v1 = lib3d.point.new(-1,2,-1)
local v2 = lib3d.point.new(-1,0,-1)
local v3 = lib3d.point.new(1,0,-1)
local v4 = lib3d.point.new(1,2,-1)
local v5 = lib3d.point.new(1,2,1)
local v6 = lib3d.point.new(1,0,1)
local v7 = lib3d.point.new(-1,2,1)
local v8 = lib3d.point.new(-1,0,1)
local cube = lib3d.shape.new()
cube:addFace(v1,v2,v3,v4)
cube:addFace(v4,v3,v6,v5)
cube:addFace(v5,v6,v8,v7)
cube:addFace(v7,v8,v2,v1)
cube:setClosed(false)
node:addShape(cube)
node:setWireframeMode(2)
node:setFilled(false) -- MUST be called after setWireframeMode
node:setWireframeColor(1)
node:setColorBias(0)
node:translateBy(x, 0, z)
end
local function loadWorld()
scene = lib3d.scene.new()
scene:setCameraOrigin(cameraX, cameraY, cameraZ) -- set the camera to the origin
scene:setCameraTarget(cameraX, cameraY, cameraZ + 10) -- start looking down the Z axis positive direction
rootNode = scene:getRootNode()
CreateCube(0,10)
CreateCube(3,10)
CreateCube(-3,10)
end
local function dumpWorld()
scene = nil
rootNode = nil
end
local function applyCurrentAngleToCamera()
lookX = math.cos(math.rad(cameraAngle))
lookZ = math.sin(math.rad(cameraAngle)) * -1
lookAtX = cameraX + (lookX * 10)
lookAtY = cameraY
lookAtZ = cameraZ + (lookZ * 10)
scene:setCameraTarget(lookAtX, lookAtY, lookAtZ)
end
local function moveCameraForward()
local moveX = math.cos(math.rad(cameraAngle)) * moveLimiter
local moveZ = math.sin(math.rad(cameraAngle)) * moveLimiter * -1
cameraX = moveX + cameraX
cameraZ = moveZ + cameraZ
end
local function moveCameraBackward()
local backDegrees = cameraAngle - 180.0
if backDegrees < 0 then backDegrees = backDegrees + 360.0 end
local moveX = math.cos(math.rad(backDegrees)) * moveLimiter
local moveZ = math.sin(math.rad(backDegrees)) * moveLimiter * -1
cameraX = moveX + cameraX
cameraZ = moveZ + cameraZ
end
local function normalizeCameraAngle()
if cameraAngle < 0 then cameraAngle = cameraAngle + 360.0 end
if cameraAngle > 360 then cameraAngle = cameraAngle - 360.0 end
end
local function spinCameraLeft()
cameraAngle = cameraAngle - 1
normalizeCameraAngle()
end
local function spinCameraRight()
cameraAngle = cameraAngle + 1
normalizeCameraAngle()
end
local function drawHUD()
local x, y = 25, 1
gfx.drawText(string.format("Camera X,Y,Z: %3.1f, %3.1f, %3.1f", cameraX, cameraY, cameraZ), x, y)
y += 20
gfx.drawText(string.format("Camera angle: %3.1f degrees", cameraAngle), x, y)
y += 20
gfx.drawText(string.format("Unit circle x,z: %3.1f, %3.1f", lookX, lookZ), x, y)
y += 20
gfx.drawText(string.format("Camera Target X,Y,Z: %3.1f, %3.1f, %3.1f", lookAtX, lookAtY, lookAtZ), x, y)
end
function playdate.update()
if worldLoaded == false then
loadWorld()
worldLoaded = true
gfx.setImageDrawMode(gfx.kDrawModeFillWhite)
gfx.setColor(gfx.kColorWhite)
end
gfx.clear(gfx.kColorBlack)
scene:draw()
playdate.drawFPS(0,0)
local x, y = 120, 200
if playdate.buttonIsPressed(playdate.kButtonUp) then
moveCameraForward()
gfx.drawText("Moving Camera Forward", x, y)
end
if playdate.buttonIsPressed(playdate.kButtonRight) then
spinCameraRight()
gfx.drawText("Spinning Camera Right", x, y)
end
if playdate.buttonIsPressed(playdate.kButtonDown) then
moveCameraBackward()
gfx.drawText("Moving Camera Backward", x, y)
end
if playdate.buttonIsPressed(playdate.kButtonLeft) then
spinCameraLeft()
gfx.drawText("Spinning Camera Left", x, y)
end
scene:setCameraOrigin(cameraX, cameraY, cameraZ)
applyCurrentAngleToCamera()
drawHUD()
end
I imagine I'm running up against a combination of (potentially) buggy lib3d camera, my own assumptions/misunderstanding of lib3d internals, and my very fuzzy recollections from my single undergrad 3D graphics math class (35 years ago or so, now ).
I'll set the wireframe work on hold, for now (lots of other stuff to work on, and a wireframe minigame was a 'nice to have', not a 'must have', so that's ok).
Hopefully the test case proves useful to Panic - and even more hopefully, this is a PEBCAK situation that someone can point out to me, rather than an actual bug (but NaOH suspects there are some camera bugs, so... we may not get that lucky).
Be well, all. 