Video encoder (work in progress, mac only)

I promised this months ago, but never found the time to get it to a minimally useable state. Until now!

1bitvideo.app.zip (856.7 KB)

It’s not pretty but it works, mostly. Open a video file or drag it to the preview window, hit the play button to preview, or Save to write out a “pdv” file to disk. That also extracts the audio and saves that separately, which I’ll explain later. It saves from the current position (set with slider) to the end of the file, or when you hit Stop. I can’t decide if that makes more sense than starting from the beginning… You can change the different settings during preview to see how they affect the output.

Dither is which dithering algorithm to use. The first four are ordered, the rest are error-diffusing and pretty similar. I’m a Stucki guy, myself.

Contrast, Brightness, Scale to fit are obvious, I hope.

Frame rate is stored in the pdv and your code uses the value to figure out which frame to show when (again, more on that later).

fwd bias is a weird one. That’s how much a pixel resists flipping, if it can avoid it. This reduces flicker and also improves compression, but too much of it causes ghosting and jumbled pixels. I think 30 is a reasonable value? It probably depends on the video.

iframe every x frames: Since the fwd bias carries error forward we need a clean frame (“iframe”, for some reason) every once in a while to reset things. And since it flips more pixels than a biased frame (delta frame, or “dframe”), the iframe can produce a noticeable flicker, especially on the device screen. Changing the rate at which iframes occur affects the flicker in unpredictable ways. Play around with it, see what works! Iframes are less compressible than dframes so more iframes equals bigger files.

add reverse step to iframes: The video api lets you grab frames at random from the file, but is optimized for playing in forward order. If you’ll be playing backwards as well as forwards, this setting adds an extra dframe to generate the previous frame from an iframe.

So about that audio file… The pdv format does not (yet) contain audio data. To play a video file synced with audio, you play the audio file in the normal way, then use its current position to figure out which frame to show. The audio file the tool saves isn’t in a format the playdate compiler recognizes so you’ll have to convert it to aif or wav yourself. (Sorry! I’ll see what it takes to make it save aif instead…) Here’s a simple video player:

playdate.display.setRefreshRate(0)

video = playdate.graphics.video.new('station')
video:useScreenContext()

audio, loaderr = playdate.sound.fileplayer.new('station')

if audio ~= nil then
	audio:play()
else
	print(loaderr)
end

lastframe = -1

function playdate.update()
	
	local frame = math.floor(audio:getOffset() * video:getFrameRate())
	
	if frame ~= lastframe then
		video:renderFrame(frame)
		lastframe = frame
	end
end

station.pdx.zip (3.4 MB)

(In retrospect I shouldn’t have used a time lapse video, but that’s what I had at hand and it’s late and I want to post this and go to bed. The sped up time isn’t a glitch, it’s supposed to look like that.)

But now let’s talk about file size. :frowning: The pdv in that sample there is only 15 seconds of 30 fps video and it’s 2.4 MB. (The audio is 44kHz mono, 1.3 MB uncompressed but would be 1/4 that if I’d ADPCMed it.) If you push the bias up and compress the audio, that’s around 10MB per minute of 30 fps video. That’s… not insignificant! I’m putting this tool out there as something you can play with if you’re interested, not to suggest you should be making giant FMV games. :slight_smile:

Let me know if you have any questions or suggestions or comments or run into problems or etc.!

11 Likes

This is cool and spurred me on to add ordered dithering to my image dithering tool.

Do i remember correctly that @Nic concluded Blue Nosie gave the “best” video quality? (most stable? highest fidelity?) But I see that it also produces larger output than Stucki or Bayer (I didn’t test any others), which is contrary to my expectations.

It would also be cool if this accepted still images, so it can be a more general purpose dithering tool for Playdate.

That’s really weird that the blue noise dither made larger files. I’d expect it to be pretty much identical to Bayer on that front, and better than Stucki. :thinking: I’ll check that out. I had Nic’s scheme working but that code was in a weird broken state when I dug this tool out of storage it, so I rolled that back. I’ll go back and find my conversation with him and see if I can get it going again.

Blue noise works really well for smooth gradients, but like with Bayer you lose fine detail. As a compromise, Nic’s algorithm checks the local gradient at each pixel; below a threshold it uses blue noise, above it uses error-diffusing.

1 Like

Also, I was thinking about the savings that can be made by optimising GIFs (eg. 2.9MB became 259KB in my situation) but those kind of optimisations will only pay back with video that has a lot of common pixels frame-to-frame.

This is great Dave!

The App is really simple to use and straightforward.

  • fwd bias is indeed a strange value to set. Can you give us a bit of indication how it works? What are the lower and upper bounds?
  • Default iframes was at 1 which might not be a great default
  • Nitpicking but I am not a big fan of the Play Button Unicode icon :slight_smile:
  • it would be great to be able to go to the next frame or previous frame with the keyboard.

I tried it on a video of La Linea with really good results. Around 3MB (video only) for 2 minutes 40 seconds. This is kind of the perfect content to have good compression but still, this is really good.

I had no problem playing back with lua but I also tried with C and I couldn’t use useScreenContext(). I only had a black screen (clearing the screen showed that the frames were properly read for some reason). I had to render the frame in a bitmap to make it work.

removed border from play button (but it’s still unicode :stuck_out_tongue:), reduced decode pipeline length so that parameter changes are visible sooner in playback:

1bitvideo.app.zip (857.4 KB)

added left/right arrow key actions for moving back/forward a frame, space bar starts and stops playing.

1bitvideo.app.zip (858.5 KB)

2 Likes

fwd bias changes the threshold for the pixel for the next frame to bias it against flipping. If an output pixel is black in the current frame and the fwd bias is 40, on the next frame that pixel’s input value needs to be over 167 instead of 127 for the output to be white.

I’ve been experimenting with super high fwd bias and iframe numbers to make really interesting ghosting effects that end up feeling like early video art. I love it!

Two weird requests:

Would it make sense for the app to double as (or also include) a .pdv player?

A downscale option like this After Effects plugin I’ve been using.

@Dave, I’m having mixed results trying to get videos longer than 60 seconds to play without being interrupted by Auto Lock. I put this in the first line of your player code, should that do it?

playdate.setAutoLockDisabled(disable)

Argument boolean should be true to disable

Yep, yep, yep. That did it. Thank you!

1 Like

Is the source for this available anywhere? Or is there a specification for the .pdv file format? I'm stuck on Windows, and I'd like to utilize video for opening and closing cinematics of my game.

Is the file size of a .pdv significantly smaller than an image table? Maybe it might be easier to just utilize the animation APIs if I don't have access to a Mac.

The video format is reverse engineered here: GitHub - jaames/playdate-reverse-engineering: Unofficial Playdate reverse-engineering notes/tools - covers file formats, server API and USB commands (unofficial resource)

Looks like he got almost everything. That unknown 1 at offset 28 is actually the pointer to the first frame, offset=0 and type=1. With the other formats I avoided redundant data like that but I think by the time I got to video I wasn't that worried about saving 4 bytes. :slight_smile: So it looks like his frame type is off by one--a 2 in the type field at offset 32 in the file means the second frame has a P-frame, not the first. Frames can also have type=3, in which case the frame data is a uint16 length followed by an I-frame data of that length then a P-frame. This is so you can step backwards from an I-frame without having to jump to the previous I-frame then apply P-frames all the way forward.

1 Like

Any chance of a Windows version here?

2 Likes

Thanks for the hints! I've just made another pass other those docs, hopefully they're more accurate now!

I've dug out my 2013 MacBook so I can use the OS X video encoder, but a PC version would be super nifty.

Any further work on this? Or a parallel Windows project?

the app uses a lot of macOS framework, even under the UI, so there's not a lot that's easily portable. I was hoping someone outside of panic would have done it for us by now. :sweat_smile:

2 Likes

For anyone interested, I made a web PDV encoder: https://pdv.hteumeuleu.com (source code is on GitHub: https://github.com/hteumeuleu/pdv). Not as fully functional as 1bitvideo.app yet, but hopefully I'll get there and it can help people not on macOS.

7 Likes