So i know this isn't practical AT ALL, but has anyone played around with playing back frames of video? Back in the day I did it on Pac-Man hardware Yorgle Notebook: Full Motion Video on Pac-Man Hardware which is even less practical than playback on a playdate...
(I imagine it would work a lot better native in C++ or maybe even Lua in a real dev kit, but we're not talking about it working better... just about it working at all.
Anyway, am I going to have to write some video processing tools and make a proof of concept, or has someone else already done it?
Panic did some work with videos already. However I don't know if there is any plan to add it to pulp (converting the video in browser might be the biggest challenge).
But yeah videos on the Playdate screen is pretty interesting
I have messed around with it a tiny bit. I converted a video to 1-bit in ffmpeg and then wrote a python script that would slice the video frames into 8x8 chunks with the animation as frames within the tiles. Then mangled them into a barebones json pulp game file. You can see the captured result here: https://twitter.com/i/status/1492030187525054471
The results are interesting but even using some optimizations. Such as referencing the same frames in multiple tiles, this method is not really practical. It took over 10,000 frames in the tiles to make it work. I also ran into a hard limit with the json file. Either file size or the number of objects,. It just would not load. This is also only in the web player. I bet it would fail hard on real hardware, but I cannot test it.
I can post the .json if you are interested. I am actually very interested in explorer more ways to hack the json game file to explore and or break what is possible.
Scaling down to a resolution of 25x15 and just swapping between solid black or white tiles would presumably look awful for almost all videos but would eliminate the frame limit problem? You could subdivide the tiles and make up all the combinations, but I think the best you can do is quarter them (16 permutations) giving you a resolution of 50x30. Though that's much less fidelity than your demo at 200x120!
I suppose you might also use dithering to make some greyscale tiles if working at a lower video resolution, which might make a slight improvement to appearance.
I did a couple tests. I found there is a lower limit of resolution. It starts to look like random blocks real quick. I used a hamming distance to look for similar tiles and built a palette (although waaay more than 16). I found that a more aggressive distance threshold you lose the image very quickly. It was a great way to explore the json file structure though. I think it would be interesting to try and build an editor that would make manipulation and layout of multi-tile images easier. I have some ideas for things but trying to manage the larger tile sets is becoming a real pain. especially with objects that are designed to span multiple tiles.
I have made this Low quality "Bad Apple!" recreation in Pulp which plays fine at 64px x 48px 5 FPS (with a little lag sometimes) in the web player but can not produce a PDX file as it resulted in a 504 Gateway Time-out.
Awesome! Checking these out. I've already written up some scripts to convert a short clip (captured via "Giphy Capture") to grayscale, and reduce the resolution.
Oh, I don't expect it to look good, but it sure is awesome!
