I did indeed!
And also the collection of 278 fan-made levels uses the same binary format. That's a big relief.
There is a challenge of differing tile sizes, where the original level uses 4x4 pixel units and I use 8x8. There is some anti aliasing needed. What do you do with a 4x4 block? Make it an 8x8 or delete it? I think I found a fair solution.
The project to auto-convert 278 fan-made Crazy-Gravity levels is complete!
With some pretty impressive - and sometimes touching - results.
For example this level by "Myriam and Esther", probably made in 1999, now resurrected to be enjoyed once more.
Unfortunately, many of these levels are huge and don't fit in the playdate's 16MB memory. We can also blame my inefficient memory format and lua's 32-bit (or 64-bit?) number format where 8-bits would have been enough to store most data in these gargantuan 400x400x5-numbers sized tables.
I need your help to sift through this massive pile of levels!
Please join the crowd-sourcing effort at Gravity Express: Fan-made levels playtesting
I ran into some out of memory issues on lots of user levels, which isn't that strange; considering this game apparently was the minecraft of the nineties.
Also had this with level 13 of the official set tho.
So I devised a memory compression approach, which I will share more about later, if it turns out it doesn't hurt performance too much.
Would someone please test this on hardware and tell me if a solid 30fps is still achieved (and whether it loads in the first place)
If needed, grab a previous build for comparison
Please check levels 1, 3, 13, and 20.
13 being the largest one, that would crash with out of memory without this compression technique.
In other news, I commissioned some start-screen and loading screen art which is being made right now.
cgpd.pdx.zip (3.1 MB)
These all load fine on device, and play fine. I didn't complete them though.
Looks fine, one of the levels had 27-29fps, the others seemed pretty solid at 30.
Memory "compression" explained
So the memory compression approach turns out to work even better than I'd hoped. It's a bit hard to measure what level of compression is achieved because the data is now in metatables, and I don't think the Simulator's memory profiler can read those. I'm estimating the gains to be about three quarters though.
It relies heavily on string packing
The idea here is that storing small numbers in tables is very inefficient because lua has only a single number type. Whether it's the number 1, 35435636, 3.133333333, 5.13E12 or 202432451351634, it all takes the same amount of memory, depending on the system that would be 32 or 64 bits.
The numbers I use for my tilemap are all positive and smaller than 256. So, 8 bits should be enough. Considering a tile map could have dimensions of 300x300 tiles, with 5 properties each, that does add up to several megabytes. Add to that the overhead of storing it in nested tables instead of a binary array, and I was looking at 7MB of tile data. That's laugably inefficient, that's why I wrote "compression" in quotes: de-bloating might have been a better term.
To the rescue comes string packing, which lets you define a data format and convert the data to raw binary. So my 5 numbers in the range of 0-255 could be stored as 5 bytes written to a file.
When provided with the same format string, the data can be read just as easily. But, you can't read it all in memory, or it'll take up just as much memory as the original table would (remember Lua's inefficient number format).
The solution, then, is to keep these packed strings in memory, and unpack them whenever a tile is needed. The memory requirements for this are the size of all packed data, plus the size of a single unpacked tile.
One problem remains: the existing game logic heavily uses the tile map, and expects it to be a 3 dimensions ([column][row][tile property]) table. Rewriting all that would be a gruesome task.
To solve this last problem, we can turn to metatables . With metatables, you can define the indexing operation. And in that indexing operation, we can unpack a tile. The implementation of loading a file is:
local brickT = {}
local format = levelProps.packFormat
local packSize = string.packsize(format)
local packSizeOffset = packSize-1
local unpackMeta = {
__index = function(tbl, idx)
-- tbl["compressed"] contains the packed string.
-- idx*5-4: each tile entry is 5 bytes long,
-- for idx 1 we should start reading at pos 1. So, 1*packSize-packSizeOffset = 1
return {unpack(format, tbl["compressed"], idx*packSize-packSizeOffset)}
end
}
local brickFile = file.open(path..".bin")
for x = 1, levelProps.sizeX do
brickT[x]= setmetatable({compressed = brickFile:read(5*levelProps.sizeY)}, unpackMeta)
end
Suppose our code needs the second property of the tile at x=5,y=3. It will be accessed as brickT[5][3][2]. What happens under the hood?
brickT is an ordinary table. brickT[5] returns a table containing all data for a column of tiles. This ,however, is a metatable without numerical entries. The only entry it has is called compressed and it is a string of 5*sizeY bytes representing a column of tiles. When we index it with [3], that entry cannot be found, and the metatable is queried. This has the __index method which calculates the byte-offset of the 3rd tile in the column-string, unpack a tile there and returns it. The returned value is a regular indexed table of size 5*, and the second entry [2] can be accessed in the regular manner. We're done.
The great thing about the approach above is that the game logic is not aware that the table data is compressed. It requests brickT[5][3][2] and gets what it expected. Note that if the code needs more of the tile than just entry [2], the tile should be stored in a local variable to prevent unpacking multiple times.
My last remaining fear was about performance: how much cpu time would the unpacking consume?
Luckily this turned out to be very minimal, perhaps 6% of the total cpu time according to the (Simulator!) profiler. There will be a compounding effect of earlier optimisation, because I already optimised the amount of redraw per frame for the tile-map. Not a lot of tile data is needed every frame.
@fosterdouglas and others have confirmed solid fps even with this technique applied; so I couldn't be happier 
!
*the actual size is 6, as the unpacking operation inserts and extra number at the end, representing the index of the next byte in the packed data. We don't mind it being there; removing it costs cpu time while it does not bring us any advantage.
That's really cool! There is always more to learn about Lua. (Does this mean all those biggest user-built levels now work? Or are some of the extreme cases still not possible?)
Can't wait for this to release!
(when viewing the video above, make sure to select the highest quality (720p), Youtube seems to make weird quality decisions for this content)
I like the sound of chiptune / midi music, but when loading a midi into a Sequence, no sound is produced. That's because a midi file only contains the score/notes and no instruments.
I wrote MIDI Master as a tool to program instruments for midi files. It is being used by someone on the community discord to compose and "master" music for Gravity Express.
Camera Controller
To avoid flickering, camera position has to be updated by multiples of two pixels. Still, we want smooth camera movement.
This camera controller uses an algorithm incorporating speed, plane orientation and speed-hold to prevent erratic movement.
Graph shows horizontal camera movement speed. Set video quality to 720p to see the graph.
Gravity was disabled for this test.
Note that I chose a worst case scenario for this test, a 2x2 dither pattern for the bricks.
Now this is what I'm talking about!
Game Launch animation
Flying the ship for the UI is a cool idea! Might be tricky for newbies, but I'm used to these kinds of games. And it's a built-in low-stakes flight tutorial.
@AdamsImmersive that's the idea indeed. I made sure to use the exact controls and physics on the start screen as in the game, including the button that rotates the ship upwards for landing. My initial idea was to have players land on top of the buttons. So it's an infinite-fuel training stage.
I like that landing idea too.
this looks so cool! apologies for what i'm sure is a dumb question, but where/how can i get this game?
Not a dumb question at all, thanks for the interest!
This is a devlog, where I post an update about the latest developments every now and then. The game has been pitched to Panic for an official release. Until I hear back from them, I can't say anything about a release date except the following. The latest statement is that season 2 will not happen this year. The official game store, Catalog, might be released this year.
appreciated and i definitely look forward to being able to play the game!
Meet your new CEO
Here is my process for a live-action onboarding video:
File size:
pdv: 1.7MB
adpcm mono 16 bit 22khz audio: 400KB
The live action video is a fantastic touch! Looking forward to playing this
HOW AWESOME IS THAT!
Great stuff!