Sideloading certain ZIP files results in error on-device & fail to install

[notpeter (Peter Tripp)]

Hi!

I have a PDX build which works as expected, in simulator and sideloaded via USB and play.date/account/sideload. I used the MacOS "Compress" right click option to create a ZIP of the same PDX and while the website accepts it, it fails on device when you attempt to install.

I have attached two zip archives, one created with the MacOS zip compressor (11.6.5) and one created with the command line zip (zip -r gimmefriction_clizip.pdx.zip net.notpeter.gimmefriction_1.0.0-build13_2022-05-10_232421.pdx).

gimmefriction_clizip.pdx.zip (1.5 MB) - Attachment removed
gimmefriction_osxzip.pdx.zip (1.5 MB) - Attachment removed.

The CLI-created zip file works great.
The MaxOS-created zip file fails.

I have done no on-device troubleshooting, but it appears the MacOS zip compressor includes an extraneous file:

__MACOSX/._net.notpeter.gimmefriction_1.0.0-build13_2022-05-10_232421.pdx

Which I believe is Apple's attempt to preserve the resource fork/custom icon.

My guess is that dotfile is messing up on-device -- I assume it's also related to the Windows Zip issue reported here:

Unable to Sideload pdx.zips Built with pdc on Windows

Thanks for looking into it! I thought I was going crazy there for a minute. I was able to successfully sideload the zip that you attached without getting the error. I just had an idea to try another compression utility (7zip) and it resolved the problem with both flippy fish and my own game. I was using the windows 11 default zip utility, so I guess that could be where the problem originated.

For reference you can inspect a zip without extracting by running zip -sf filename.zip

[matt (gingerbeardman)]

I do believe there is a bug report about ignoring the mac files at a system level.

Meanwhile, here is a command (Automator Service) I created to create a clean zip on macOS

Create Clean Archive.zip (16.1 KB)

Put this in ~/Library/Automator/ and it will show up in right-click the context menu > Services.

You can also assign a keyboard shortcut using System Preferences > Keyboard > Shortcuts > App Shortcuts > Finder

[notpeter (Peter Tripp)]

Word. I’m totally ok with using the CLI zip command for releases in the meantime, but figured since I had a reproducible file I wanted to share.

[notpeter (Peter Tripp)]

For reference, on device what it says is: Error
There was a problem installing this game.

[SHiLLySiT]

@notpeter I'm actually still seeing the same error, even with your latest version (1.1.2). I was able to fix the issue by unzipping and rezipping the PDX folder.

I had the same problem with another game, that was resolved with the same method. However important to note that the dev said they were on Windows

Is this maybe a bug with how the Playdate (or the sideload server) unzips archives?

EDIT: erp, I think the bug I'm seeing is related to the discussion you linked above but I suppose its still worth mentioning here.

[notpeter (Peter Tripp)]

Thank you @SHiLLySiT. I believe there is an issue with how itch processes the upload.

Specifically:

1. gimme-friction-baby-1.1.2.pdx.zip -- My original zip upload. (OK)
2. gimme-friction-baby-1.1.2.pdx -- Extracted from my zip. (OK)
3. gimme-friction-baby-playdate.zip -- Zip file downloaded from Itch (NOT OK)
4. gimme-friction-baby-1.1.2.pdx -- Extracted from the itch zip. (OK)

I have NO IDEA what is causing the issue decompressing on-device.
The pdx directory extracted from each is identical (as compared by diff -bur
As expected the files contained in the two zip files are identical, but it appears the order they appear in the archive differs. Specifically the itch zip defines all the directories first:

• gimme-friction-baby-1.1.2.pdx/images/loader/
• gimme-friction-baby-1.1.2.pdx/fonts/
• gimme-friction-baby-1.1.2.pdx/sound/
• gimme-friction-baby-1.1.2.pdx/
• gimme-friction-baby-1.1.2.pdx/images/
• gimme-friction-baby-1.1.2.pdx/images/balls/
• gimme-friction-baby-1.1.2.pdx/main.pdz
  etc

While my original (working) zip:

• gimme-friction-baby-1.1.2.pdx/
• gimme-friction-baby-1.1.2.pdx/main.pdz
• gimme-friction-baby-1.1.2.pdx/images/
• gimme-friction-baby-1.1.2.pdx/images/balls/
• gimme-friction-baby-1.1.2.pdx/images/balls/14-1.pdi
  etc

I can totally imagine that whatever software the playdate hardware is using to extract the zip archive is making an assumption about zip archive structure that is not true of this zip file.

It appears my original zip (created by the command line zip (Info-Zip 3.0 on MacOS) creates directories and then their contents (a sane structure) while the Itch tooling creates all directories first (in no particular order) and then files in those directories.

I've attached four files.

1. My original upload: gimme-friction-baby-1.1.2.pdx.zip (1.5 MB)
2. The broken Itch zip gimme-friction-baby-playdate.zip (1.5 MB)
3. zip -sf gimme-friction-baby-1.1.2.pdx.zip output orig112.txt.zip (509 Bytes)
4. zip -sf gimme-friction-baby-playdate.zip output itch112.txt.zip (509 Bytes)

I will reach out to Itch support, but in the meantime the workaround seems to be to extract pdx diretcories before sideloading at play.date/account/sideload -- cause if you sideload the itch zip you're gonna have a bad time.

[SHiLLySiT]

Hmmm. I have no problem installing other games from itch.

Also the other game I had the same error with was a zip archive directly from the developer. It was shared via a Google Drive link 🫤

[notpeter (Peter Tripp)]

My guess is that it may be because I use nested subdirectories (e.g. /images/balls/ and /images/loader ) while many folks may not. It might also happen non-deterministically (e.g. reading directories from an unordered hash table) but I'm not sure. Looking at the PKZIP specification I don't believe the Itch zip to be technically invalid (my understanding is that file/directory entries may occur in any order) but in my investigations the other zip implementations follow the somewhat sane pattern of creating parent directories before their child directories and files.

I don't know which zip library Playdate is using on device (or god forbid they created their own). But it's not surprising that there are some gremlins in a compression format created in 1989 (33 years ago) and shoehorned onto a modern device.

[notpeter (Peter Tripp)]

My guess is that it may be because I use nested subdirectories

I have confirmed this is indeed the issue. I moved all the images out of subdirectories and into the images folder. The itch zip now has a saner structure:

% zip -sf gimme-friction-baby-playdate_1.1.3.zip                         
Archive contains:
  gimme-friction-baby_1.1.3.pdx/
  gimme-friction-baby_1.1.3.pdx/images/
  gimme-friction-baby_1.1.3.pdx/fonts/
  gimme-friction-baby_1.1.3.pdx/sound/
  gimme-friction-baby_1.1.3.pdx/main.pdz

And this zip correctly installs! Yay! So the solution is just don't use subdirs! Or wait until Panic fixes their zip library or Itch fixes their zip writer. Thanks everyone for your help especially @SHiLLySiT as your comment led me down the correct thought process, so many thanks!

[timboe (Tim Martin)]

Thanks for the posts here @notpeter - I was building my initial itch zip today, and it did have one nested subdirectory. Until I read this post at least!

You could very well have saved me much annoying debugging.

[dan (Dan Messing)]

Thank you for all of the investigation into this issue!

As you figured out, the bug is triggered on the device when the first entry of the zip file list is not the root directory, so in the example problem game posted, it's gimme-friction-baby-1.1.2.pdx/images/loader/ rather than gimme-friction-baby-1.1.2.pdx (which is instead the 4th entry). I am not sure what the circumstances are that cause this to happen when creating an archive or how we haven't managed to run into the problem earlier, but we have a fix in the works that will be included in the next firmware update.