"error: compress failed: buffer is empty" error from pdc in SDK 2.0 beta 2

DanB91 · May 19, 2023, 12:48am

Hi, I installed SDK 2.0 beta and would like upgrade my Zig template for Playdate to work on firmware version 2.0. However pdc is now giving me an error: error: compress failed: buffer is empty.

This is the pdc command I am running pdc --skip-unknown zig-out/Source zig-out/example.pdx. The contents of zig-out/Source: is

$ ls zig-out/Source 
pdex.dylib         pdex.elf           pdxinfo            playdate_image.png

Any ideas on what is causing this error? I updated the linker.ld file to match the new linking format of the 2.0 ELF files. Apologies for tagging you here, @dave. But I feel you might have the most insight here.

Thank you!

willco · May 19, 2023, 4:51pm

I installed zig 0.10.1 (on the Mac via brew) and the example project built without an issue using the build.0.10.1.zig build file. Any chance the build.zig file has some-sort of error?

DanB91 · May 20, 2023, 1:43am

Thanks for checking that out! Unfortunately, I did the same thing as you and got the same error message:

➜  Zig-Playdate-Template git:(main) ✗ /opt/homebrew/bin/zig build  
Skipping libpdex.dylib.o
Skipping pdex.bin
error: compress failed: buffer is empty
The following command exited with error code 1 (expected 0):
cd /Users/danielbokser/Downloads/Zig-Playdate-Template && /Users/danielbokser/Developer/PlaydateSDK/bin/pdc --skip-unknown zig-out/Source zig-out/example.pdx 
error: UnexpectedExitCode
/opt/homebrew/Cellar/zig/0.10.1/lib/zig/std/build/RunStep.zig:277:17: 0x100f5730b in runCommand (build)
                return error.UnexpectedExitCode;
➜  Zig-Playdate-Template git:(main) ✗ /Users/danielbokser/Developer/PlaydateSDK/bin/pdc --version                                        
2.0.0-beta.2

The pdc you are using is also 2.0.0-beta-2?

willco · May 20, 2023, 5:13am

I was. Do you happen to know if zig uses makefiles behind the scenes?

DanB91 · May 20, 2023, 11:58am

it does not. the build system itself is written in Zig

DanB91 · May 20, 2023, 12:04pm

Also, one thing to note, is that if I run pdc manually I get the same error. So I doubt this has anything to do with the build system, and more to do with the contents to the Source folder, or the installation of pdc iteslf. Is it possible you could send the contents of your zig-out/Source folder so I can compare?

willco · May 20, 2023, 3:38pm

What OS are you building on? Let's start there. The error you're getting is new in 2.0, I just need to figure out why you're getting it and I'm not.

willco · May 20, 2023, 3:41pm

One thing to try, remove the skip unknown flag. Does that make any difference?

DanB91 · May 21, 2023, 4:42am

I am using macOS Ventura 13.3.1. I also tried with removing the --skip-unknown flag and same error unfortunately.

DanB91 · June 13, 2023, 1:35am

Bump. Still happening with the latest SDK. Unfortunately, I cannot get my game working with 2.0 unless I can get past this, I believe.

dave · June 13, 2023, 3:29am

When we added --skip-unknown we also added --verbose to print which file it's compiling. Does that help at all?

DanB91 · June 13, 2023, 3:45am

Thanks for the response! Unfortunately no, it just says the same error with no new info. Maybe I am using the flag wrong? (Behaves the same when adding the --skip-unknown flag as well)

➜  Zig-Playdate-Template git:(main) ✗ /Users/danielbokser/Developer/PlaydateSDK/bin/pdc  --verbose zig-out/Source zig-out/example.pdx  
error: compress failed: buffer is empty
➜  Zig-Playdate-Template git:(main) ✗

However. I did notice it copied over only pdex.dylib to example.pdx

➜  Zig-Playdate-Template git:(main) ✗ ls -ltr zig-out/Source    
total 1616
-rwxr-xr-x  1 danielbokser  staff  697824 Jun 12 23:33 pdex.dylib
-rwxr-xr-x  1 danielbokser  staff  115452 Jun 12 23:33 pdex.elf
-rw-r--r--  1 danielbokser  staff     248 Jun 12 23:47 playdate_image.png
-rw-r--r--  1 danielbokser  staff     123 Jun 12 23:47 pdxinfo
➜  Zig-Playdate-Template git:(main) ✗ ls -ltr zig-out/example.pdx
total 1368
-rw-r--r--  1 danielbokser  staff  697824 Jun 12 23:47 pdex.dylib
➜  Zig-Playdate-Template git:(main) ✗

And here is the pdxinfo

name=Hello World Zig
author=Daniel Bokser
description=A small demo of the Zig API
bundleID=com.panic.hellozigapi
imagePath=

I can attach the Source folder if that would be helpful.

DanB91 · June 13, 2023, 3:52am

I just noticed that if I delete pdex.elf, pdc succeeds, so that is probably the culprit. readelf output:

➜  Zig-Playdate-Template git:(main) ✗ readelf -h zig-out/Source/pdex.elf
ELF Header:
  Magic:   7f 45 4c 46 01 01 01 00 00 00 00 00 00 00 00 00 
  Class:                             ELF32
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              EXEC (Executable file)
  Machine:                           ARM
  Version:                           0x1
  Entry point address:               0x1
  Start of program headers:          52 (bytes into file)
  Start of section headers:          114732 (bytes into file)
  Flags:                             0x5000400, Version5 EABI, hard-float ABI
  Size of this header:               52 (bytes)
  Size of program headers:           32 (bytes)
  Number of program headers:         3
  Size of section headers:           40 (bytes)
  Number of section headers:         18
  Section header string table index: 16

dave · June 13, 2023, 12:52pm

yeah, looking at the changes in 2.0 I'm guessing the code that builds the relocation table isn't working right in this case. Would you mind posting or DMing me the pdex.elf file so I can check what's going on?

DanB91 · June 13, 2023, 4:30pm

Ah it might be that. I compiled the Hello World C example and noticed that .rel.* sections are missing in my pdex.elf:

readelf -S zig-out/Source/pdex.elf
There are 18 section headers, starting at offset 0x1c0b8:

Section Headers:
  [Nr] Name              Type            Addr     Off    Size   ES Flg Lk Inf Al
  [ 0]                   NULL            00000000 000000 000000 00      0   0  0
  [ 1] .text             PROGBITS        00000000 010000 000657 00 AXMS  0   0  8
  [ 2] .data             PROGBITS        00000657 010657 000001 00  AX  0   0  1
  [ 3] .bss              NOBITS          00000658 010658 000030 00  WA  0   0  4
  [ 4] .debug_loc        PROGBITS        00000000 010658 000481 00      0   0  1
  [ 5] .debug_abbrev     PROGBITS        00000000 010ad9 0001f3 00      0   0  1
  [ 6] .debug_info       PROGBITS        00000000 010ccc 0035cc 00      0   0  1
  [ 7] .debug_ranges     PROGBITS        00000000 014298 000198 00      0   0  1
  [ 8] .debug_str        PROGBITS        00000000 014430 00449d 01  MS  0   0  1
  [ 9] .debug_pubnames   PROGBITS        00000000 0188cd 000b26 00      0   0  1
  [10] .debug_pubtypes   PROGBITS        00000000 0193f3 001358 00      0   0  1
  [11] .ARM.attributes   ARM_ATTRIBUTES  00000000 01a74b 000049 00      0   0  1
  [12] .debug_frame      PROGBITS        00000000 01a794 000428 00      0   0  4
  [13] .debug_line       PROGBITS        00000000 01abbc 000d45 00      0   0  1
  [14] .comment          PROGBITS        00000000 01b901 000013 01  MS  0   0  1
  [15] .symtab           SYMTAB          00000000 01b914 000290 10     17  35  4
  [16] .shstrtab         STRTAB          00000000 01bba4 0000bc 00      0   0  1
  [17] .strtab           STRTAB          00000000 01bc60 000457 00      0   0  1
Key to Flags:
  W (write), A (alloc), X (execute), M (merge), S (strings), I (info),
  L (link order), O (extra OS processing required), G (group), T (TLS),
  C (compressed), x (unknown), o (OS specific), E (exclude),
  D (mbind), y (purecode), p (processor specific)

vs the Hello World pdex.elf:

readelf -S Source/pdex.elf 
There are 27 section headers, starting at offset 0x32a10:

Section Headers:
  [Nr] Name              Type            Addr     Off    Size   ES Flg Lk Inf Al
  [ 0]                   NULL            00000000 000000 000000 00      0   0  0
  [ 1] .text             PROGBITS        00000000 010000 000171 00  AX  0   0 16
  [ 2] .rel.text         REL             00000000 02b5d4 000070 08   I 24   1  4
  [ 3] .data             PROGBITS        00000174 010174 000014 00  WA  0   0  4
  [ 4] .rel.data         REL             00000000 02b644 000008 08   I 24   3  4
  [ 5] .bss              NOBITS          00000188 010188 000008 00  WA  0   0  4
  [ 6] .debug_info       PROGBITS        00000000 010188 00c67e 00      0   0  1
  [ 7] .rel.debug_info   REL             00000000 02b64c 0041f0 08   I 24   6  4
  [ 8] .debug_abbrev     PROGBITS        00000000 01c806 00065d 00      0   0  1
  [ 9] .debug_loc        PROGBITS        00000000 01ce63 0003cb 00      0   0  1
  [10] .rel.debug_loc    REL             00000000 02f83c 0003c0 08   I 24   9  4
  [11] .debug_aranges    PROGBITS        00000000 01d22e 000060 00      0   0  1
  [12] .rel.debug_a[...] REL             00000000 02fbfc 000040 08   I 24  11  4
  [13] .debug_ranges     PROGBITS        00000000 01d28e 000068 00      0   0  1
  [14] .rel.debug_ranges REL             00000000 02fc3c 0000a0 08   I 24  13  4
  [15] .debug_macro      PROGBITS        00000000 01d2f6 002507 00      0   0  1
  [16] .rel.debug_macro  REL             00000000 02fcdc 002bc8 08   I 24  15  4
  [17] .debug_line       PROGBITS        00000000 01f7fd 000a09 00      0   0  1
  [18] .rel.debug_line   REL             00000000 0328a4 000030 08   I 24  17  4
  [19] .debug_str        PROGBITS        00000000 020206 00aebc 01  MS  0   0  1
  [20] .comment          PROGBITS        00000000 02b0c2 000079 01  MS  0   0  1
  [21] .ARM.attributes   ARM_ATTRIBUTES  00000000 02b13b 000034 00      0   0  1
  [22] .debug_frame      PROGBITS        00000000 02b170 0000cc 00      0   0  4
  [23] .rel.debug_frame  REL             00000000 0328d4 000060 08   I 24  22  4
  [24] .symtab           SYMTAB          00000000 02b23c 000300 10     25  35  4
  [25] .strtab           STRTAB          00000000 02b53c 000097 00      0   0  1
  [26] .shstrtab         STRTAB          00000000 032934 0000dc 00      0   0  1

What gcc flags in the Makefile enable the relocation table? Is it -mword-relocations and -Wl,--emit-relocs?

I have attached the pdex.elf. Thank you very much for looking at this!

pdex.elf.zip (18.6 KB)

DanB91 · June 13, 2023, 4:36pm

Quick update, I just enabled relocations, but unfortunately, I do have the same error. These are the new sections:

There are 26 section headers, starting at offset 0x1f5e4:

Section Headers:
  [Nr] Name              Type            Addr     Off    Size   ES Flg Lk Inf Al
  [ 0]                   NULL            00000000 000000 000000 00      0   0  0
  [ 1] .text             PROGBITS        00000000 010000 00064f 00 AXMS  0   0  8
  [ 2] .data             PROGBITS        0000064f 01064f 000001 00  AX  0   0  1
  [ 3] .bss              NOBITS          00000650 010650 000030 00  WA  0   0  4
  [ 4] .rel.text         REL             00000000 010650 000130 08   I 23   1  4
  [ 5] .debug_loc        PROGBITS        00000000 010780 000481 00      0   0  1
  [ 6] .rel.debug_loc    REL             00000000 010c04 000098 08   I 23   5  4
  [ 7] .debug_abbrev     PROGBITS        00000000 010c9c 0001f3 00      0   0  1
  [ 8] .debug_info       PROGBITS        00000000 010e8f 0035cc 00      0   0  1
  [ 9] .rel.debug_info   REL             00000000 01445c 002b70 08   I 23   8  4
  [10] .debug_ranges     PROGBITS        00000000 016fcc 000198 00      0   0  1
  [11] .rel.debug_ranges REL             00000000 017164 0002e0 08   I 23  10  4
  [12] .debug_str        PROGBITS        00000000 017444 00449d 01  MS  0   0  1
  [13] .debug_pubnames   PROGBITS        00000000 01b8e1 000b26 00      0   0  1
  [14] .rel.debug_p[...] REL             00000000 01c408 000008 08   I 23  13  4
  [15] .debug_pubtypes   PROGBITS        00000000 01c410 001358 00      0   0  1
  [16] .rel.debug_p[...] REL             00000000 01d768 000008 08   I 23  15  4
  [17] .ARM.attributes   ARM_ATTRIBUTES  00000000 01d770 000045 00      0   0  1
  [18] .debug_frame      PROGBITS        00000000 01d7b8 000424 00      0   0  4
  [19] .rel.debug_frame  REL             00000000 01dbdc 000270 08   I 23  18  4
  [20] .debug_line       PROGBITS        00000000 01de4c 000cee 00      0   0  1
  [21] .rel.debug_line   REL             00000000 01eb3c 000130 08   I 23  20  4
  [22] .comment          PROGBITS        00000000 01ec6c 000013 01  MS  0   0  1
  [23] .symtab           SYMTAB          00000000 01ec80 000390 10     25  51  4
  [24] .shstrtab         STRTAB          00000000 01f010 000140 00      0   0  1
  [25] .strtab           STRTAB          00000000 01f150 000493 00      0   0  1

Here is the new pdex.elf:
pdex.elf.zip (22.6 KB)

dave · June 15, 2023, 4:01am

I don't know much about ELF internals, but stepping through the processing code in the debugger I see an obvious bug: the bit that finds the total code size only winds up using the last segment, then the later code always uses the first segment. For whatever reason your pdex.elf has three segments: text and code in the first, bss in the second, and then.. bss also in the third but the type is GNU_STACK? I don't know what that's about, but bss doesn't take up space in the file so the compiler marks the total data size as zero, then it doesn't copy anything to the code buffer and then the compression function complains that it doesn't have anything to compress. It looks like our C API examples wind up with just one segment containing text, data, and bss, so I'd guess that's what Marc was testing against when he wrote that code and he didn't hit that bug.

I changed that first size checking loop to break when it sees a segment with non-zero code size and note which segment that is so it can copy its code later on. I'm assuming here you'd only ever have one segment with code+text? Here's a macOS build with that change. Not sure though what all you'll have to do to convince the OS that yes you really want to run this random binary you downloaded..

pdc.zip (793.4 KB)

DanB91 · June 15, 2023, 9:20pm

Wow thank you very much! Unfortunately, I still get the same error with your new pdc build. BUT, that was a huge insight that it the fact that it was due to the fact my ELF defined more than one segment. So I followed this c - How to put 2 sections in 1 segment (Using ld scripts) - Stack Overflow and pdc (this new version and also the original 2.0 version) now works and I now have a proper .pdx file!

However, while it does work in the simulator, it crashes with an e0 error on hardware. I have even simplified my test program to just call playdate->system->setUpdateCallback(update_and_render, null) and update_and_render just returns 1, but it still crashes. The disassembly is below.

I have attached the new pdex.elf and pdx, which includes the pdex.bin file. Would you be able to take a look and see why it is crashing?

Thank you very much for all your help!


zig-out/Source/pdex.elf:	file format elf32-littlearm

Disassembly of section .text:

00000000 <eventHandler>:
;     switch (event) {
       0: 09 b1        	cbz	r1, 0x6 <eventHandler+0x6> @ imm = #2
;     return 0;
       2: 00 20        	movs	r0, #0
       4: 70 47        	bx	lr
       6: 80 b5        	push	{r7, lr}
;             playdate.system.setUpdateCallback(update_and_render, null);
       8: 00 68        	ldr	r0, [r0]
       a: 00 21        	movs	r1, #0
       c: 02 6a        	ldr	r2, [r0, #32]
       e: 40 f2 21 00  	movw	r0, #33
      12: c0 f2 00 00  	movt	r0, #0
      16: 90 47        	blx	r2
      18: bd e8 80 40  	pop.w	{r7, lr}
;     return 0;
      1c: 00 20        	movs	r0, #0
      1e: 70 47        	bx	lr

00000020 <main.update_and_render>:
;     return 1;
      20: 01 20        	movs	r0, #1
      22: 70 47        	bx	lr

elf_and_pdx.zip (13.8 KB)

dave · June 17, 2023, 2:28pm

Okay, turns out the problem is the code is using movw at 0xe and movt at 0x12 to load the address of the update_and_render function to pass it to setUpdateCallback(), and it puts R_ARM_THM_MOVW_AB/R_ARM_THM_MOVT_AB entries in the relocation table to match, but our loader doesn't know how to deal with that relocation type so the compiler ignores it. The address doesn't get fixed up at load time, then the system jumps to the wrong address for the update function.

Do you know if the -fPIC flag was set at link time here?

dave · June 19, 2023, 4:43am

Quick note: It looks like the -mword-relocations compiler flag might work around this, forcing it to use word-sized addresses in the literal pool instead of splitting them with movt/movw.