I hope we won't need to make any more changes, and if we do we'll keep support for older builds as long as we can. What happened with 2.0 is the new hardware maps external memory to a different address, so we had to add a relocation table to the pdex.bin file. If you have a v1 device you can still run games with C API components compiled with the old SDK, but there's no way for the v2 hardware to do that. I can't predict the future but I also can't think of what would require another change to the binary format.
Oh, and I see I said
You shouldn't need to change compilation optionsā¦
up there. As we discovered with "error: compress failed: buffer is empty" error from pdc in SDK 2.0 beta 2 - #22 by DanB91 it turns out you do need to add the -mword-relocations flag to your compile options (and it's now in the SDK's C_API/buildsupport/common.mk file) so that it doesn't generate any "split" addresses that our loader can't deal with. Support for that type of address load is something we could add in the future.. but I don't think it'd be worth the hassle.
1 Like
Seems to I need help.
I tried really many options and didn't understand where I was wrong. So I have no idea for now.
My elf before pdc have correct entry-point with correct (as I suppose) relocation section.
After pdc a new elf foo.pdx/pdex.bin does not contain anything. Not really anything, mostly seems to problem with relocations.
But no errors by pdc.
1 Like
There's a bug where the compiler can't handle pdex.elf files with more than one segment, and it looks like that's the problem here. I'm not sure why there's a second segment in the file, doesn't look like there's anything in there, but its existence causes the compiler to forget how large the binary is. I posted a fixed macOS build of pdc here: "error: compress failed: buffer is empty" error from pdc in SDK 2.0 beta 2 - #17 by dave Can you put that in your SDK's bin folder and give it a try? As far as I can tell, loading and running works correctly with that in place. You get an error after that because an update callback hasn't been set and there's no Lua main.pdz file, but that's a separate issue. 
1 Like
Dave, already tried yesterday. Same result.
Iāll try tomorrow with less dummy āprojectā with update-cb.
And I donāt get a speaking error, just crash and āneed reboot, press Aā.
Double-checking that, I first ran into the problem that gatekeeper isn't letting me run that pdc build from the SDK/bin folder (but it does let me run it from the Downloads folder??) so I had to sign it with codesign -f -s - pdc. But after that I still don't get the same pdex.bin I did before. It looks like that pdc build didn't actually have the bug fix. 
Let's try this again: pdc.zip (793.3 KB)
And here's the resulting pdx I get
foo.pdx.zip (1007 Bytes)
2 Likes
Seems to IT WORKS. Iāll test and tell you later today.
Also instead of codesign Iām using just xattr -d com.apple.quarantine.
2 Likes
@Dave, Iāve tested and have to say ^that patched pdc works properly.
I have no classic giant bunch of tests, just
- not inlined functions,
- heap allocs, pointers to various parts of prog
- two or more sections.
Looks good for me. Thanks, Dave!
Also have to say, Iāve used PIC relocation model.
1 Like
@Dave, I need debug help. Could you please run foo.pdx.zip (4.8 KB) with debugger and tell me where it crashes and maybe why. Pleeeeeese 
on-device only, no sim build there
For some reason the values in the relocation table are incorrect. The first entry actually points to itself. 
Later values point to other entries, corrupting their values, which then leads to crashing when the bad values are taken as offsets to apply the relocation to. So, first thing is we need to be validating the values in the relocation table and throwing an error if they don't point to actual code. Next is figuring out where those values came from in the first place. Can you send me the pdex.elf this was compiled from? It must be another elf format thing that we're not dealing with properly.
2 Likes
I can't find exactly that elf, but I can build new and it could be reproducible
What I mean:
This is not exactly that elf (source of previously attached pdex.bin),
but it built from that code, so mostly highly probably the same. Anyway I've attached entire build - elf + pdc's output: foo.pdx.zip (12.4 KB)
Thank you so much. Probably that's my wrong 
and not a pdc problem. I don't know. I just tried to build "executable" elf from rust with rustc+ld+arm-gcc-stdlib and it looks like bad idea but interesting case.
Anyway I can build correct binaries with your latest patched pdc - it work awesome! Espetially if not experimenting more
I just looking for a way to minimise size, it needed because LTO and stripping are suffer from relocations requirement. Not so heavy. But I know potentially best option - just to build executable using one toolchain including linker and std, not to mix lime me
So, it's not critical at this point now.
I agree with what you said about needing an input/output validation tool.
Input (elf, that goes to pdc) needs more validation and proper errors output.
Output of pdc needs validation to avoid regular testing on the hardware for example. It can be something like simulator or loader. It could be really useful on CI-testing too.
Also, what tools you are using to analyse relocation tables? "Just hands and head" is ok answer if it so, just interesting.
Thanks for the pdex.elf, and sorry it took me a while to check this out. My previous fix for pdex.elfs with multiple segments was to just look for the first segment with data in it, but somehow this pdex.elf has the text and data sections in different segments. 
So that's why the relocations were all outside of the output binary, we didn't copy the data section to the output.
Here's an updated build that copies all non-empty sections to the output binary: pdc.zip (793.9 KB) When I run the resulting pdx the screen is black, but it doesn't crash or hang. When I show and dismiss the menu I get "Paused!" and "Resumed!" in the console. It crashes when I quit to the launcher, with a message like "It's finally working!" Well, sort of.. 
1 Like
Wow! Thank you so much!
Thatās the little test of allocations and relocations and thatās works finally! If so, thatās awesome I see a sea of opportunities.
Currently Iām in a little vacation because of health but later in a week Iāll test new pdc and improve build system. And then Iāll report about results here as soon as possible. Thank you Dave!
Dave, we are (I mean community) need a test plan for any produced elf, that could show validity without testing on hardware or disturbing you.
Also, what you think about opening source of pdc and pdex.bin format? It could be helpful for others. Is it possible at all?
2 Likes
Hi @dave, by any chance do you have a build of this pdc for Linux? I'm trying to help @fzzr test the changes, but I don't have a Mac. Thanks!
sure thing! Here's the Linux build: pdc.zip (248.9 KB)
And here's Windows, just for completeness: pdc.exe.zip (249.6 KB)
The bin folder in the Windows SDK has a bunch of dlls in it, not sure if any of those are required for pdc but I'd guess running this build from that folder would make it work.
1 Like
Thanks! I see that pdc in SDK 2.0.1 has the segment fix, too.
1 Like
I think that was the fix for multiple segments, but the 2.0.1 pdc will still break on elf files with text and data in different segments. I wonder how you wind up with an elf like that..
1 Like
@Dave thank you so much! 
You're saving the rusty crab :))
Probably you could describe in details low-level requirements for elf structure such as sections, ordering, positions, reloc-table(s) format, etc... And so I and everyone can build exactly what needed to pass through pdc. Or maybe somehow release latest patched pdc from this thread?.. That would be incredibly cool!
By "build elf" I mean break apart and assemble what the pdc needs 
but better way would be your official support multi-sectioned elfs definitely .
No so, partially. It works for binaries that produced by current version of crank, but that isn't optimal. Better way is build exactly one bin with rustc/llvm and only one linker like it in my latest elf in the thread. So we're need (I mean we're very want and will be awesome if it happens) that latest patch with support various multiple sections/segments.
Also, I just put two little screencasts here
First one presents build & run bin (executable) using latest pathed pdc from this thread:
Second is same but about build & run in simulator:
I'm not a pro video editor, so pardon the quality
2 Likes