Support for Swift Runtime

Glad you're making some headway on this. I had looked at swift-embedded prior to forging ahead with just the dylib approach but gave up after making very little progress. I wish I could follow it better and contribute but hopefully that day will come.

5.2MB is big but not obviously a deal-breaker, right? That will fit in memory unless there's some limit I'm not aware of. We can work on improving the footprint down the road if we get that far.

Obviously we'll need devices in our hands to make this real. In the meantime, I suspect you could find a volunteer to try it on a device if you ask in the forums, once you have a build you think is worth trying.

Exciting stuff!

1 Like

Brief update on my progress:

I wanted to try a more interesting/realistic app requiring some non-trivial API coverage, so I ported the SpriteGame example from the SDK. That involved dealing with managing allocation (and freeing) for Bitmap, Sprite, and collision-related objects, and some strange callback shenanigans. Anyway it's all working, the Swift API is a pleasure to use, and I pushed it in case anyone wants to check it out.

As before: GitHub - mossprescott/swift-pd: Bare-bones support for Playdate console apps in Swift. A work-in-progress.

1 Like

Looking in to how SwiftWasm and AVR swift work. I’m thinking it could make sense to make changes to the compiler myself. The overall process of adding a new triple itself isn’t difficult and it appears LLVM has some level of support for thumbv7. It also may be interesting to take a look at how zig works, since it’s also llvm based.

But starting from bottom and working up may be the best way to go at this point.

I don’t think the bins I’m getting out of the compiler would work either and they’re about 2x larger than they should be.

Edit: I think one basic approach would be trying to cross compile parts of the swift runtime using the same settings as you would with typical PD games.

I have had yet another shower thought on this problem. During my exploration of swift being cross compiled for different platforms, SwiftWasm came up a few times and it’s also upstreamed into swift itself (mostly).

WASM has a pretty good performance profile and from my experience with using JS via espruino on even more limited platforms, realized that we could probably utilize an interpreter such as wasm3 (built with such limited platforms in mind) to execute SwiftWasm code.

This leaves us with creating a toolchain that is something like:
Swift -> Wasm -> armgcc or cc.

One of the larger “headaches” is WASI support, which is sort of like the syscalls of WASM. I’ve had decent success implementing syscalls for other playdate projects (like SDL2) and think it should be reasonable to port. There are also other WASM runtimes which allow fine grain disabling of WASI modules.

It’s possibly less than ideal to get swift running on a device this way but actual performance will determine if that is the case, plus, we could support even more than swift if it works well (and possibly provide just a static lib that is linked instead of fussing with a c project)

Anyway, I’m currently revamping a proper cmake toolchain and I’ll use this to build an example!

Edit: Well, it can at least run in sim. Tested using the SwiftWASM 5.5 toolchain using WAMR, which is promising enough on the porting side of things for running on device. WAMR has a route for adding new platforms.

That approach doesn't appeal to me as much as compiling/linking directly for the target platform. For one thing, won't you need a garbage collector? I know even less about WASM than I do about the Swift compiler, unfortunately, but I feel more optimistic about getting a good result from LLVM than by adding another layer.

If I get any further with the compiler/linker, I'll update here.

1 Like

It is actually quite performant surprisingly ha. I agree, I looked into SwiftWASM specifically because it will be a similar approach for running on playdate.

I’m going to attempt to fiddle with the compiler again this evening with the goal of getting the new triple added and cross-compiling parts of the runtime.

Some potential problems:

  • foundation will almost certainly not be supported at first (and never entirely) as it relies on libc we can’t use.
  • Unicode support is pretty big
  • I think parts of stdlib also require clib, so more stubbing might be necessary
  • allocators need to use the one provided by the C sdk, that shouldn’t be too problematic but it is something to note, as using simulator often masks this problem (once we are linking swift dylibs based on the pd allocator we should see changes to the malloc log we don’t see now)

Some of the better news is that once we can compile, linking shouldn’t be so bad, and things should pretty much “just work”.

Somewhat related but I think it would be way easier to compile swift for something with an RTOS, like freeRTOS or mbed. Then you wouldn’t need to deal with all the libc headaches. We can’t take that approach since we want access to the C api and I’m pretty sure the only way to get it is through the event handler.

Ninja edit: we also won’t be supporting things like concurrency or libdispatch (no GCD) anytime soon since there is no threading we can latch onto. Again, another point for the RTOSes, they can provide that. We should look toward SwiftWASM for how they handled the libc problem. WASI is similar to clib but definitely not the same.

Edit 2: I have the swift project setup with targets and presets for playdate, including targeting the correct architecture, it gets pretty far along with trying to cross compile the runtime but I have quite a bit more work to do in order to ensure that all the right tools are being used. On a side note, it appears we can likely include the playdate c api as part of the stdlib, so when using this toolchain you will get that out of the box! I also have plenty of stubbing / things to disable. WASI is helpful in that we have somewhat similar limitations, and referencing embedded-swift + the madmachine repos is proving fruitful as well. Ideally, this won’t become as easily defunct as previous attempts for providing this functionality.

Edit 3: making it a little over halfway through compiling the runtime, the new concurrency changes are making a lot of assumptions about the existence of threads. There are levers to turn most of that off, but it’s not used everywhere it seems.

Edit 4: I’m able to get a substantial amount of the runtime to compile & link but there are bugs in the swift side of compiling stdlib. Went too far down the rabbit hole trying to keep everything within the swift build scripts themselves. Going to revert my edits to the scripts and instead attempt to build and link standalone, with the intention of putting the scripts backs. Most of the edits in source can stay, they’re straight forward and shouldn’t change. Some of the checks can be cleaned up (the messy script problem started seeping into everything), but otherwise it should be good. A big problem I was having with the baremetal approach to scripting was how it glommed on to the other Apple / Darwin based SDKs in order to get similar treatment. It’s an extremely brittle approach as these build scripts change between minor versions of swift in a pretty substantial way. There is not much work that actually is to be done in order to support a new platform, it’s figuring out the right incantations in the build script that is hard.

Edit 5: went back to the approach that adding WASI support took, including adding playdate as a proper OS to llvm. Having stared at all of this code long enough and seeing approaches people have taken inside and outside of the swift project, this one seems to most manageable to get working, further, it should make keeping up with a fork easier. Most of the other attempts at this seem to alter swift in such a way that it’s very hard to keep in sync with upstream. It could also be worth upstreaming this code, with the biggest difference from the other embedded attempts being that we aren’t exactly targeting baremetal. This means less change in general to the overall swift project, is actually more correct, and doesn’t open the can of worms that attempting to bring true baremetal support would. Another benefit would be the folks who keep refactoring the build scripts would need to do (most) that work for us, which is also the bulk of the work typically. More challenging is creating an environment for running stdlib tests, which would be required for upstream, challenging because it needs to be run on qemu or the like in a semi-hosted fashion.

I have successfully compiled a single piece of runtime!

/Users/ericlewis/developer/swift-toolchain-exp/host-build/Ninja-Release/llvm-macosx-arm64/bin/clang++ --target=thumbv7em-unknown-playdate -DGTEST_HAS_RTTI=0 -DLLVM_DISABLE_ABI_BREAKING_CHECKS_ENFORCING=1 -DSWIFT_INLINE_NAMESPACE=__runtime -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I/Users/ericlewis/developer/swift-toolchain-exp/target-build/swift-stdlib-playdate-thumbv7em/stdlib/public -I/Users/ericlewis/developer/swift-toolchain-exp/swift/stdlib/public -I/Users/ericlewis/Developer/swift-toolchain-exp/swift/stdlib/include -I/Users/ericlewis/developer/swift-toolchain-exp/target-build/swift-stdlib-playdate-thumbv7em/include -I/Users/ericlewis/Developer/swift-toolchain-exp/swift/include -I/Users/ericlewis/developer/swift-toolchain-exp/target-build/llvm-playdate-thumbv7em/include -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wdelete-non-virtual-dtor -Wno-comment -fdiagnostics-color -DOBJC_OLD_DISPATCH_PROTOTYPES=0 -O3 -DNDEBUG -fvisibility=hidden -fvisibility-inlines-hidden  -fno-exceptions -fno-rtti -Werror=gnu -Werror=c++98-compat-extra-semi -DswiftCore_EXPORTS -DSWIFT_SUPPORT_OLD_MANGLING=0 -DSWIFT_STDLIB_HAS_TYPE_PRINTING -DSWIFT_TARGET_LIBRARY_NAME=swiftDemangling -DSWIFT_RUNTIME -target thumbv7em-unknown-playdate -O2 -g0 -DNDEBUG -I/ -DSWIFT_OBJC_INTEROP=0 -DSWIFT_LIBRARY_EVOLUTION=0 -DSWIFT_STDLIB_SUPPORT_BACK_DEPLOYMENT -DSWIFT_ENABLE_REFLECTION -DSWIFT_STDLIB_HAS_DLADDR -DSWIFT_STDLIB_HAS_DARWIN_LIBMALLOC=1 -DSWIFT_STDLIB_HAS_STDIN -DSWIFT_STDLIB_HAS_ENVIRON -DSWIFT_STDLIB_HAS_LOCALE -DSWIFT_STDLIB_SINGLE_THREADED_RUNTIME -DSWIFT_RUNTIME_OS_VERSIONING -DSWIFT_STDLIB_SHORT_MANGLING_LOOKUPS -DSWIFT_STDLIB_ENABLE_VECTOR_TYPES -DSWIFT_STDLIB_SUPPORTS_BACKTRACE_REPORTING -D SWIFT_STDLIB_ENABLE_UNICODE_DATA -MD -MT stdlib/public/CMakeFiles/swiftDemangling-playdate-thumbv7em.dir/__/__/lib/Demangling/Context.cpp.o -o Context.cpp.o -c /Users/ericlewis/developer/swift-toolchain-exp/swift/lib/Demangling/Context.cpp -I/Users/ericlewis/Downloads/gcc-arm-none-eabi-10.3-2021.10/usr/include/c++/10.3.1 -I/Users/ericlewis/Downloads/gcc-arm-none-eabi-10.3-2021.10/usr/include/c++/10.3.1/arm-none-eabi --sysroot=/Users/ericlewis/Downloads/gcc-arm-none-eabi-10.3-2021.10

Yes, this is using the wrong arm toolchain but it also works with the playdate distributed one, and yes some of the flags are most certainly incorrect. There are also some changes that I made to the folder in order for this to work, such as moving c++ bits to the top level and hacking on this to include newlib. Another noteworthy change is this is not using arm-none-eabi-gcc, but rather using clang and LLVMs support for the thumb architecture (which is considered experimental). Changing the compiler used is definitely a more complex task so my first effort is going to be seeing if what clang outputs is usable on device. The thesis mentions this as a possible path forward and instead decides that it would be easier use the entire toolchain provided by arm. I have a feeling this would most likely impact the linker step (I have only compiled, not linked yet), but that should be resolvable since WASI also uses it's own linker. We could indeed change any of this behavior in our PlaydateToolChains as well and if we need to switch compilers / linkers / etc, it would all happen there & in the driver. Getting the right include flags in place is the next step so I can continue compiling the rest of the runtime. It is easy enough to do so in a hacky way, so I may attempt that first, then move it to the ToolChains file (there is a non-trivial amount of work to creating such a file). Another worthy bit of investigation is around the swift stdlib freestanding flag, which appears to be helpful when you don't have a clib, (we sorta do, in newlib, but it needs to be stubbed / some features don't work)

Edit: another bit of consideration to have is that we are needing to use an entirely new swift toolchain, which becomes slightly problematic when we consider how compilation works now. I am unsure if we are able to completely alter the compiler behavior based on wether or not we are creating an executable or dynamic lib. This may require investigation into how building for iOS for sim or device works. It could very well be that we simply change the SDK we are pointed at to one that supports x86/aarch64, but again, I am not sure. For now the workaround is to simply switch toolchains depending on what you are targeting. It is possible that we need to alter this behavior in spm.

Aside: @mossprescott do you think we should move the dev progress of these things to this thread: Using the swift programming language on Playdate - #11 by ericlewis, since it is technically in dev by us.

2 Likes