Pdc on Windows doesn't seem to support import path relative to the current file

This is, at a minimum, a report of a different behaviour for pdc on macOS and Windows.

Easy way to replicate is to clone my game repo, get a Windows Powershell in the project's folder and typing:

pdc ./source

This give the error:

error: ./source\../toyboxes/toyboxes.lua:10: No such file: github.com/DidierMalenfant/Aspen/source/Aspen.lua

This path does exist and is relative to the file toyboxes.lua which contains this import.

On macOS this works fine but it seems like on Windows, pdc seems to expect all the paths from being relative to the original source folder which is a bit inconvenient.

Did I miss something on my end?

Hmmm I'm not sure if importing files outside of the folder is officially supported. The docs don't explicitly say this, but the recommended project structure has all files under the same parent folder. The docs say "Place all scripts and assets together in a single project directory", and in the example structure, all Lua files are under the same "source" directory.

With that said, PDC is including the SDK Lua files from the SDK directory, but I'm not sure if those are a special case :confused:

There is a limitation currently, you have to add the file extension. See this post: Import statement needs .lua extension for paths containing `..` - #3 by DidierMalenfant

Edit: Ha, this is your post. :slight_smile: It works for me on Windows in a simple example project if I add the lua extension. However, relative upward paths are definitely YMMV at this point.

Hmmm I'm not sure if importing files outside of the folder is officially supported.

Look to me like a bug. It's not working consistently on both platforms so there's definitely an issue there.

it works for me on Windows in a simple example project if I add the lua extension.

Yeah, my case already has .lua at the end of that path.

I wonder if, on Windows, they just treat relative paths as always relative to the source folder maybe? That would be odd but would explain the behaviour I'm seeing. I should test some more on a Windows machine.

Did some further tests and I could confirm that, on Windows at least, paths for an import are always relative to the initial folder where pdc was invoked and not the file's actual folder.

If this can be added to the list together with that other one it would be great.

This bug, of course, is a bit more important since it currently prevents code from being cross-platform.

Any updates on this for our Windows devs?

I found the bug where it needs the lua extension when you're importing through a parent folder: my "return the file extension" function wasn't checking for the path separators as it walked backwards from the end of the filename, so it thought the imported file already had an extension and didn't append .lua. You'd get the same problem with any folder with a dot in its name.

But about the Windows problems: I suspect the problem is that pdc is expecting backslashes on Windows but we're using a UNIX style filesystem interface that handles forward slashes. That would explain why we haven't had a flood of complaints that our examples have the slashes backwards.. Here's a pdc.exe build with the path separator swapped:
pdc.exe.zip (224.1 KB)

If it's even possible to run it (I can't test it here, sorry!) can you check if it works better? Thanks!

If it's even possible to run it (I can't test it here, sorry!) can you check if it works better? Thanks!

I was also using OS-centric slashes for generating my lua files so this builds now chokes on the backslash. But if I change that to use forward slashes everywhere when generating Lua, this seems to build fine now :hugs:.

I can't check the resulting pdx because my simulator won't run something build for 1.13 though.

I don't think there were any file format-level changes in 1.13 and you obviously aren't using any new 1.13 API, so you should be able to just change the pdxversion field in the pdx's pdxinfo file to 11200 and it'll run on a 1.12 simulator.

Seem to work for me. toybox.py v1.1.2 has the change for slashes in the import path on Windows.

Thank you! :pray:t2:

1 Like

I realized that the build I sent didn't include the fix for .. paths requiring the .lua extension, so here's that merged in and I changed pdxversion to 11200:
pdc.exe.zip (224.1 KB)

Are there still any problems affecting toybox.py after this, or can I close the issue? We won't be able to get these fixes into 1.13.0, but we'll have 1.13.1 out as soon as possible after that ships.

  • My current tests seem to indicate that pdc concatenates these relative paths ad-infinitum. I've seen any error message looking like ../../Dir1/Folder/../OtherFolder/../Test. Since there seems to be a 127 character limit this is not going to work in all uses cases.

okay, I know that's still going to be a problem--we don't do any handling of .. in the compiler, we just let the filesystem functions resolve those. But if we want to do proper import tracking, making sure we don't include the file more than once, we'll definitely need to normalize those paths. I don't think that'll be too tricky.

I realized that the build I sent didn't include the fix for .. paths requiring the .lua extension

The new build now works for that too.

we'll definitely need to normalize those paths. I don't think that'll be too tricky.

cool!

I just bumped up against this today - I'm on SDK 1.13.2 and Toybox.py v1.2.1.

Yeah it doesn't seem fixed in 1.13.2 either...

I'm so sorry, y'all. We put all these bug fixes in the queue expecting they'd turn around within a week but since we didn't properly isolate the Catalog changes from the working branch that threw off the whole QA process. We're slowly getting things turning again--1.13.3 which fixes the performance regression introduced in 1.13.0 should be out today or tomorrow :crossed_fingers: then we'll get all the simple fixes like this one into the next update shortly after that.

Again, I know how frustrating this is for everyone (we also keep running into these long since "fixed" bugs) and I apologize. We'll try and make it up to you going forward.

3 Likes

Really appreciate the work you guys are doing though. We know it's not easy and that your resources are limited.

1 Like

This seems to be fixed in 1.13.5... is this true @dave ???? :heart_eyes:

1 Like

Sounds like it is! In 1.13.5 we switched from backslashes to forward slashes when putting paths together, since the POSIX APIs we're using there can handle it. Problem up and disappeared. Anyone who's using backslashes will likely see this issue now, but hopefully that's nobody. I'll file an issue to make a note in Inside Playdate that we use forward slashes on Windows, too.

2 Likes

pdc.exe is failing for Windows paths with back slashes when I use the -m/--main flag after upgrading to 1.13.5. Is it possible that this is related to this change? Here are some examples:

Back slash with relative path fails

> pdc -sdkpath C:\Path\To\PlaydateSDK --main source\main.lua main.pdx
error: writeToFile:main.pdx/source\main.pdz failed

Forward slash with relative path succeeds

> pdc -sdkpath C:\Path\To\PlaydateSDK --main source/main.lua main.pdx

Back slash with absolute path fails

> pdc -sdkpath C:\Path\To\PlaydateSDK --main C:\Path\To\Game\source\main.lua main.pdx
error: main.lua or pdex.bin required

Forward slash with absolute path succeeds

> pdc -sdkpath C:\Path\To\PlaydateSDK --main C:/Path/To/Game/source/main.lua main.pdx