I’ve been tracking down a performance problem that I seem to have on device, the main symptom of which is that my loadBitmap calls take 20+ms even for 300 byte files. Other I/O operations seem to be similarly surprisingly slow, including AudioSample load, stat and open(kFileRead|kFileReadData) but interestingly not open(kFileRead), read, seek, etc.
I created a small reproduction using C API examples and it seems that these operations increase in time quite dramatically when files are added to the data directory. For example, with some sample files I get approximately:
0 data files:
stat: 3ms
loadBitmap (300 byte pdi): 7ms
read file via open/seek/tell/read (3kb json file): 2ms
AudioSample load (500kb pda): 40ms
50 data files:
stat: 4ms
loadBitmap: 10ms
read file: 2ms
AudioSample load: 42ms
100 data files:
stat: 5ms
loadBitmap: 12ms
read file: 2ms
AudioSample load: 45ms
200 data files:
stat: 9ms
loadBitmap: 20ms
read file: 2ms
AudioSample load: 52ms
Note that these operations are all on files in the game’s pdx, not in the data directory which is growing in size. Also interesting is that these do not return to baseline after deleting the files. They do return to baseline if I delete the whole data directory (by mounting to a computer).
So, guessing a little, I suspect what’s happening is:
stat checks both pdx and data folder for the given path.
stat’s performance varies a lot based on the number of directory entries.
loadBitmap and AudioSample load call stat or something equivalent internally (twice?)
the PD FS is FAT32 which does not clear out directory entries immediately when deleting a file.
So my main ask is: would it be possible to stop loadBitmap from looking at the data directory at all if the file is openable in the pdx? (And AudioSample load, too, but that’s relatively easy to work around.)
I will also stop writing so many files to the data directory but this seems like perhaps a permanent problem for any game that has a lot of files there. Thank you!
I think their intent is the data folder acts like a patch over the game folder - so if you have blah.pdi in both places, it'll load the data one in preference to the one from the pdx - so if a game saves over its own files, the game should load back the modified version
One workaround might be to put your data files into a subdirectory, if they're not intended to shadow the pdx files
Ah, I was confused why the data directory was involved at all, but your mental model makes it all make sense! And making subdirectories seems like a good path.
If that’s the desired effect, I’m not sure what to suggest other than exposing an options argument so people can load from the pdx only if desired? (Or attacking the underlying performance but I assume that’s much harder.)
There definatly seems to be a performance impact of some sort. There are users reaching the 10 second watchdog limit on certain games and at least one user when having renamed the Data folder, creating a new Data folder and copying the files from original renamed data folder back said the crashing in certain games stopped.
Also as a way to test on my rev A playdate i renamed my data folder to databackup created a new data folder then created 2500 subdir in it named a_1 till a_2500 and rebooted the playdate, if i started some games they crash hitting the watchdog timeout. What i do not understand is how those 2500 unrelated subdirectories affect the 2501th directory created for a game if the games themselves normally can only access /Data/gamesdatafolder or /Data/Shared it’s almost as if the the whole /Data is being looped at somepoint somehow potentially multiple times and this seems to match with what you are seeing. I’m not sure how the system checks for existance of the data folder of the game or sub entries of it but it feels weird
From what I understand, FAT32 stores the directory table as a linear list, so it very well might have to loop sequentially over all of them each time it needs to find a particular folder
wouldn’t delete said directories keep same slow speed then ? I’m not sure anymore (can’t remember exactly) but i thought some of the games loaded fine after deleting the directories so it feels to me that there is more happening than just traversing the directory table list. But that’s just a guess. I would have expected adding lets say 1000 directories then adding a games folder as 1001th entry to remain equally slow as when i deleted the 1000 directories and said games folder still being 1001th entry with 1000 deleted items before it which did not seem to be the case when i tested. Thats why i thought the system functions either when creating a new games directory or through certain system calls do more than just looking in the actual game’s data folder as well like there 2 things at once happening the fat32 directory table traversal but also just plain code from the firmware or function from the api doing things in the data folder