Generalizing pdeploy

General Discussion
1

Hi all,

For some time now, I’ve been looking at ways to pack a python application inside a single binary, in a cross-platform way. Seeing how beautifully Panda3D handles the same problem with pdeploy, that got me thinking.

Maybe the same mechanism could be applied in a more general way?

In any case, I was wondering if there was any documentation on the way pdeploy works internally. Same thing for p3d files (which are where the real magic happens, if I’m not mistaken, pdeploy simply embeds a p3d into a binary wrapper for each platform, or something along the lines)?

Maybe someone can point me to where in the source I should be looking?

Thanks in advance!

2

Sounds like you’re looking for something like PyInstaller or cx-freeze.

3

Actually, PyInstaller is what I’m using right now. That type of tools have a big drawback though : they demand that you have access to all platforms each time you want to make a build.

I would love to try and come up with a system that would let me make some kind of binary wrapper that I could compile once on each platform, then use to bundle my Python in a native executable for all platforms at once from a single platform. So basically what pdeploy does.

4

Basically, here’s how pdeploy works internally:

On runtime.panda3d.org where all of the plugin packages are hosted, there is also a “p3dembed” package, which has a “p3dembed” executable for every platform out there. This is basically a modified version of the Panda3D runtime “panda3d” executable, except it works a little differently:

  • It defines a const volatile 32-bits unsigned integer, that is initially set to 0xFF3D3D00.
  • It reads this variable, assumes that it is an offset in the file pointing to a block of metadata followed by the .p3d data, and starts running the .p3d at that location in the file.

This p3dembed executable is useless in its current form, since the variable contains a meaningless marker, 0xFF3D3D00. Running it will make it crash, since no .p3d data exists at that offset (actually, it checks for the marker and will cleanly exit with a friendly error, but that’s just an implementation detail.)

Now, when you run pdeploy, it downloads all of these versions of p3dembed. It then tacks appropriate metadata followed by the .p3d data onto the end of the executable. Then, it scans the executable code for the 0xFF3D3D00 marker, and replaces it with the actual offset of the .p3d data (which in this case, is equal to the size of the original p3dembed file), so that next time you run the executable, the variable will contain the actual offset instead of the marker.

So when you run this new executable, it’ll know exactly the offset of the .p3d data in the executable file, and it will start running the .p3d file at that offset.

For more information, you can consult the source code at direct/src/plugin_standalone/ (for p3dembed) or direct/src/p3d/DeploymentTools.py (for pdeploy). Or feel free to ask me more questions about it. :slight_smile:

It shouldn’t be difficult to do the same for raw Python. Since Python bytecode is platform-independent, all you need to make is a wrapper executable that runs (using the Python C API) whatever archive of bytecode is tacked onto the end of the executable, statically link it with the CPython ABI, and compile it for every platform.
I bet that the existing tools like py2exe and pyinstaller use a similar approach. I think that py2exe uses a zip file with bytecode files attached. (Python can implicitly read py/pyc/pyo from zip files, I believe).
If you need any help, let me know.

5

Awesome! Thank you for the detailed information, rdb. I will soon start a project on GitHub and try to use a similar system for pure Python apps.

Each time I post here, I’m more amazed at the supportive community around Panda3D.

If I have more precise questions, I’ll return to this thread. Thanks again for all the info!

6

If you do, be sure to tell me - I could sure use a system like this. :slight_smile:

7

Just as an update, I’ve started to look into this, and it seems like it is indeed what PyInstaller (at least) does. Apparently, PyInstaller cannot cross-package, even though the binary ‘stubs’ at the end of which data is appended are pre-compiled and included in the download (at least the ones for macOSX, Linux and Windows). If I’m not mistaken, this is because their dependency-checking includes the standard library stuff to minimise the end of the resulting executables.

I suppose that, even if that makes files a bit bigger, you could make ‘more generic’ bootloaders and simply have to provide your dependencies’ python bytecode plus pre-compiled dynamic libraries.

It’s a little bit more involved than I first thought, but I should probably be able to get the ball rolling by starting with pure python applications, without external dependencies, then work from there. I’ll update this thread as soon as I have news.

(And for the curious, I have 2 main test targets in mind : applications using pyglet, like our first game, and applications using libtcod, like Ultima Ratio Regum. If I manage to get cross-‘compiling’ working for these two targets going, I’ll consider the project a success.)

[Edit] : I created the repository on GitHub, if you would prefer to follow the development as soon as it starts.