Pdef files: confusion, issues (and bugs?)

Scripting Issues
1

I decided to try and use ppackage.

I took the example pdef file:

class mypackage(package):
    file(Filename('neededfile.dll'))
    module('my.python.module')
    dir(Filename('/c/my/root_dir'))

(From panda3d.org/manual/index.php/The_pdef_syntax )

and tried it. It throws an exception:

Well, thats kinda lame. So off I went to debug, starting with looking at “direct/src/p3d/Packager.py” as recommended. That path does not exist though. Inside Panda3D/lib I do have direct/p3d/Packager.py so I looked in there. Well over 3000 lines of python is not the best file format specification, but I’ll take what I can get.

Filename is used in there, but not defined or imported as far as I can tell. I think its coming in through a * import, which stylistically shouldn’t be used for this reason. Oh well.

Anyway, looking through Packager.py seems to suggest that I can use Filename objects, or strings, so I just go for strings. For organization, I tried this:

def baseFiles():
    dir('api')
    dir('settings')
    dir('core')
    file('main.py')
    file('events.txt')

class mypackage(package):
    baseFiles()

which runs fine, but I get “:Packager: No files added to mypackage” and I imagine an empty package. Instead I need to put the code directly in mypackage, which seems to work, but I don’t see how it should be different.

Now I have:

from panda3d.core import Filename

class mypackage(p3d):
    require('wx','morepy')
    dir('api')
    dir('settings')
    dir('core')
    dir('data')
    mainModule('main',filename=Filename('main.py'))
    file('events.txt')

Now for the really creepy thing: everytime I run ppackage with this pdef file, it deletes my main.py source file. Yes, ppackage deletes my main module source file (and I think/hope nothing else). I could add some code to my build script to copy and restore my main.py everytime, but that should not be needed. So, how to I make ppackage not delete my main module? (This really seems like a bug, and a really nasty one too)

Another question: is mainModule’s filename argument suppose to be able to take a string? Apparently it can’t, but I expected it to work. Also, in what case can you use mainModule without an explicit filename? I can’t make it work that way.

2

It is true that you have to import (most of) the symbols you are going to use within the pdef file, just as you would have to into a Python file. If the pdef manual page were more complete, it would probably point out that Filename is defined in panda3d.core, but I guess you figured that out eventually.

Again, if the manual page were more complete, it would tell you precisely what you can put in a pdef file, instead of expecting you to dig for it. My apologies. As it is, it does make quite a few shortcuts, including neglecting to point out the possible difference between direct/p3d/Packager.py and direct/src/p3d/Packager.py (this difference depends on whether you have a source installation or a prepackaged installation of Panda).

ppackage relies on some underhanded Python tricks to interpret the things that look like function calls into structures that get encoded within the class object. But they aren’t actually function calls, so you can’t nest them in another function like this. Granted, sneaky Python tricks aren’t always friendly to newcomers, so this might have been a bad approach; but the goal was to create a file format to fully define a Panda3D package with (a) a minimum of unneeded syntax and (b) utilization of the existing Python interpreter. Since the Python interpreter wasn’t designed with this kind of thing in mind, satisfying (a) required some shenanigans.

Really? That’s really weird. I have no idea why it would do that. Are you perhaps running ppackage from within your source directory itself? Are you specifying a -i parameter to write the output to some other directory, or are you letting it write the output right into the current directory? If you’re not specifying -i, maybe somehow it’s getting confused by the recursion that this implies? Even still, I don’t know how it could be deleting files. It should be creating files, not deleting them.

Not the way it’s written. A string is a little ambiguous–is it os-formatted or Panda-formatted–but it could be made to take a string. Still, this is normally a rarely-used parameter. It should normally be unnecessary, if the Python module could be imported directly, for instance with an “import main” command. If that doesn’t work for you, something’s indeed wrong.

David

3

I’m running it from this code inside build.py which is in my source directory:

def release():
    makeP3ds("../Release",'release.pdef')

def makeP3ds(destDir,pdef):
    args=["ppackage","-i",destDir,pdef]
    p = subprocess.Popen(args)

from this build.py file, I can import main successfully.

If I leave out the filename specification this happens:

If I run it again with the filename=Filename(‘main.py’), I get this, and my main.py file gets deleted (but included in the resulting p3d)

Edit: I tried deleting all the code in main.py and it did not change the deletion issue.

4

Ah, I see. There is indeed a typo in Packager.py, such that if you specify a filename parameter to mainModule(), it will delete the filename when it has imported. (The typo is that it do_mainModule() passes deleteTemp = True to addFile(), which is intended for deleting a temporary file after it has been read. This must be a copy-paste error.)

I can fix the bug, my apologies. Hmm, but the parameter should still be unneeded. Perhaps you can omit it if you first explicitly define the “main” module with a call to module(‘main’)–I believe the mainModule() call is intended to declare which particular module will be launched initially, but it’s intended to have been already imported, either via a previous call to module() or a previous call to dir().

David

5

It seems like the right thing is “file(‘main.py’)”. This makes mainModule work. I also tried module(‘main’), but it didn’t help.

Thanks!

By the way, the current pdef format could work just fine as a regular python file. I guess it might require an import to get the package and p3d classes available to subclass (you could splice this line onto the beginning of the file, or add them to the namespace with an import hook I think.). The packages would be available (I believe in order defined) in package.subclasses() and p3d.subclasses(). I think the best thing would be to temporarily stick package and p3d into builtins. If using subclasses and running multiple pdef files, you would need to redefine the package and p3d classes for each file you do (but thats easy, just define the classes in the method that loads the pdef file)

6

Hopefully this isn’t getting too horribly long and annoying. I guess it can be considered my blog on usability issues with ppackage.

New problem (I did manage to work around this, but its still a bug): direct.wxwidgets is not included (not even in wx package)

To fix this for packp3d, I resorted to “-p /Developer/Panda3D/lib”
(As my p3d file does not yet work, I’m not sure if this really fixed the problem)

For my pdef file I tried “module(‘direct.wxwidgets’)”
I get: Unknown module direct.wxwidgets
I’ve never see module actually work. I finally got mainModule to work after including the files directly, so I tried:

    dir('/Developer/Panda3D/lib/direct/wxwidgets',newDir='direct/wxwidgets')
    file('/Developer/Panda3D/lib/direct/__init__.py',newDir='direct')

I get:

I feel like I’ve been here before, stuck with a modulefinder keyError. Last time it was with my own modules that used import hooks to load from outside packages and thus their parents couldn’t be found. Thats a bug I think. It should probably raise a warning, but not crash.

Solution:

file('/Developer/Panda3D/lib/direct/__init__.py',newDir='direct')
    file('/Developer/Panda3D/lib/direct/wxwidgets/__init__.pyc',newDir='direct/wxwidgets',literal=True)
    file('/Developer/Panda3D/lib/direct/wxwidgets/WxAppShell.pyc',newDir='direct/wxwidgets',literal=True)

Thats a mess, but it seems to work on windows. On mac, the wx package is broken, so its not going to work anyway.

Also, whats with rdb’s absolute paths showing up in the VFS? Are any of my full paths ending up in there?

I’m trying to include my *.so files and my .pyd files.
For each dir with a .so file, I can do something like this:
dir(‘api’,newDir=‘api’) # Get the py files
file('api/model/.so’,newDir=‘api/model’)
but I can’t see how to automatically include all of them as “-e so” does for packp3d.

I thought it would rename the .so to no extension on mac based of the comments I read, but it does not rename it.

If I try to include my pyd file for Windows, via this:
file('api/model/.pyd’,newDir=‘api/model’)
I get this:
:stuck_out_tongue:ackager(warning): No such file: api/model/.dylib
and no pyd file in the package.
I have to do this:
file(‘api/model/_model.pyd’,newDir=‘api/model’,literal=True)
This does work. I can import the python extension module on Windows.
I’ll have to do that for every single extension module. I managed to make an automated build script that compiles all the extension modules. I’d really rather not deal with them individually anywhere.

Currently I have this:

from panda3d.core import Filename

class mypackage(p3d):
    # using wx package makes this windows only until panda3d's wx package is fixed for other OSs
    require('wx','morepy')
    
    dir('api',newDir='api')
    
    # get mac extension modules, one dir at a time
    file('api/model/*.so',newDir='api/model')
    
    # get windows extension modules, one module at a time
    file('api/model/_model.pyd',newDir='api/model',literal=True)
    
    dir('settings',newDir='settings')
    dir('core',newDir='core')
    dir('data',newDir='data')
    file('main.py')
    mainModule('main')
    file('events.txt')
    
    # fix missing wxwidgets module (bug in panda3d's wx package)
    file('/Developer/Panda3D/lib/direct/__init__.py',newDir='direct')
    file('/Developer/Panda3D/lib/direct/wxwidgets/__init__.pyc',newDir='direct/wxwidgets',literal=True)
    file('/Developer/Panda3D/lib/direct/wxwidgets/WxAppShell.pyc',newDir='direct/wxwidgets',literal=True)

This makes the file “mypackage.osx_i386.p3d” which only works on Windows (Due to the broken wx package). I have a few file path related things to debug to get it all working on windows, but at least the wx modules can import ok.

I feel like there is a whole lot of magic going on which I can’t control. As I can’t write my own methods and such to walk the directory in the pdef file. I’m thinking writing code to generate the pdef file as it seems like the easiest approach. Its trivial to walk the directory with os.walk and find all the files I want.

While it could come out pretty tidy and easy to use that way, I feel kinda silly basically wrapping the entire ppackage system with a different build script setup hooked up to pdef code generation.

7

It does seem like something fundamental isn’t working here.

First, we should figure out why your module command is failing. This uses Python’s own modulefinder to look for the named modules along sys.path. It works perfectly for me. Is it possible that sys.path is not set up correctly in your case to find the modules you are naming?

Also, note that the ppackage command, and the pdef syntax, is just intended as a convenience for developers who wish to build packages from the command line. If you are writing a Python program, you can use the Packager interface directly from Python; there is no need to use pdef at all. See this thread for a little bit more about this.

The omission of direct.wxwidgets has been previously reported and will be corrected in the future.

Absolute paths are burned into the compiled executable by MSVS, and into the Python binaries by the Python interpreter, at the time the code is generated. That’s unfortunate, and it causes confusion, but usually not any real harm. And it appears to be hard to avoid. (I think there’s a bit more we can do to remove the absolute paths from the Python side, but I don’t know what to do about the paths that the compiler puts in.)

To simulate “-e so” you only need to do:

packager.binaryExtensions.append('so')

You can put this at the top of your pdef file, or if you are using the Packager interface directly, you have your own packager object.

I don’t understand why you would want this kind of renaming. The .so extension is important.

I think all of these problems are stemming from the use of the file() command instead of the module() command. This is not the intended way to load Python code, either .py files or .pyd files. The module() command is supposed to work for both kinds of files, and it doesn’t have these weird problems. Really, the file() command is intended to be used to load non-code files, like models and textures.

Are you really running ppackage on Mac OSX in order to build a Windows p3d file? That’s also unexpected. In general, a p3d file is either platform-independent (it contains only pure Python code, no extension modules), in which case it can be built and run on any platform, or it is platform-dependent (it contains some extension modules), in which case it can be built and run only on the target platform. The system is not designed to support cross-building as you appear to be doing; nor is it designed for multi-platform p3d files, p3d files that contain multiple different versions of the same extension modules for different platforms. If you need to achieve multi-platform support like this, you should put the platform-dependent bits into a “package”, and reference it from the p3d file.

But if all of this platform-dependent stuff is only to support wxPython properly, then maybe it’s just on us to fix the “-r wx” package? Or are you using other platform-dependent modules as well?

David

8

I have a python extension module I wrote and compiled (_model.so and _model.pyd). The pyd is under 8kb, and the so is also quite small so I figured I could just put both in the p3d for now. About removing the .so, I though that was odd, but I guess I misread the comments in Packager.py.

So I’m trying to make either multiple platform versions of package, or one for all my platforms (I really don’t acre which). It seems to think it’s suppose to be mac only because of the .so, but it works on Windows. I found how to tell it to be platform_specific or not now, so I guess that is solved.

So is it really the case that I’m not supposed to try and make anything with platform specific stuff in it on another platform? Do I need to make separate package definitions for each platform? (differing only by the file extensions for the extension modules)

I have (several) directories which are python packages containing several modules and inner packages. I used lines like dir(‘core’,newDir=‘core’) for these packages (which may contain extension modules)

Adding packager.binaryExtensions.append(‘so’) to my pdef file does not seem to cause dir to pickup my .so file. Maybe it would make module work on them, but module doesn’t work.

I did:
import os
print os.path.abspath(os.curdir)
in my pdef file and it does print my source directory, however I can’t import any of the modules or packages in there:

This seems quite odd. I’m not sure what kind of setup the pdef file is run under, but I can even do this:

print open('main.py').read()

which works, but I can’t import main.

Doing module(‘main’) also fails with “Unknown module: main” (Printed twice)

How am I supposed to use module? Can I just use module on one of my big nested packages root folder, and have it get all the py files and compiled extension modules if I set things up right? (well, I tried that and it didn’t work, but is that whats supposed to work?)

When I just use dir for all my packages, I get the .pyo files from my core package in 2 places, at the root and in their proper place in the core package. I guess this could be caused by the fact that inside core, I often use local imports. Seems like another bug to me though, and I don’t see a workaround other than changing all my code.

I’ve actually got something that fully works (on windows) now. Its a bit messy, but not too bad. I’d still like to figure out whats up with module not working though.

9

I figured out why module does not work. Its an issue with sys.path. There is no ‘’ entry on it. If I do:

import sys
sys.path.insert(0,'')

I can then import local modules (not that I need to), and module(‘name’) works.
I used to have to move things into the /mf path with file and dir first, then use module or mainModule

Hopefully by using module properly and messing with sys.path I can fix my other issues.

Mac OSX 10.5.8, Python 2.6

I doubt its related, but on this computer (but not my 10.6 mac), punzip is busted: I tried to un pzip my multifile and got this:

Craig:0.0 Craig$ punzip
dyld: Library not loaded: libp3dtool.1.7.0.dylib
Referenced from: /Developer/Tools/Panda3D/punzip
Reason: image not found
Trace/BPT trap

Edit:
Adding this lets me avoid my ugly hack to get direct.wxwidgets
sys.path.append(‘/Developer/Panda3D/lib’)
I didn’t realize so many of my issues could be solved using sys.path. Looks like using module solved my duplicate module issues too.
When things that try to be smart work, they are pretty cool, but oh man when they start breaking, it makes a mess!

I still can’t make it automatically include my so or pyd files. This seems to do nothing:

packager.binaryExtensions.append('so')
packager.binaryExtensions.append('pyd')

Also, I tried making multiple packages. I noticed that my main package (the p3d) seems to have nice short paths in my traceback, but my additional ones have massive paths (the path to my source directory). Setting start_dir seems to fix this, but I can’t get it to mount at anything other than “”. start_dir = ‘/mf’ does not seem to work.

Edit Again:
This is just magical, check out this trace back:

No mater what I put on line 13 of my init.py file in core, it runs what used to be there, import ui, but prints what is there now in the traceback. Its the magic semi out of date p3d file! I’m having trouble getting the auto updating packages to work. For now I’ll manually empty the cache to avoid such oddities, though I’m still not sure how that particular thing happened.