Compiling 1.9.4, rtdist: __path__ empty

[Thaumaturge]

I’m attempting to create a custom build of 1.9.4, including an attempted back-porting of 1.10’s vertex-colour handling. Thus far, the SDK and runtime seem to build successfully (although I’ve neither tested nor installed either yet, I believe). The “rtdist”, however, is giving me trouble.

Specifically, the build is exiting with an “IndexError”, noting that a list index is out of range, in FreezeTool.py, line 633. Here’s the full error:

Traceback (most recent call last):
  File "direct/src/p3d/ppackage.py", line 254, in <module>
    packages = packager.readPackageDef(packageDef, packageNames = packageNames)
  File "/home/thaumaturge/panda3d-release-1.9.x/built_thaumaturge/direct/p3d/Packager.py", line 2811, in readPackageDef
    solo = solo)
  File "/home/thaumaturge/panda3d-release-1.9.x/built_thaumaturge/direct/p3d/Packager.py", line 2915, in beginPackage
    package = self.Package(packageName, self)
  File "/home/thaumaturge/panda3d-release-1.9.x/built_thaumaturge/direct/p3d/Packager.py", line 381, in __init__
    self.freezer = FreezeTool.Freezer(platform = self.packager.platform)
  File "/home/thaumaturge/panda3d-release-1.9.x/built_thaumaturge/direct/showutil/FreezeTool.py", line 633, in __init__
    modulefinder.AddPackagePath(moduleName, path[0])
IndexError: list index out of range

I added some simple “print” statements above the offending line, and it looks like the problem is happening when it attempts to run the above for “pkg_resources._vendor.six.moves.urllib”. The module itself is loaded, but “path” seems to contain an empty list–hence the IndexError.

What’s going wrong here? It’s entirely possible that I’m missing some dependency–I’ve never quite gotten the hang of figuring out which complaints in the build output are relevant, I’m afraid–but which one?

My thanks for any help given!

[rdb]

Is this Python 2 or Python 3? I believe the 1.9 rtdist won’t compile with Python 3.

Any idea why your code is importing this problematic module? Perhaps you can explicitly add it as an excludeModule to proceed?

[Thaumaturge]

@rdb

Is this Python 2 or Python 3? I believe the 1.9 rtdist won’t compile with Python 3.

It looks as though I’m building with Python 2.7.12

@rdb

Any idea why your code is importing this problematic module? Perhaps you can explicitly add it as an excludeModule to proceed?

I have no idea why it’s there, I’m afraid–it’s not something that I’ve made a point of adding, as far as I’m aware.

As to removing it, what is it’s purpose in Panda3D–what might I be losing if I remove it?

Oh, one thing that I perhaps forgot to mention: I think that I recall reading somewhere that the “path” variable may be empty in the case of built-in methods–as I think that this was indicated to be when I had Python “print” the module. I’m not sure of whether this applies to Python 2.7 or 3, however.

[Thaumaturge]

bump Any further thoughts, here? I’d really like to get a working rtdist built, but I’m hesitant to attempt to exclude the above-mentioned module without knowing whether or not it’s important… ^^;

[Thaumaturge]

one more bump

Is “pkg_resources._vendor.six.moves.urllib” important to Panda3D? What functionality might I lose if I exclude it? It looks like something related to networking–given that my game is offline-only, is that something safe to break?

RDB, you mentioned adding it as an “excludeModule”–looking around, do I guess correctly that this is done by adding a line calling that function to “panda3d.pdef”, and presumably the one in “direct/src/p3d”? If not, how is it done, please? (I don’t see a mention of it in the manual, although I may have missed it.)

[rdb]

I believe it’s not part of Panda, so it must be pulled in by a module imported in your code. By placing an excludeModule command in the pdef file that builds your .p3d file. If you use the (less configurable) packp3d command, then there may be an equivalent option, or maybe not, I’m not sure.

Do first check whether it might actually be part of the built rtdist by grepping for the module name in the .import.xml files in the stage directory. If so, you need to check which package it’s part of and place an excludeModule in the appropriate .pdef file in direct/src/p3d.

I highly doubt that it’s important. It looks like it is part of a vendored module inside a vendored module, and it may not even be used by any code. I suppose you’ll find out when you try it.

[Thaumaturge]

@rdb

I believe it’s not part of Panda, so it must be pulled in by a module imported in your code.

But I’m just building the Panda3D source code, with the back-ported vertex-colour changes from 1.10, I’m pretty sure. o_0

(I’m building with makepanda.py, in case that’s relevant.)

@rdb

Do first check whether it might actually be part of the built rtdist by grepping for the module name in the .import.xml files in the stage directory. If so, you need to check which package it’s part of and place an excludeModule in the appropriate .pdef file in direct/src/p3d.

Hmm… Where is that directory? Having just built the SDK and runtime, I don’t see it…

Come to that, looking again at my “runtime” and “rtdist” makepanda commands, what directory should be specified for their “–host” parameters? I currently have them as “/built_thaumaturge/stage”, but I see that the runtime, when built, produces the directory “built_thaumaturge_rt” (although said directory doesn’t have a subdirectory named “stage”). Should the “_rt” be appended to the runtime’s host-parameter, or to the “rtdist”'s, perhaps? (I don’t see anything in the documentation that covers this.)

@rdb

I highly doubt that it’s important. It looks like it is part of a vendored module inside a vendored module, and it may not even be used by any code. I suppose you’ll find out when you try it.

Hah, fair enough!

[rdb]

Oops, sorry, I thought you were encountering it when building your application code. Indeed, try and find out which package it’s in, then put the excludeModule into the definitions in the panda3d.pdef or thirdparty.pdef file. Or just put it in all of them if you don’t know. You should be able to see in the build log which package it was building last, though, right?

The --host should be a URL pointing to the directory containing the contents.xml, which will be “built_thaumaturge/stage” by default. Note that this should be a file:/// URL.

The _rt is for appended for the runtime build. You don’t strictly need to build your own runtime, other than to get a version of the runtime that defaults to the host you specify, but the host location of the “panda3d” package is determined from the package requirement specification in the .p3d file.

On a sidenote, the new deploy-ng system that is intended to replace the rtdist path is becoming increasingly stable, so at some point it may become less effort to get it to work the way you want than the old rtdist system.

[Thaumaturge]

@rdb

Oops, sorry, I thought you were encountering it when building your application code.

Fair enough! I thought that that might have been the case, I believe.

@rdb

Indeed, try and find out which package it’s in, then put the excludeModule into the definitions in the panda3d.pdef or thirdparty.pdef file. Or just put it in all of them if you don’t know. You should be able to see in the build log which package it was building last, though, right?

Ah, you’re right. I just ran the “rtdist” build again, and now that it’s failed, I see that the message just before the traceback reads as follows:

Generating twisted/thaumaturge_1.9/linux_amd64/twisted.thaumaturge_1.9.linux_amd64.mf

Looking that up, I see that it’s a networking framework–for which the inclusion of something called “urllib” makes sense, I daresay.

I recall that Panda3D uses a client-host approach to distributing games–even self-contained installers retrieve elements from one or more sources while building. As a result, this might not be safe to omit.

On the other hand, the issue seems to only show up in the “rtdist” build–the “runtime” and SDK builds seem unaffected. As such, perhaps it’s only used for game-level networking, in which case perhaps it is safe to omit, given that I’m making an offline-only single-player game?

@rdb

The --host should be a URL pointing to the directory containing the contents.xml, which will be “built_thaumaturge/stage” by default. Note that this should be a file:/// URL.

I take it that the “stage” directory is created as part of the build process, then? And should the same directory (as opposed to the “_rt” version) be used when building the runtime?

@rdb

The _rt is for appended for the runtime build. You don’t strictly need to build your own runtime, other than to get a version of the runtime that defaults to the host you specify, but the host location of the “panda3d” package is determined from the package requirement specification in the .p3d file.

Hum… So what determines the host referenced in the .p3d file? I don’t see an option for that in packp3d…

@rdb

On a sidenote, the new deploy-ng system that is intended to replace the rtdist path is becoming increasingly stable, so at some point it may become less effort to get it to work the way you want than the old rtdist system.

That does sound good! I’m a little shy of risking a new and untried deployment system as things stand (especially since I do want this game to be commercial, eventually), but if deploy-ng proves sufficiently stable and effective I may well end up switching over to it.

[rdb]

Yes, twisted isn’t used by anything in Panda. You can comment out the definition in thirdparty.pdef. In fact you can remove anything in thirdparty.pdef that you don’t need.

The “stage” directory contains what you’d upload to a package host such as runtime.panda3d.org although if you only ever use it locally that isn’t very relevant.

@Thaumaturge

I take it that the “stage” directory is created as part of the build process, then? And should the same directory (as opposed to the “_rt” version) be used when building the runtime?

Assuming I understand your question correctly, yes, if you want your runtime to pull its “images” and “coreapi” packages from the rtdist you have built, you need to specify the built_thaumaturge/stage directory via a file:// url as host.
There is no stage directory in the built_thaumaturge_rt directory.

@Thaumaturge

Hum… So what determines the host referenced in the .p3d file? I don’t see an option for that in packp3d…

packp3d is built as part of an rtdist, which was built with a --host flag. So packp3d itself has a dependency on a ‘panda3d’ package from some host or another, which I believe is used as default for other package references.

(I could be wrong and it might actually be using the host that the runtime was built with, I’m not entirely sure.)

Either way, when specifying a dependency, you can explicitly specify a host URL and version with something like -r package,cmu_1.9,https://runtime.panda3d.org or, if you are using the more flexible .pdef specification to create your p3d files, via extra arguments to the require() command.

[Thaumaturge]

@rdb

Yes, twisted isn’t used by anything in Panda. You can comment out the definition in thirdparty.pdef. In fact you can remove anything in thirdparty.pdef that you don’t need.

Ah excellent–that’s good to know!

I’m currently rebuilding, having edited “thirdparty.pdef”. It’s taking some time, so I’ll report back once it’s done, I intend!

@rdb

The “stage” directory contains what you’d upload to a package host such as runtime.panda3d.org although if you only ever use it locally that isn’t very relevant.

Ah, fair enough.

@rdb

Assuming I understand your question correctly, yes, if you want your runtime to pull its “images” and “coreapi” packages from the rtdist you have built, you need to specify the built_thaumaturge/stage directory via a file:// url as host.
There is no stage directory in the built_thaumaturge_rt directory.

Fair enough, and noted!

@rdb

packp3d is built as part of an rtdist, which was built with a --host flag. So packp3d itself has a dependency on a ‘panda3d’ package from some host or another, which I believe is used as default for other package references.
…

Ah, fair enough. That seems straightforward, then!

Thank you for all of your help!

[Thaumaturge]

Update: The builds (both SDK and rtdist) seem to have completed successfully! I haven’t yet installed or tested either, but this particular issue seems solved. Thank you again for your help!

[Thaumaturge]

Okay, one more question, if I may: What version of GCC is usually used to build Panda3D (presuming that GCC is used at all)? I’ve hit a problem when attempting to create a distributable build of a game, and some quick Googling suggests that it may be related to a bug in GCC 5…

Specifically, while packp3d seems to work, pdeploy (the version found in “built_thaumaturge/stage”, I believe), exits with the following error:

Couldn't load /home/thaumaturge/.panda3d/coreapi/linux_amd64/p3d_plugin.so: /home/thaumaturge/.panda3d/coreapi/linux_amd64/p3d_plugin.so: undefined symbol: __cpu_model
Unable to launch core API in /home/thaumaturge/.panda3d/coreapi/linux_amd64/p3d_plugin.so
Unable to load Panda3D plugin.

[rdb]

Yeah, this is an issue in gcc. I think on the master branch this was eventually worked around in this commit:
github.com/panda3d/panda3d/comm … dc5adef324

However there is another copy of stb_image.h in direct/src/plugin which may also need to be patched.

Rather than applying the whole patch, the easiest way for you to fix it might be to look for the stbi__sse2_available function in both copies of stb_image.h, and removing its contents so that it only returns 1, eg.

static int stbi__sse2_available() {
  return 1;
}

and also changing this line:

#if defined(__GNUC__) && (defined(STBI__X86_TARGET) || defined(STBI__X64_TARGET)) && !defined(__SSE2__) && !defined(STBI_NO_SIMD)

…to instead read this:

#if defined(__GNUC__) && defined(STBI__X86_TARGET) && !defined(__SSE2__) && !defined(STBI_NO_SIMD)

As for compiler versions, for building Panda 1.10 we require at least clang 3 or gcc 4.8, though any version after that should work.

[Thaumaturge]

Does the issue exist in gcc 4.8, do you know? If not, perhaps it would be easier to simply downgrade to that…

[rdb]

I have no idea. I think it exists in GCC 5 and maybe also in 4.9, but I don’t know about 4.8.

Switching from gcc to clang would also work.

[Thaumaturge]

Ah, clang looks like a good option, I think. I’ve installed it and set the relevant environment variables, I believe, and intend to report back once I’ve rebuilt the engine and tried to create a distributable with it.