ppackage issue with importing a .so file (TOUGH ONE)

Scripting Issues
21

This is the workaround for the ldd problem. We use ldd to automatically determine the complete set of .so files that should be included. Because ldd is not running, that determination isn’t being made, so some .so files are being incorrectly omitted. That’s the nature of the problem. The workaround is to determine by hand the set of needed .so files, and include them explicitly.

There is no new ppackage. There cannot be, until we update the official release. This is because ppackage is tied to the version of the Panda3D runtime that is provided for download. So you can’t have an updated ppackage without also having an updated Panda3D runtime, and vice-versa.

In this case, the file you need to include is libxml.so.2, since that’s the filename that is trying to load at runtime. But I was assuming you need a different one than the one in /usr/lib, since that’s where the error is coming from. Do you have more than one on your system? It might be that the error is coming from a different .so than libxml, for instance libz. Try running your Python program with strace interactively, to see precisely which so’s are being loaded when it runs normally.

Edit: or, another idea, use your /bin/sh hack to build a p3d file correctly, then look at the build log to see which .so files it selected.

David

22

Here’s the full strace of the app:
noop.pl/~deetah/pynk-photos.strace.bz2

And just the filelist of successfully opened files extracted via such a command:

> strace -f python photos.py 2>&1 | grep open | grep -v ENOENT | ruby -e 'files = STDIN.read ; files.split("open(\"").each { |x| puts x.split("\"")[0] } ' | sort | uniq > files.txt

noop.pl/~deetah/pynk-photos.files.bz2

I tried including /lib/libz.so.1, it didn’t change anything. Some debug data (first command just for the readibility purposes):

> readelf -Ws /usr/lib/libxml2.so.2 | head -n 3 | tail -n 1
   Num:    Value  Size Type    Bind   Vis      Ndx Name
> readelf -Ws /lib/libz.so.1 | grep gzopen64
    61: 00003f30    32 FUNC    GLOBAL DEFAULT   13 gzopen64
> readelf -Ws /usr/lib/libxml2.so.2 | grep gzopen64
    11: 00000000     0 FUNC    GLOBAL DEFAULT  UND gzopen64
23

Ah, btw, the ldd hack didn’t make the package work anyway :stuck_out_tongue:

24

Ah, hmm. So, is it possible that’s it’s pulling in some incorrect version of libz.so, one that lacks the gzopen64 definition? I know you tried explicitly including one that does, but maybe it’s somehow finding a different one anyway? Is there another libz.so on your system that lacks this definition? Do you see an error message on output that indicates it is selecting a particular libz.so arbitrarily?

David

25

Looks like I do. Take a look here:


> grep '\.so' strace.log | grep open | grep -v ENOENT | egrep 'libxml|libz'                                           
open("/lib/libz.so.1", O_RDONLY)        = 3
open("/home/deetah/.panda3d/hosts/runtime.panda3d.org_96a0ec538744def6/panda3d/cmu_1.7/libz.so.1", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY|O_CLOEXEC) = -1 ENOTDIR (Not a directory)
[pid  4594] open("/home/deetah/.panda3d/hosts/runtime.panda3d.org_96a0ec538744def6/morepy/cmu_1.7/libz.so.1", O_RDONLY <unfinished ...>
[pid  4594] open("/home/deetah/.panda3d/hosts/runtime.panda3d.org_96a0ec538744def6/panda3d/cmu_1.7/libz.so.1", O_RDONLY <unfinished ...>
[pid  4594] open("/usr/lib/libxml2.so.2", O_RDONLY) = 13
[pid  4594] write(2, "/usr/lib/libxml2.so.2: undefined"..., 49/usr/lib/libxml2.so.2: undefined symbol: gzopen64) = 49
> readelf -Ws /home/deetah/.panda3d/hosts/runtime.panda3d.org_96a0ec538744def6/panda3d/cmu_1.7/libz.so.1 | grep gzopen  
    22: 0000372b    20 FUNC    GLOBAL DEFAULT   10 gzopen
> readelf -Ws /lib/libz.so.1.2.3.3 | grep gzopen                                                                      
    51: 00003f50    32 FUNC    GLOBAL DEFAULT   13 gzopen
    61: 00003f30    32 FUNC    GLOBAL DEFAULT   13 gzopen64
26

So the libz.so that’s getting picked up by default lacks this symbol. I wonder where it’s coming from?

When you included /lib/libz.so explicitly, did it not replace the libz.so that was being picked up automatically? Or did they both get included somehow? Check out the contents of the resulting p3d file (or the install directory) to see how many libz.so’s you end up with in different directories, and to see which of them are correct.

David

27

The libz file was there before, in the directory from the strace log. The one in the archive is the correct one.

28

Isn’t the problem just that I packaged a too-old version of libz in the rtdist build?

29

Oh, I see. Right, that’s the libz.so that’s being provided as part of morepy. So it appears we have a fundamental incompatibility between cmu’s version of morepy, and your libxml.so.

Though I’m a little surprised that your libz.so didn’t get put first on the path. Try adding extract = True, executable = True to the parameter list to the file() call.

If that doesn’t do it, you might need to remove morepy from the package list.

David

30

Unfortunately, that didn’t help. Current contents of the pdef file:


import sys
sys.path.insert(0,'')


class MyP3D(p3d):
    packager.binaryExtensions.append('so')
    require('sqlite','panda3d','morepy')
    sys.path.append('/usr/share/pyshared')
    sys.path.append('/usr/lib/python2.6/dist-packages')
    config(
        version="0.0",
        display_name = "MyP3D")
    module('mechanize')
    file('libxml2.so.2')
    file('./libz.so.1',newDir='.',extract=True,executable=True)
#    file('/usr/lib/libxml2.so.2.7.6',newDir='lxml')
    file('lxml/etree.so',newDir='lxml')
    module('lxml')
    excludeModule('cjkcodecs.aliases')
    excludeModule('config')
    excludeModule('iconv_codec')
    excludeModule('panda3d.core')
    #dir('lxml')
    mainModule('main')

So I gotta remove morepy, right? No hope then? :<

31

So, there’s no way forcing a so file to load. BTW, if I remove the newDir=".", the file doesn’t even get included. And after removing morepy, it still loads the libz:

> strace -f panda3d output_dir/MyP3D.0.0.linux_i386.p3d 2>&1 | grep open | grep -v ENO | grep '\.so' |  egrep 'libz|lxml'
open("/lib/libz.so.1", O_RDONLY)        = 3
[pid  8220] open("/home/deetah/.panda3d/hosts/runtime.panda3d.org_96a0ec538744def6/panda3d/cmu_1.7/libz.so.1", O_RDONLY) = 7

I even excluded it explicitly, still no help.

32

…So, no hope to fix it from this point?

33

Hmm, well, as rdb pointed out, the most straightforward fix is for us to include the more recent version of libz in the future releases of the Panda3D runtime.

But for your short-term needs, it does appear troubling. :frowning:

David

34

I’d rather hear an honest “we can’t do anything” :stuck_out_tongue:

35

I’m not trying to dissemble. I just mean that it won’t be fixed quickly.

There appear to be two different problems. (1) the libz version provided by the Panda3D runtime is the wrong version. (2) the PATH variable set up by the runtime isn’t allowing the files on your p3d file to shadow the ones in the Panda3D runtime.

We could solve your problem by fixing either one of these problems. But either fix will require a new update of the Panda3D runtime, which won’t be possible in the short term. So, yes, “we can’t do anything”, at least not immediately. But we will solve it eventually.

David