Panda 1.10.0 w/ Python 3 - DCPacker UnicodeDecodeError

EDIT: The first four posts in this thread pertain to an issue regarding code which I do not own, and thus I have removed the reference to it. My reply about four posts down contains a much more succinct example of the issue.

In summary, attempting to call getString() on a DCPacker instance after packing large uint fields seems to be causing trouble with Python 3’s unicode decoding.

This looks like an issue in Astron, which may not support Python 3. This is not related to Panda3D, so it would be best to report this on the Astron github issue tracker.

In this case it looks like it is passing a string to a function that accepts a bytes object.

ping @cfsworks

The Astron Panda3D fork has not yet been tested on Python 3, which is one of the reasons why it isn’t merged back down into mainline Panda3D yet. The Panda3D forums would not be a good place to look for support on the Astron extensions to direct.distributed until that happens. (However pull requests are definitely welcome on that repository in the interim.)

That being said, the leaked code you’re using is definitely not written to support Python 3. Furthermore, because I hold the copyright to that file (I wrote it from scratch; it is not a derivative work nor placed under any F/OSS license) and I have not released it to the public (the only reason you have it now is because it was copied without my permission), I strongly urge that you delete it and all copies in your possession and refrain from seeking technical support on a file you are not supposed to have, and by extension, is not supported.

Thank you for your reply. I apologize for using your code without permission, and I was unaware of your ownership of the file, since no author is mentioned in the file, and I had retrieved it from a public GitHub repository. Nevertheless, per your request, I have deleted the file and will work to resolve the issue using code written from scratch.

I’ll compile a few tests for the DCPacker to hopefully find a specific cause of the problem, and I’ll get back with what I find. I’m wondering about the compatibility of the Datagram.appendData() method and the getString() method’s return type between Python3 and C++
EDIT: After some testing, the Datagram class seems to be irrelevant to the issue. See below.

This appears to be an issue with the Panda3D’s DCPacker, or some mechanism related to it.
In a test AI repository, I have the following code:

# Test DCPacker for **** errors ****
def fake_updateObject(dclass, fields):
    # Create DCPacker
    fieldPacker = DCPacker()
    fieldCount = 0
    for k, v in fields.items():  # We pass in the array below, it's the only item.
        field = dclass.getFieldByName(k)
        fieldPacker.rawPackUint16(field.getNumber())
        fieldPacker.beginPack(field)    # Begin
        field.packArgs(fieldPacker, v)  # |
        fieldPacker.endPack()           # End
        fieldPacker.getString()         # !!! Causes an error!
        fieldCount += 1
fake_updateObject(self.dclassesByName['SomeClass'],
                  {'importantField': 100000123})

It it designed to pack only the ‘importantField’ field, with a uint32 similar to the size of an avId. However, the code does report the following error:

  File "TestAIR.py", line 34, in __init__
    {'importantField': 100000123})
  File "TestAIR.py", line 30, in fake_updateObject
    fieldPacker.getString()         # !!! Causes an error!
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe1 in position 3: invalid continuation byte

For reference, here’s the basic dclass code for SomeClass:

dclass SomeClass
{
    uint32 importantField required db;
};

The problem seems to be occuring with the third byte of 100000123, which has a value of E1. Again, any assistance would be greatly appreciated!

The error message is fairly clear: it is reading a sequence of bytes and then interpreting it as a utf-8 string, but it is not actually encoded in utf-8.

It looks like DCPacker uses string for all types, rather than distinguishing between vector_uchar and string (which in Python 3, map to bytes and unicode str, respectively). So packing arbitrary binary data does not seem possible at the moment with Python 3.

Thanks for clarifying that for me, rdb! I’ll try downloading the source from github and compiling it with a few revisions to the DC system to help with Python 3 compatibility. If any of my changes work, I’ll submut a pull request to the repository. Thanks again for all your help!

I’ve put forward a proposed set of changes to fix the Python 3 issues with dcparser here:

It lets you use getBytes() instead of getString() to get the result of a packer as bytes object instead of UTF-8 str.

1 Like