SimpleHashMap is invalid error

teedee · October 24, 2011, 10:50pm

I’ve built the latest source from CVS this morning, and I am getting a problem where after changing an EGG file I get an error. Loading the program again it runs just fine and I will not get the error again until I change another EGG file.

The error:

:util(error): SimpleHashMap is invalid: key 30B7005C should be in slot 10758 instead of 588 (ideal is 10758)
SimpleHashMap (1101 entries): [ * * * * * * * * * * * * * * * 09C158DC * * * * * * * * * 31676DF4 * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * 30E56A14 * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * 0CBB8114 * * * * * * * 316787DC * * 30C79624 * * * * * * * * * * * * * * * * * * 2F77416C * * * * * * * * * * * * * * * * * * * 30BD9D3C * * * 3172497C * * * * * * * * * * blah blah blah VERY LONG ]
Assertion failed: _attribs->validate() at line 257 of c:\work\panda3d\panda\src\pgraph\renderAttrib.cxx
Traceback (most recent call last):
  File "C:\work\sp1\Panda3D\direct\showbase\ShowBase.py", line 1711, in __garbageCollectStates
    RenderState.garbageCollect()
AssertionError: _attribs->validate() at line 257 of c:\work\panda3d\panda\src\pgraph\renderAttrib.cxx
:task(error): Exception occurred in PythonTask garbageCollectStates
Traceback (most recent call last):
  File "client.py", line 564, in <module>
    game.run()
  File "C:\work\sp1\Panda3D\direct\showbase\ShowBase.py", line 2648, in run
    self.taskMgr.run()
  File "C:\work\sp1\Panda3D\direct\task\Task.py", line 502, in run
    self.step()
  File "C:\work\sp1\Panda3D\direct\task\Task.py", line 460, in step
    self.mgr.poll()
  File "C:\work\sp1\Panda3D\direct\showbase\ShowBase.py", line 1711, in __garbageCollectStates
    RenderState.garbageCollect()
AssertionError: _attribs->validate() at line 257 of c:\work\panda3d\panda\src\pgraph\renderAttrib.cxx

My guess is the BAM caching has a problem.

drwr · October 25, 2011, 12:00am

Hmm, that’s a very bad error message, and it means that some of the internal data structures have become corrupt. It shouldn’t be possible to get into this state. It also doesn’t happen for me. Are you sure your build is intact, and you’re not inadvertently mixing and matching dll’s from different builds? What OS are you running on? Is there anything else exotic in your situation? What happens if you clear out your model cache altogether?

David

teedee · October 25, 2011, 12:51am

I’m fairly sure I am not mixing old and new, I cleaned out my build directory and did a full rebuild.

I am running on Windows 7 64 bit. I do always run two Panda processes, with one being the server running with no graphics window and the other being the client. The server program never has an issue, only the client, even though the server does load some EGGs for Actor skeletons, collision and so on.

If I delete the cache ( AppData/Local/Panda3D-1.8 ) everything works properly (until an EGG is changed of course).

Now that I try it in a simple test program, it works fine, so it looks like something in my game is exposing this behaviour.

drwr · October 25, 2011, 3:40am

Are you using C++ code, or is it strictly Python code? One of the things that can trigger this error is if the hash value for any of your RenderAttribs changes at some point after its creation, which means the RenderAttrib itself has changed (these are all supposed to be immutable objects, constant over their entire lifetime). This shouldn’t be possible with Python code, but it might be possible if you write your own custom C+±based RenderAttrib that doesn’t follow the rules, or if you’re otherwise mucking about in C++ space.

In any case, you might be able to narrow down when the problem actually occurs by putting “paranoid-const 1” in your Config.prc file. This will make things run considerly more slowly, but it will also check the RenderAttrib cache more frequently for inadvertent violations of the immutable rule.

David

teedee · October 25, 2011, 5:28am

My game code is all Python. I do have a few small C++ modifications but most have been integrated with Panda already. I don’t have any new classes or anything of that sort.
I’m not sure what I’m looking for in the output spew. The one thing that stands out is the “ideal slot” it is looking for does not seem to exist.

Assertion failed: validate_attribs() at line 414 of c:\work\panda3d\panda\src\pgraph\renderAttrib.cxx
:util(error): SimpleHashMap is invalid: key 0D55ADA4 should be in slot 226 instead of 31 (ideal is 226)
SimpleHashMap (101 entries): [ 084DB07C * * 150E42BC * * * * * * 084D9E44 * 0450BA3C * * * 0F69C02C * 0450B704 * 0A21F5BC * * 0F63970C 0F69010C 0DB277AC 0DB27E7C(1) * 0450B8DC 0F69BF14 * 0D55ADA4(61) * 0450B6D4 * * * * 084D663C * * * * * * 0A21F20C * * 0A21F47C 0D521E64(61) 099C9CB4 099C5B4C(2) 099C000C(3) * 099C4D84 * * 099C8E44 * * 099CE9B4 * * * 084DE6FC 099CFD84 099CFBB4 0F0F1C74 * 099CA904 099CF234(1) * * 0F690B14 * * 099CFAEC * * * * * * 099C860C * * * 0F34F0A4(207) * 0CC49AEC * * * * * * * * 099CFE64 * * 099C7EFC 084DE5FC * 099C4E94 099C91CC * 099CFCEC 099C843C * 0CC0611C * * * * 099CFC2C 0DAAD5AC(38) 0450B84C 099CFB74(1) 099CFE34(1) 0718FC5C 099CFBE4(1) 0CC04304(3) 084DEC3C 0D543DE4 07189F4C(7) 150E3B94(8) * 0D532D94(151) 0D53CB74 * 099CFC8C * 0DB27DDC * 099C55BC * * * 099CD224 * 0718F814 * * * 099C4E2C * 099C92C4 099C7ECC(1) 099CACB4(1) 0CC432EC(2) 099C8D34 0F690354(1) 099CFCBC 0F693F8C(198) 099CFDFC 084DB80C(9) 0D54DE94(6) 0CC0659C * * * 0F68948C(125) * * * * 099CFDC4 * * 099CF674 0CC71C9C(1) * * * * * * * * * * * * 0450B8AC * * * * * 0CC38AE4 084DEB5C(1) 141CEE9C(2) * * * * 0450B87C 0718FE1C 0D5549FC * * * * * * * * * 099C8E14 * * * * * * 099CDD84 * * * * * * 099CFB2C 099C82C4 099CFC5C * * * * 099CB33C 0F63957C(1) * 099CD1F4 0F69BCAC * * 0F69BDD4 07181724 * * * 099C9D5C 084D923C * * 099CA744 * * * * * * * * * ]
:pgraph(error): RenderAttrib::_attribs cache is invalid!
0: 084DB07C
  ShaderAttrib
3: 150E42BC
  ShaderAttrib
10: 084D9E44
  ShaderAttrib
12: 0450BA3C
  CullBinAttrib:unsorted,0
16: 0F69C02C
  ShaderAttrib
18: 0450B704
  CullFaceAttrib:cull_clockwise
20: 0A21F5BC
  TexGenAttrib: default(point_sprite)
23: 0F63970C
  ShaderAttrib
24: 0F69010C
  LightAttrib:on
    entity_light
25: 0DB277AC
  CullBinAttrib:agent,22
26: 0DB27E7C
  LightAttrib:on
    agent_light
28: 0450B8DC
  CullFaceAttrib:cull_none
29: 0F69BF14
  ShaderAttrib
31: 0D55ADA4
  TextureAttrib:on cloud_png:enemyShield_0001 Shadow_Map_01_png:Shadow_Map_01
33: 0450B6D4
  RescaleNormalAttrib:none
38: 084D663C
  ShaderAttrib
45: 0A21F20C
  RenderModeAttrib:unchanged, thick 0.32, perspective
48: 0A21F47C
  TextureAttrib:on default:smoke
49: 0D521E64
  TextureAttrib:on cloud_png:enemyShield_0001 Shadow_Map_01_png:Shadow_Map_01
50: 099C9CB4
  LightAttrib:all off
51: 099C5B4C
  ShaderAttrib
52: 099C000C
  ShaderAttrib
54: 099C4D84
  ColorBlendAttrib:add(one,one)
57: 099C8E44
  AudioVolumeAttrib:identity
60: 099CE9B4
  ClipPlaneAttrib:identity
64: 084DE6FC
  TextureAttrib:on default:font_helptron
65: 099CFD84
  LightRampAttrib:default()
66: 099CFBB4
  ColorWriteAttrib:rgba
67: 0F0F1C74
  LightAttrib:on
    agent_light
69: 099CA904
  ShaderAttrib
70: 099CF234
  TextureAttrib:on default:_0
73: 0F690B14
  LightAttrib:on
    entity_light
76: 099CFAEC
  ColorAttrib:off
83: 099C860C
  ShaderAttrib
87: 0F34F0A4
  TextureAttrib:on player_0001_png:player_0001 Shadow_Map_01_png:Shadow_Map_01
89: 0CC49AEC
  CullBinAttrib:level,19
98: 099CFE64
  TexMatrixAttrib:
101: 099C7EFC
  (SRS_front_enable, 0)(SRS_back_enable, 0)(SRS_front_comparison_function, 7)(SRS_front_stencil_fail_operation, 0)(SRS_front_stencil_pass_z_fail_operation, 0)(SRS_front_stencil_pass_z_pass_operation, 0)(SRS_reference, 0)(SRS_read_mask, 4294967295)(SRS_write_mask, 4294967295)(SRS_back_comparison_function, 7)(SRS_back_stencil_fail_operation, 0)(SRS_back_stencil_pass_z_fail_operation, 0)(SRS_back_stencil_pass_z_pass_operation, 0)(SRS_clear, 0)(SRS_clear_value, 0)
102: 084DE5FC
  TextureAttrib:identity
104: 099C4E94
  TextureAttrib:on default:symbography
105: 099C91CC
  TransparencyAttrib:none
107: 099CFCEC
  LightAttrib:identity
108: 099C843C
  ColorAttrib:(0 0 0 0.4)
110: 0CC0611C
  ColorAttrib:(1 1 1 0.5)
115: 099CFC2C
  DepthOffsetAttrib:(0)
116: 0DAAD5AC
  TextureAttrib:on dummy2_png:dummy2 dummy1_png:dummy1 cloud_png:enemyShield_0001 Shadow_Map_01_png:Shadow_Map_01
117: 0450B84C
  DepthTestAttrib:none
118: 099CFB74
  ColorScaleAttrib:identity
119: 099CFE34
  ShadeModelAttrib:smooth
120: 0718FC5C
  ColorWriteAttrib:r
121: 099CFBE4
  CullBinAttrib:(default)
122: 0CC04304
  ColorAttrib:vertex
123: 084DEC3C
  ShaderAttrib
124: 0D543DE4
  ShaderAttrib
125: 07189F4C
  ShaderAttrib
126: 150E3B94
  ShaderAttrib
128: 0D532D94
  TextureAttrib:on bolt_0001_png:bolt_0001 Shadow_Map_01_png1:Shadow_Map_01
129: 0D53CB74
  ShaderAttrib
131: 099CFC8C
  DepthWriteAttrib:on
133: 0DB27DDC
  ColorWriteAttrib:off
135: 099C55BC
  ShaderAttrib
139: 099CD224
  ShaderAttrib
141: 0718F814
  ShaderAttrib
145: 099C4E2C
  ShaderAttrib
147: 099C92C4
  CullBinAttrib:background,10
148: 099C7ECC
  AuxBitplaneAttrib(0)
149: 099CACB4
  TexGenAttrib:
150: 0CC432EC
  TextureAttrib:on default:explosion
151: 099C8D34
  ColorAttrib:(1 1 1 1)
152: 0F690354
  CullBinAttrib:entity,21
153: 099CFCBC
  FogAttrib:(off)
154: 0F693F8C
  TextureAttrib:on weapons_0001_png1:weapons_0001 Shadow_Map_01_png3:Shadow_Map_01
155: 099CFDFC
  ScissorAttrib:[0 1 0 1]
156: 084DB80C
  ShaderAttrib
157: 0D54DE94
  ShaderAttrib
158: 0CC0659C
  TexMatrixAttrib: default(T:(pos 0 1 scale 1 -1))
162: 0F68948C
  TextureAttrib:on radio_0001_png:radio_0001 Shadow_Map_01_png:Shadow_Map_01
167: 099CFDC4
  RenderModeAttrib:filled
170: 099CF674
  TransparencyAttrib:alpha
171: 0CC71C9C
  RenderModeAttrib:unchanged, thick 7
184: 0450B8AC
  MaterialAttrib:(off)
190: 0CC38AE4
  RenderModeAttrib:unchanged, thick 1.28, perspective
191: 084DEB5C
  LightAttrib:on
    agent_light
    entity_light
192: 141CEE9C
  ShaderAttrib
197: 0450B87C
  DepthWriteAttrib:off
198: 0718FE1C
  CullFaceAttrib:cull_unchanged(reverse)
199: 0D5549FC
  ShaderAttrib
209: 099C8E14
  AntialiasAttrib: none
216: 099CDD84
  AlphaTestAttrib:greater,0
223: 099CFB2C
  ColorBlendAttrib:none
224: 099C82C4
  AlphaTestAttrib:always,1
225: 099CFC5C
  DepthTestAttrib:less
230: 099CB33C
  CullFaceAttrib:cull_unchanged
231: 0F63957C
  LightAttrib:on
    agent_light
233: 099CD1F4
  CullFaceAttrib:cull_clockwise(reverse)
234: 0F69BCAC
  ShaderAttrib
237: 0F69BDD4
  ShaderAttrib
238: 07181724
  ShaderAttrib
242: 099C9D5C
  TextureAttrib:on default:CHANNEL1:sky_view
243: 084D923C
  ShaderAttrib
246: 099CA744
  ShaderAttrib
Assertion failed: validate_attribs() at line 414 of c:\work\panda3d\panda\src\pgraph\renderAttrib.cxx
Traceback (most recent call last):
  File "client.py", line 187, in poll_reader
    self.process_datagram(datagram)
  File "client.py", line 322, in process_datagram
    level.fx.add(c.FX_TYPES[fx_type], pos=pos, normal=normal)
  File "C:\work\sp1\fxmanager.py", line 37, in add
    FxMesh(self, effect, pos, normal, parent)
  File "C:\work\sp1\fxmanager.py", line 92, in __init__
    model = self.manager.level.game.loader.loadModel('assets/models/%s' % self.template['model'])
  File "C:\work\sp1\Panda3D\direct\showbase\Loader.py", line 160, in loadModel
    node = self.loader.loadSync(Filename(modelPath), loaderOptions)
AssertionError: validate_attribs() at line 414 of c:\work\panda3d\panda\src\pgraph\renderAttrib.cxx
:task(error): Exception occurred in PythonTask Poll connection reader
Traceback (most recent call last):
  File "client.py", line 565, in <module>
    game.run()
  File "C:\work\sp1\Panda3D\direct\showbase\ShowBase.py", line 2648, in run
    self.taskMgr.run()
  File "C:\work\sp1\Panda3D\direct\task\Task.py", line 502, in run
    self.step()
  File "C:\work\sp1\Panda3D\direct\task\Task.py", line 460, in step
    self.mgr.poll()
  File "client.py", line 187, in poll_reader
    self.process_datagram(datagram)
  File "client.py", line 322, in process_datagram
    level.fx.add(c.FX_TYPES[fx_type], pos=pos, normal=normal)
  File "C:\work\sp1\fxmanager.py", line 37, in add
    FxMesh(self, effect, pos, normal, parent)
  File "C:\work\sp1\fxmanager.py", line 92, in __init__
    model = self.manager.level.game.loader.loadModel('assets/models/%s' % self.template['model'])
  File "C:\work\sp1\Panda3D\direct\showbase\Loader.py", line 160, in loadModel
    node = self.loader.loadSync(Filename(modelPath), loaderOptions)
AssertionError: validate_attribs() at line 414 of c:\work\panda3d\panda\src\pgraph\renderAttrib.cxx

The full log was actually over 340 thousand lines of what looks like the same thing repeated over and over.

drwr · October 25, 2011, 3:09pm

I had been hoping that the paranoid-const flag would trigger an exception before model loading, thus pointing to a different part of code that’s actually causing the error. But that appears to not be the case.

Interestingly, the log does seem to single out a TextureAttrib as the object at fault (the first line mentions pointer 0d55ada4 as the failing object, and searching through the table below, we find that 0d55ada4 is a TextureAttrib). That’s plausible, since Panda does do some shenanigans with TextureAttrib when loading files, in order to unify the different Texture objects from the TexturePool. But this code is fundamental to all model loading, so I still don’t know why it would be failing in your application and not anywhere else.

Is there anything else you can find to isolate what triggers the bug?

David

teedee · October 25, 2011, 7:16pm

I will have to cut down the game code I suppose, although this provides a good opportunity for me to also find out why the threading is not working for me.

teedee · October 26, 2011, 12:04am

I’ve made a tiny demo that causes the crash.
I load 2 EGG files and run. No problems.
I swap one of the EGG files with another and run. It crashes.
Run again, no problems.
There are two EGGs in the models directory which are the ones that should be swapped.
Interestingly if the texture files are deleted it does not crash.
http://novuscom.net/~spacebullet/out/hashmap_crash.zip

I did also find the cause of the threading not working. I am pretty sure it is related to dynamic geoms created with the GeomVertexWriter. I am working on a minimal demo of that.

teedee · October 30, 2011, 9:45am

Could I bother some kind soul to repro the steps in my previous message (using the files provided) on a semi-recent version of Panda and confirm that it crashes for them as well? I want to be sure that it is not an issue with my Panda build or system.

drwr · October 30, 2011, 8:58pm

I have just successfully reproduced the crash using your code. I will now investigate. Thanks!

David

drwr · October 30, 2011, 9:45pm

Found it, and fix committed. That was a good one, thanks for your help in tracking it down.

David

teedee · October 30, 2011, 10:04pm

Ah the old delete a line of code trick!
Thank you, I think I would have sprouted a few more grey hairs figuring that out.