The Project
I’m working on a large virtual-world project. Check screenshot here. The terrain is generated using a slightly modified marching cubes algorithm with the terrain divided into 32x32x32 chunks (size to be optimized later).
The Problem
I’ve tried a few different approaches to generating new terrain on the fly. Building a whole chunk at once is decently quick, but causes framerate hiccups. Spreading the job out over time causes no hiccups, but takes forever to build a single chunk. Every frame I do a simple search for missing chunks within a short distance of the player. When a chunk is found that needs loaded, the loading process is started and the search exits. It doesn’t search again until the new chunk is done loading so there’s only ever one chunk loading at a time.
The Question
Is this a case that would benefit enough from multi-threading to make it worth compiling panda with full threading support? I imagine it is because I would have this loading process always running on the second core (which would give me fast loading without slowing down the game.)
Semi-related Questions
Are there instructions somewhere on compiling panda3d from source? I haven’t found any yet but I also haven’t downloaded the source.
Can anyone suggest a different option I haven’t thought of yet? Could I run another thread without any panda code without compiling?
Yes, but it also means they must never touch any Panda3D objects at all, not even to allow them to destruct.
And note that Python threads will generally not give you the parallelism you are hoping for, since Python itself is inherently single-processor, and the Python interpreter will enforce this.
But it sounds like panda3d threads don’t have this problem because they’re not implemented in python?
Anyway, I’ve successfully built panda for real threading (thanks to Chrys for his helpful links) so now I’m closer to my goal. From what I’ve got working so far it loads in the data much faster than before and doesn’t cause framerate issues.
The best I’ve got so far is using an asynchronous task to generate the necessary data, then when it’s done I synchronously build the Geom from that data and stick it in the scene graph. This synchronous part is the only thing I still need to resolve right now. However I try and put this code into the task it locks up the main thread. It should be obvious that this is my first go at multi-threading so I really don’t even know what to look out for. What kind of operations should I be doing to make this process safe?
“Synchronous” means it runs in the main thread, so of course it will lock up the main thread for whatever period of time it takes to generate the Geom. If that period of time is very small, that’s not a problem; but if it’s noticeable, then you get a noticeable hitch in the rendering.
So your two approaches to eliminate a noticeable hitch are:
(a) reduce the amount of time required to generate the Geom, or
(b) do this part of the task asynchronously as well.
It is certainly an option to generate a Geom in a sub-thread. Just parent it into the scene graph in the main thread.
Loading in the task then creating the Geom in the main thread worked, but stuttered as you said.
Creating the Geom in the task and then attaching it to the scene graph in the main thread locked up the program.
Just tried something different: changed the main update task to asynchronous. Now both threads keep going like they should (they keep writing stuff to the console) but the program itself is still unresponsive.
Oh, hmm. Deadlock, then? This is a common problem when writing multithreaded coded. How are you passing the Geom from your child thread to the main thread for adding to the scene graph?
I was just handling the objects directly from both threads, but I can see that’s a bad idea. Did some more reading and found that queues seem to be used commonly for this sort of thing, so I tried. Still deadlocking and it seems to keep trying to load the same chunk all the time. So here we go:
from direct.stdpy import threading
from direct.showbase.PythonUtil import Queue
loadQueue = Queue()
doneQueue = Queue()
class EChunkLoader(threading.Thread):
def run(self):
# just loop forever looking for stuff to load
while True:
if not loadQueue.isEmpty():
chunk = loadQueue.pop()
if chunk.read:
chunk.readDensityMap() # this and fillDensityMap() get the data used to create the geometry
else:
chunk.fillDensityMap()
chunk.generateGeometry() # this actually creates the Geom
doneQueue.push(chunk)
#end EChunkLoader.run()
#end class EChunkLoader
# this is inside the map class
def update(self, x, y, z):
print 'update'
self.playerX = x # player position
self.playerY = y
self.playerZ = z
if not doneQueue.isEmpty():
doneQueue.pop().finishLoad() # puts the Geom on the scene graph
start = int(self.chunkDimensions / 2)
end = self.chunkDimensions - start
startz = int(self.chunkZDimensions / 2)
endz = self.chunkZDimensions - startz
for x in range(-start, end):
for y in range(-start, end):
for z in range(-startz, endz):
chunkX = int(self.playerX / self.chunkSize) + x
chunkY = int(self.playerY / self.chunkSize) + y
chunkZ = int(self.playerZ / self.chunkDepth) + z
key = str(chunkX) + '-' + str(chunkY) + '-' + str(chunkZ)
if key not in self.chunks:
self.chunks[key] = EMapChunk(self.npMapRoot, self.loadingChunkX, self.loadingChunkY, self.loadingChunkZ, self.chunkSize, self.chunkDepth)
loadQueue.push(self.chunks[key])
#end for
#end for
#end for
#end EMap.update
