Multiprocessing for built binaries

Hi, I’m experiencing an issue incorporating multiprocessing into my game. Almost everything in the game is procedurally generated. It’s quite cpu heavy and I don’t want to run it in the main thread blocking the UI. Instead I implemented a background job manager which receives requests from the main thread. The requested task is stored in a multiprocessing.Queue from which the child processes fetch their tasks. When they complete the task they return the result along with the associated task id via the output Queue to the manager. The whole thing works very well.

I was trying to share an ad-hoc build so I did python3 setup.py bdist_apps. When I launched the binary it seems that every time the game spawned subprocesses, instead of them doing their task loop they each executed the if __name__ == '__main__': logic and started several new instances of the game (one for each subprocess). Then each of those started their own subprocesses recursively, cascading until my laptop crashed.

Reproducible example (Have a pkill -f ready in another terminal if you build and run the binary):

from time import sleep
from direct.showbase.ShowBase import ShowBase
from multiprocessing import Process

def subprocess():
    while True: sleep(5)

class App(ShowBase):
    def __init__(self):
        super().__init__()
        self.subprocess = Process(target=subprocess, daemon=True)
        self.subprocess.start()

if __name__ == '__main__':
    App().run()

Is this a known issue, and can I somehow work around it?

1 Like

Have you tried using threading2? Threading — Panda3D Manual

Hi Simulan, I tried it, but it seemed to behave as If I was using python’s built in threading module, which can’t take advantage of multiple cpu cores. Instead, despite running asynchronously, it still heavily impacted the performance of the main thread and my game still only utilized a single core. Am I missing something, can I instruct direct.stdpy.threading[2] to run on multiple cores?

If multiprocessing tries to spawn another Python instance, that’s going to break a built binary I believe. Here’s an old thread talking about it: Issue with multiprocessing.Connection when running as a p3d - #2 by drwr

This might also be of interest to you: PEP 703 – Making the Global Interpreter Lock Optional in CPython | peps.python.org

Hmm, the second link says that will be possible in python 3.13, that’s probably at least a year away. You’re saying I can’t really address it until then? Do you know if I can expect the same issue if I rewrite everything in c++?