Python and Multicores

Python, Panda and multicores - A Summary


Since a few month I own a multicore CPU and with that I realized that pyhton is only running on a single core and not utilizing more than one core. I tried to find out whats wrong, cos my code was threaded.
I noticed that many coders stumble over this problem so it thought it might be helpful to summarize what I found out about Python, the global interpreter lock (GIL) and multi-threading vs. -processing.

I have to admit that I dont have decent knowledge about the ins and outs of this topic, so if you notice mistakes or I was missing something it would be nice to point it out and I will add it to this summary.

The Panda forum offers two threads about multicore utilisation so far. I took two quotes form them (David and Yosh) about Panda and multicores. … +threading … =multicore

In the link section you will find the links to all papers I took into account of this summary, so feel free to have a look at them.


> The GIL - global interpreter lock <

The GIL prevents multiple threads to run at the same time, so they cant interfere. But this also means you cant run threads on different CPUs, too. It is important to understand the GIL, cos else you will wonder why your pyhton-code is only running on a single CPU on your shiny new “multi-octo-core-CPU”. GIL is not a part of Python but of the interpreter cPyhton.
The GIL has advantages (no locks, no interferences of threads) and disadvantages (cant utilize multicores via threading).
Currently, the only languages that support multi-core threading are Java and C++, as far as I understand it. Most other languages go other ways.
It is often requested to remove GIL from pyhton or at least make it optional, but up to now no usefull solotion has been found. 1999 it was tried to remove GIL with a fork of pyhton 1.5 (?) with drastical performance reductions when running threaded (30-50%).
Also, there are ways to bypass the GIL. One way are c-extension. From what I read c-extensions release the GIL so multicore threading becomes available. A good read might be the article “Threading the Global Interpreter Lock”.

Related to Panda theres also an interesting quote from Yosh:

and a quote from David:

> Multi-threading vs. multi-processing <

Van Rossum said in this blog that we have to get over the “brainwashing that threads are the only way to do multicoreing”. Cos of the GIL, the current “phythonic way” seems to be multi-processing - running different processes that communicate to each other via IPC (interprocess communication). Multiprocessing has the advantage that you dont need to have your processes on the same machine, so its the common way for clusters (lots of networked computers).
A problem is that python currently isnt offering neat ways to realize easy and painless IPC, YET.

> IPC - Interprocess Communication <

Rather than trying to explain it myself I let the wiki speak. You will also find different ways to utilize IPC with pyhton, but also notice that most of this is (unfortunatly) platform specific.

From the wiki:


MPI APIs like pyMPI might also provide ways to mulicoreing, but I cant provide much to that topic.

APIs that provide multicore-access:

Jython via Java VM
Iron Python via MS VM(?)
Parallel Python:
PyMPI: (outdated?)
APIs that use sockets, rather than coding on pure sockets (e.g. Twisted network)

> Links <

How GIL works:

Thread State and the Global Interpreter Lock:

Threading the Global Interpreter Lock: … reter.html

About “remove GIL” requests:

open letter Brendel:
response van Rossum: … ead=214235
Ongoing discussion:
van Rossum blog: … msRange=15
brendel blog:

Problems of removing GIL: … 17099.html

Latest multi-core improvements from the core developer A. Olsen (9/2007):
The posts from Olsen are a very interesting read! … msRange=15

Hmmm… is this a question? A statement? Or a Request for Comments? :slight_smile:

This sums pretty much up what I know about the multicore"problem" with current Panda3D releases. However, I can suggest using “pyro” for some basic (and slow) IPC stuff. :slight_smile:

Regards, Bigfoot29

Thx, for the reply. I wasnt sure if I got most or all aspects of adressing multicores. As you noticed it sums all this stuff up, hence the word “summary”! :smiley:

Hi, I discovered this limitation myself very recently, and was very bummed out (I wanted to get a very heavy simulation going on my i7… but alas it is bound to run at 1/8 capacity max). I thought about switching to another engine to get around it… but I really like Panda3D and would avoid switching if I could find another way to run this sim in multiple cores. So… I am working on a .NET wrapper for the C++ libraries. Progress will be slow as I don’t intend to use SWIG, but the idea right now is to work through the tutorials and engineer the bindings as needed. For testing I will be duplicating the C++ logic in C#, so I can provide example code to those who would use the bindings. Ultimately I would like a fully-functional binding that allows Ironpython to run on top of Panda3D exactly the same as its CPython counterpart (or at least very close to the same).

Any ideas on a good place to host the project? Only using dropbox for now…

What do you find lacking in ?

points to date of origional post and comments that he might not be around anymore

Wow necrod!

Aye, still, multiprocess is a very viable option as long as all panda modules are only touched by one process. Can also spawn different python processes too and communicate via pipes or sockets.