Was just wondering, why are there composite files? I’m guessing something to do with interrogate perhaps?
Hugh
Nope, not at all. It’s just a compile-time optimization. A C++ compiler spends most of its time reading and parsing all of the header files, and in a particular directory it’s likely that many of the C++ files use the same header files. So by combining them all into one or two files, the net time to compile the whole directory is substantially reduced.
It’s kind of like using precompiled headers, only it works better and it’s platform-independent. 
David
Hmmm, that makes sense.
That begs the questions:
- why not combine all the files from one directory into a single composite? gobj for example is now split into 4 composites?
- if its because its different categories of files for the composite, can we use a postfix on the composite name to identify the category?
- can we generate these files on the fly, eg in makepanda.py?
Hugh
One limiting factor is RAM. Compiling hundreds of files at once can consume a lot of RAM, making compilation difficult on a low-memory machine.
Another factor is parallelizing the build–if you are using GNU make, you can configure it to run multiple compilation steps at the same time, so if you have a two-processor machine (with lots of RAM) you will do better to have each directory divided into two composite files which can both be compiled in parallel.
It is, however, possible to generate a single composite file on the fly; ppremake can optionally do this. It’s surely possible to extend makepanda to do this as well.
However, lately, the big bottleneck in build time has come to be the interrogate step, which has gotten a lot slower over the past twelve months or so–perhaps the code has crossed over some complexity threshold recently, and we’ll need to retool interrogate to keep up.
David
Interrogate is very slow.
Btw - and maybe I should make a new thread for this? - how does interrogate and swig (swig.org) compare? What are the advantages and disadvantages of each? (Are they even comparable?).
The reason I’m asking is that I’m probably going to be migrating my own C++ project to Python, and just wondering if I should use Swig? or Interrogate? or something else?
Hugh
Interrogate and Swig are very comparable. In fact, through some kind of convergent evolution, they have almost the same set of features. We wrote interrogate well before Swig had the sophistication that we needed (in fact, we started working on interrogate before we had even heard of Swig–it was just a tiny little project for Python at the time, and we were working with a language called Squeak back then).
Since then, Swig has followed many of the same design decisions we employed in interrogate. Between the two, I’d probably recommend using Swig for a new project, just because it’s got more inertia and support as a project in its own right.
But there are some nice features to interrogate. One thing I particularly like is its transparency–you don’t need to define any extra interfaces for your code, or for that matter, hardly anything at all–just the keyword PUBLISHED instead of public here and there. And that means it’s always generating up-to-date interfaces, even as you rework your code.
Also, we’re even now working on a major update to interrogate, which will generate Python objects directly in C++, without incurring the double-wrapper overhead that it does (and Swig does) now; this will make load time much much faster and should also make a substantial improvement on run time. That upgrade may be enough to swing the pendulum in interrogate’s favor, but ultimately it’s up to you.
David
Ok, that’s cool.
we were working with a language called Squeak back then
Squeak? I’ve heard of that. It seems popular in certain academic circles?
What were your thoughts on Squeak? What factors entered into the decision to move to C++/Python? How does Squeak compare with C++? and with Python?
Hugh
Right, Squeak is a variant of Smalltalk, and it was developed by Alan Kaye, who was at the time employed by Disney, so it seemed like a good fit. Our initial design was a C++/Squeak hybrid, similar to our C++/Python hybrid now. (We had just migrated from the SGI Irix platform, where we did everything in C++/Scheme, so we were already comfortable with the interpreted-language-on-top-of-C++ model.)
Squeak is a late-binding interpreted language, like Python and Scheme are, and we thought it would work well for us. Its language constructs were somewhat unfamiliar, though, and the learning curve was steep; but there were a lot of nice things about the language once you got into it.
What finally killed Squeak for us was its academic nature: the Squeak team was focused on maintaining the purity of the language, and not so much in pragmatically solving our deployment issues. While that’s commendable, it wasn’t particularly helpful for us, so we eventually decided we needed something that had a more real-world focus.
David
If you are interested in Squeak, check out the Croquet Project http://opencroquet.org
Alan Kay is on the committee. This is a very “academic” project indeed. Interesting though, and looking at it will give you an idea of what you would have with Squeak instead of Python. Namely, a bunch of eggheads getting more excited about HOW their code works than WHAT it does. Very necessary, for the advancement of computer science. But, too much of that will doom a project to wallflower status while other, more practical groups are inspired by parts of it and go off making programs that DO things.
Anyway…Croquet is cool…but not in a practical way in my opinion =)