Performance issues & inconclusive pstats.

Alright guys, strange issue hopefully someone can help me out a bit.

Working on the space flight sim you guys have seen me post a bout.

Symptoms - after i leave my menu and instantiate everything needed for the level (tasks etc; (this takes a big show_code hit on pstats)); about 10 seconds later I get a HUGE performance spike. Frame rate drops really low; fmod will crackle a bit etc. After a second the fps goes back to 60; and runs great from then on.

I am baffled that the lag happens at a nearly fixed interval after i load a level and that after i instantiate the whole level (and tasks etc) i dont instantiate any new objects besides missiles and bullets.

I decided that pstats would be a good way to approach finding this performance issue. It seems as if every time i run pstats and wait for the big spike, it shows a different thing spiking. One time it showed Munge spiking, so i set “premunge-data 0”; another time it told me that the particles took a huge time to render, so i turned them off; another time it told me target screen took a long time to render, turned it off too. Symptoms keep passing to different objects, and even different parts of the pipeline. Ive had culling spike once or twice, as well as show_code spike once or twice (show code WOULD make sense if it was consistent, just would have to find it), as well as a few spikes in the * sub section.

Strangest thing here is i can just sit there( after everything has already been instantiated), and wait for the huge performance hit.

If you guys have any ideas it would be awesome, because I am stumped.

edit* No tasks are being spawned of speed > doLater of time .03
and no funny thread.timer().start() stufs going on either

If PStats shows a spike bouncing around to different parts of the render pipeline at random, then it means the spike is actually outside of your application altogether, and is caused by some external forces. For some reason the operating system is not giving your application its full share of the CPU.

Usually this happens as a result of virtual memory swapping–if your application is consuming a significant percentage of your physical RAM, for instance, then the operating system will need to pause it while it swaps out all of the other applications to disk in order to make room.

Another, less likely, possibility is that some other process which is configured to run at a higher priority than your Panda process has suddenly decided to compute the value of pi to the last decimal point or something like that. I saw that happen once on Star Trek.

You can usually tell what’s going on in the Windows TaskManager and/or the Performance Monitor. (Assuming you’re using Windows, of course. There are different tools on the other OS’s.)

David

Hmm; strange thing is its happening on all of our different consoles, performance hit varies on the slower laptops its a few seconds of unplayable: whereas on the faster computers its just a 20-30 FPS hit.

too bad we cant test it on mac… stupid pygame.

Didnt think of looking at the task manager; gonna do that now.

edit* cpu % sits at about 80% and 110(ish)Meg.

No real visual indication on the process itself that any big deallocation or something is going on - but when the performance spike does happen, it pegs the cpu at 100% then goes back down to 80% after the spike.

Hard disk will be working at a certain rate - when the spike happens, hark disk reads double, go to zero then after the spike return to the rate before they began. This could be related or could be just a side effect of processor being at 100%.

If it’s trashing, CPU load usually drops to nearly 0. So it must be something else, could it be loading assets ?

oddly enough; at the point where the spike happens the whole scene is loaded and the only tasks that are running are the flighthandler, and the tasks that control each of the in cockpit screens.

Yes, but the textures might not be fully loaded and uploaded to graphics card memory.

Is this what happen to you ?
discourse.panda3d.org/viewtopic.php?t=5007

Symptoms seems somewhat similiar; but tried this and same issue :’( was really praying it was that. (where tried this == node.prepareScene())

Although my scene does use a ton of textures; you see 95% of them immediately as you load into the given scene.

I wish i could figure this one out (bangs head against keyboard)

The cpu utilization: is that for the whole computer, or for your process alone? There isn’t any other process that’s stealing away some of your cpu cycles?

It might be python garbage collection, except that it’s not always restricted to “show code”. Also, the frame rate doesn’t completely stop, it just suffers severe hits for a few seconds. So it doesn’t sound like garbage collection.

Could it be something funny in your graphics driver? Do all your PC’s have the same kind of graphics card? Does it happen in OpenGL as well as DirectX mode? How about if you run it with tinydisplay software rendering (this test will require downloading and building the latest Panda)?

What happens if you run with:
textures-header-only 1
vertex-buffers 0
vertex-arrays 0

Still, it doesn’t really sound like a graphics driver issue, since once again it’s happening across the board.

Does pygame really not support Mac? It’s layered on SDL, and SDL supports Mac.

David

Thanks for the response dave; really interesting ideas - ill have time to test them out tommorrow, hopefully I can find em. CPU utilization is 100% for the python.exe process (like 10 threads) on the core that its running.

Ya i was under the same impression with pygame too; but the guy on our team that regularly uses a mac was saying he was having issues getting it all hooked up with the already installed python on leopard (I donno i dont do mac :slight_smile: ).

Thanks again dave; ill post again once i get some progress.

Threads? You’re using multiple threads? That could cause this sort of problem too, if one of your threads goes berserk. What kinds of threads are you spawning? Are you using Panda’s threading interface, or native Python threads? Which threading model is your Panda compiled for?

David

I dont really have any big “threads” going; but i use native python thread.timer(time, function).start() as a means to limit how fast a guy can shoot bullets. The timer function just sets some bool somewhere. I assumed this thread kills itself when its done (perhaps i keep a reference count somewhere)

Now that you mention it; this is probably a bad idea. I should probably use a sequence that kills itself at the end with a Wait() then a Func(setBool) on the end of it.

I am using the threadless panda; 1.5.4 off of the site.

Well, hmm, somehow you ended up with 10 threads.

How about audio? That often spawns several additional threads. Try running without audio. Put:

audio-library-name null

in your Config.prc file, so it won’t even load the audio libraries.

David

Hey, I’m the guy on the team that uses Mac and rwhughst asked me to comment on installing PyGame.

What I’ve found regarding using PyGame on Leopard is that it doesn’t like to use PyObjC 2.0 which as I understand it allows python to talk to the rest of the OS that is written in Objective C, and is the default for Leopard. PyGame seems to only want to use the older PyObjC 1.4 which was the default for Tiger.

I have managed to find and install 1.4 on my Leopard install and now PyGame works fine (at least to the limited extent to which I’ve tested it). I have been as thorough as I can be in testing this by doing so on a clean install of Leopard. First trying to install PyGame while still using PyObjC 2.0, then 1.4. Both installs succeeded, but upon trying to import PyGame with 2.0 python said it could not find PyGame. Once I installed 1.4 everything worked fine.