Profiling?

Hey all.

I’m using a smallish GeoMipTerrain for my terrain (autoflattened using Strong), and loading a few dozen actors. I’ve got a water shader running. A couple of lights. I get 70 FPS or so. Not awesome. From Pstats I’ve got perhaps 600 nodes and 250 geoms. I gather that this is within the realm of acceptability.

I set the models to moving around the screen, though, and suddenly my FPS drops to 25 or so. The animations themselves seem to account for 25% of that drop - turning them off earns me a dozen frames.

I’ve been trying to run cProfile to check my function counts (to see if it’s my code) and can’t - I get no output. Any tips?

Thanks!

Do you have the shader generator enabled? Try disabling it, at least for the moving nodes.

Hrm, maybe there’s something you change every frame that causes a shader recompilation.

Thanks.

It’s built on the YARR code. Most of it is gone, but it uses the water shader from there. Turning shaders off (using render.setShaderOff) didn’t seem to buy me much, though.

Don’t forget to disable video sync for any performance measurements. (It’s possible that your ~70 fps was due to a video sync at 72Hz, for instance.)

I’ve never had trouble using either profile or cProfile to measure performance. In either one, you can save the output to a disk file and examine the results at your leisure. Note also that there are some tools built-in to activate the Python profiler automatically; see direct/src/task/Task.py for the code. For instance, you can try putting ‘profile-task-spikes 1’ or ‘profile-frames 1’ in your Config.prc file to see what happens. (I haven’t actually used this automatic-profile code myself, so I’m not entirely sure what will happen.)

David

Hmm. Well, vsync is off - I was getting 100+ FPS before I started adding models to the scene.

I end up getting no output from cProfile. Am I using it wrong? I’m a C/C++ programmer, primarily, so I haven’t had much reason to profile Python code. I’ll take a look at Task.py and the Config.prc file.

Zoinks: an update.

So, I was able to get some performance improvements out of the surrounding area / terrain. Picked up 30+ FPS pre-animation, +8 post. Strangely, moving the camera away from the animated units gets me up to 120 FPS… but I was hitting 250 or so when doing the same before animating.

Which means it’s gotta be something in my own code, right? I’m taking a huge hit when the animated models aren’t even being rendered. Back for more profiling.

How many bones/vertices we looking at for each model?

Maybe your bottleneck is the part where all of those vertices get animated, or when they are being sent to the GPU?

PStats really should make it clear whether the bottlenecks are in your code, or in the animation subsystem or anywhere else.

David

15K vertices, 11 nodes, 2 geoms per model.

drwr - I’d like to find out precisely where. Does Pstats have a call graph somewhere?

It’s not like that; PStats isn’t a profiler. It gives you a top-down view rather than the bottom-up view a profiler gives you.

You can tell by the relative size of the colored bands on the “Frame” chart where most of your time is being spent; in a typical application, most of the time will be spent in “Cull” and “Draw”. If you’re seeing a lot of time in “App” or “*”, that’s not a good sign; you can double-click on that color band to see specifically what part of the band is taking so long.

“App” is the application, and includes all Python code (called “Show code” in PStats) and also collisions. If most of the time is “Show code”, you can double-click again on that and see the particular breakdown by Task.

“*” is general overhead, and includes animation. If most of the time is in animation, you can double-click on that and see the breakdown by Actor; and you can double-click again to see the breakdown between animating joints and animating vertices.

“Cull” and “Draw” are used by Panda to render the scene; if “Cull” is high, you can reduce it by reducing the number of nodes; if “Draw” is high, you need to reduce the complexity of the scene by simplifying shaders, or reducing the number of Geoms.

This is just the 10,000 foot view of understanding PStats. There are of course subtleties with all of the above.

David

Lovely. Thank you. I’ll take a look.

So… it looks like it’s not my code.

During animation I’m seeing fairly constantly-spaced spikes in igLoop and and a constant tall band in skinning.

“skinning” is computing vertices and normals for animation. It can get expensive if you’re animating lots of vertices. Pre-processing your characters with egg-optchar will help reduce this cost, sometimes substantially. You might also consider stripping the normals off the model, if you can do without them. (Tangents and binormals must also be animated, if they’re present, so be sure you strip these if you’re not using them.)

Further down the list of things to try is not using intra-frame blending, if you are using it now. Finally, you can try using Actor.setLODAnimation(), to reduce the frequency of animation updates as your actors get farther from the camera.

David

Thanks very much.

LOD’ing the animations netted me 20 frames. Great.

egg-optchar got me about 70 FPS without animation, perhaps 20 with animation. Good deal.

Problems:

  1. I also got some amazing grey planes that stretch all over the place - I’m assuming egg-optchar optimized away something the animations were using.

  2. I did attempt to keep necessary joints using “-flag” as mentioned in the manual but still got warnings.

Any tips on either of these?

I’m pretty happy with 150+ FPS standing still, although I expect with more models and effects I’m going to be dipping back into Pstats. It feels funny to be optimizing so early in the process, frankly.

Hmm, egg-optchar certainly shouldn’t break any straightforward animation playback. I haven’t seen anything like what you describe. Are your character models well-defined, with all vertices assigned to one or more joints? Do you see these artifacts even when viewing your models in pview? Perhaps you can post one of your models and its animation files so I can see it too?

The -flag option keeps geometry, not joints. It’s only necessary to mark geometry that you will be finding by name in code, to perform some operation at runtime (for instance, changing a texture or color). Use the -keep option to keep joints from being removed; but you only need this for joints that you will be finding by name in code, for instance to expose the joint for parenting things into hands and stuff.

What kind of warnings are you getting?

David

I’ve seen these artifacts before. We found that (from Max2009, I believe) we could only apply animations without artifacts when the model itself was exported with an animation. As in, loading model alone as base for an Actor - artifacts when used with any anim. Model + anim - all animations play fine.

Interesting.

I’m not currently seeing the artifacts anywhere, though. As far as I know they’re well-defined. I’ll ask about being able to post one of the models.

About the -flag… ahh. Okay. I’ll use -keep.

I’m getting a “joint named not found!” warning.

Oh, OK. That sounds like your skeletons are mismatching between model and animation files. In the case of egg-optchar, it’s extremely important to run all model files and animation files that share the same skeleton (and will be played together) on the same command line. This is because egg-optchar is going to be removing joints and re-shuffling the hierarchy, and this can only work if it can do this to all involved egg files at the same time. If you do it on the model only, or the animation only, then the model file will get out of sync with the animation file, and you’ll get artifacts like this.

Well, are you sure you really do have a joint named ? You can examine the egg file directly–or use “egg-optchar -ls”–to see the actual joint names. They may or may not precisely match the joint names in Max or Blender or whatever your source file is, depending on the foibles of the egg converter in use.

David

Ahh. My fault. I thought I was using only animations from within the base egg file. Turns out I was using another animation egg as well. Thanks. Did them all together and now, no artifacts.

I do indeed have that joint (egg files are remarkably helpful!). I used “-keep” to keep it and now everything works swimmingly.

Thanks very much for your help. I’m up from 23-24 FPS mid-animation to 90-95. 60 at worst. I’m sure I’ll be back with more questions, but I’m very happy with these results. Cheers.