memory swapping low framerate

General Discussion
1

Hi,

I have a strange problem with my framerate. I made a simulator application which normally runs with > 60 Hz. After a few minutes, framerate suddenly drops to < 5.

It runs on a 4 core processor (intel core i7-2600K) with an Nvidia GTX560Ti (1GB) GPU and windows 7 Pro 32 bits. (I also tried it on 64 bits, as a 32 bits application, but the problem stays the same). I have 8 GB memory installed, but windows 32 bits ofcourse allows only the use of (a part of) 4 Gb.

The 4 processors as assigned as follows:

  • processor 1: ppython (copied as ppython-middle.exe) with settings and *.py files for rendering the middle display
  • processor 2: ppython (copied as ppython-left.exe) with settings and *.py files for rendering the left display
  • processor 3: ppython (as ppython-right.exe) with settings and *.py files for rendering the right display
  • processor 4: user interface and scenario generation
    I have set ‘lock-to-one-cpu #f’ in /etc/confauto.prc

When I run these 3 versions of panda, after a while frame rate drops from > 60 Hz to < 5 Hz, resulting in image freezes and very jittery images. In the taskmanager, it says:
Total memory: 3056 Mb, when running the simulations, free memory gradually drops to 0, but available memory stays well over 600 Mb. The drop in framerate is caused by memory swapping when free memory becomes too low.

When I limit the rendering to 2 processes (displays) instead of 3, the problem does not occur, because free memory never becomes 0 in that case.

Although this seems to make sense, it really isn’t. Available memory is still more than enough (> 600 Mb), so memory swapping should not occur.

I know there’s a hotfix for that from Microsoft (hotfix 324895) but this won’t install on my computer, saying ‘the update is not valid for your computer’. Also installing all new windows updates did not fix the problem.

I also have this application running on another computer (also windows 7, 32 bits), but on this system I have not noticed this problem. This is probably because total memory is larger there (around 3500 MB instead of 3056). Usable memory varies strongly between computers, depending on hardware installed. I use an i5 processor and a different mobo on the other PC.

It’s probably not a specific panda issue, but has anyone experienced this problem as well and found a solution for it ?

I’ve tested the memory banks and found no errors.

2

may i ask why you run 3 copies of your programm, consuming memory trice. instead of running one panda copy and opening 3 output windows with 3 cameras?

in any case you should have a look at panda’s pstats output (details for setup are in the panda manual). it should give you detailed information on what is slow, and where your memory goes exactly.

3

The reason I use 3 copies of panda for the rendering of the left, middle and right screen respectively, is that I can assign each process to a different processor. In that way all cores of the CPU are fully utilised. Some time ago, I did what you suggest: 3 different cameras in one process. But then only 1 processor is used for that and even this single process is assigned to use 3 processors, there’s still a large difference in performance. The 1-process method gives a framerate < 30 Hz while splitting the rendering in three different processes gives a framerate > 60 Hz. So, assigning each rendering process to a separate processor maximizes performance because the CPU is used fully.

Now, with this computer I use, I have 3056 MB available out of the 8Gb in the system, because it’s 32 bits windows. On other windows 7 32 bits computers that have 3500 MB available the jittering problem does not occur (or at least less frequently). So I have 2 different types of framerate variations. I used pstats to test what affects these two types:

  1. The normal framerate variation (between 30 and 80 Hz) depends on where you are in the database and the number of objects that are within view. It varies with number of (geom)nodes, as expected.
  2. Sometimes there’s a sudden drop in framerate to < 10 Hz. In that case, the application is unplayable and the images are very jittery. This lasts for a couple of seconds, and then framerate becomes normal again. On the windows taskmanager free memory is then zero (or close to zero) while available memory is always more than enough (> 600 MB). But the fact that free memory is 0 is not enough to cause this effect because most of the time when free memory is zero, performance is still normal (> 30 Hz). So, I can definitely say that, when these abrupt framerate drops occur, free memory is always very small (available memory still more than enough). In pstats, when the sudden framerate drops occur, there is no measurable effect on:
  • wait, app, cull or draw
  • geomcache size
  • number of nodes (although these are clearly related to the normal FR variations)
  • render states
  • vertex data
    I could not find any effect on variables in pstats.

I don’t think it is related to database size of number of actors or other data used in the simulation. It is not related to where you are in the database or which database used. For some reason, the system appears to start swapping memory to disk when free memory is zero, while I think the system should give available memory (from the cache) to the application, but it doesn’t.
It’s definitely related to memory usage, because if I use just 2 render processes instead of 3, this framerate drop does not occur. So, I think it is the USE of the memory by the system that causes the problem. Not the availability of memory, because there’s still more than enough (according to the taskmanager). Maybe it’s a windows OS setting that has to be changed or maybe I can change the memory management of panda. I already increased virtual memeory, but that does not solve the problem.

Any ideas would be greatly appreciated.

4

I did some more rigorous testing. When the large framerate drop occurs, the wait cycle in pstats increases. CPU usage then is, as always, still almost 100%, so that possibly means that the OS is doing something (like memory swapping) the application has to wait for ?

5

Related to this, I have a more specific question. I have made a number of databases, some small and some large. Farplane is set to 1000 meters, and in a small database all objects are in view from the start. In such a case, when the simulation is started and the database is loaded, the amount of free memory in the windows taskmanager hardly varies. It starts at 1300 MB and after the complete circuit has been driven, free memory is still about the same.

That situation is completely different when driving through a large database. Here, the taskmanager again reports around 1200 MB free memory after the simulation starts and the database is loaded. But when I drive through the databases, free memory rapidly diminishes to zero (after a few minutes).
When I set Farplane to 300 meters, memory still drops to zero ultimately, but it just takes just time.

Can this be related to the fact that new objects/polygons are coming into view and are selected by the culling process and drawn ?

I don’t quite inderstand how this works in terms of memory allocation. The database is loaded once when the simulation starts. No objects are loaded into memory during the simulation. So how can simply driving through the databases have such a large effect on memory consumption ? When new object come into view, these are selected by the culling process. So there’s probably some additional memory allocated. But when these objects are not in view anymore, isn’t that memory freed again ?

6

Perhaps you’re filling up a cache. Panda maintains a few caches of rendered geometry, because it sometimes has to process a geom before it can be rendered, and it optimistically keeps that processed geom around for a while in case the model needs to be rendered again.

You can control the size of the cache with the geom-cache-size config variable. The default is 5000, which might be too large if you have limited memory. Try setting it smaller, or turning it off altogether:

geom-cache-size 0

There are other, more advanced, controls for paging memory in and out of RAM if you really need to go this level of management, but I don’t recommend it for casual users.

David

7

Thank you. It appeared that there were a few causes. Since this problem has to be solved I replaced the mobo with another type. There was a problem with the sound as well. Often, there was no sound at all and then it switched on after a couple of minutes, so I removed the soundblaster card and used the on board sound. Then the system was much more stable. However, the most important thing is that texture memory on the GPU was overloaded. So, I applied “compressed-textures 1” in the config.prc and then these framerate drops were fixed. Should have thought about that earlier, but there was a lot of time pressure here. Sorry.