Animation very jerky unless another application is also animating something?!?

feuermurmel · April 28, 2020, 7:26pm

I’ve run into this issue of all animations being very jerky on one machine I use for development, but not on another (both running macOS 10.13.6). While investigating this issue, I made some curious observations, which might point to a problem within Panda3D.

Generally, all types of animations (using e.g. posInterval()) run very jerkily (uneven speed, frame rate varying wildly) independent of the complexity of the scene. I tried many, many things, including:

Setting the frame rate of the global clock to different values.
Putting the global clock into different modes.
Manually calling step() on the task manager.
Enable/disable vertical sync.
Running from an app created by build_apps instead of from the command line (macOS is known for applying magic to bundled apps).
Trying the supplied asteroids sample project (shows the issue too).
Rebooting the machine and using a test account without any background applications running.
Monitoring CPU and GPU usage (both stay in the range of 1 to 3 % when running the project).

Demonstrating the Issue

To demonstrate the issue, I created a minimal example project:

The project runs a continuous animation and writes a number to the console each frame. 100 is printed, if the time between the last and this frame (get_dt()) was exactly 1/60 of a second (my monitor is set to a refresh rate of 60 Hz). The number is lower or higher if the delay was smaller or larger, respectively. Here is a sample of its output:

 203 203 222  25 149 203 197 202 200  46 108 142  57  93 152 200 203 201 198 205
  52  98 142  56 101 146 200 204  52  99  99 100 144  57 100 148 200 200  53 101
  94 108 141  56 151 125 172 199 201 201 199 200 202  53  99 145  59  99 148 200
 198 200 202 203 197 198 203 202 199  53 100  98 101 141  60 147 133 164 202 199

As you can see, the numbers are all over the place, ranging between 50 and 200.

The machine I have this issue with has the following spec:

Macmini6,2
2.3 GHz Intel Core i7
16 GB 1600 MHz DDR3
Intel HD Graphics 4000 1536 MB

The other machine, which does not show the issue at all, has these specs:

MacBookPro11,1
3 GHz Intel Core i7
16 GB 1600 MHz DDR3
Intel Iris 1536 MB

The Observation

Now to the observation. I have found three cases in which the problem completely disappears:

Run the application full screen.
Run another application drawing animated stuff to the screen at the same time.
Record the screen using QuickTime Player

For the second case, I just ran the game Broken Age and let it sit on the title screen. Just playing a YouTube video in Safari or Google Chrome also works, but not in Firefox. To create the demonstration recordings below, I used the template chooser of Quartz Composer (one of the development tools for macOS), which displays an animated preview of the selected template. There I can easily switch between a template with and without animation.

I created two screen recordings which should make the issue clear. One uses the asteroids sample project, the other uses the example project linked above.

Video 1: https://www.youtube.com/watch?v=5HIWjQqpT9g
Video 2: https://www.youtube.com/watch?v=g1bmxKX-_nI

Next Steps

I’d like to get smooth animations with Panda3D on both of my machines! What do you suggest should I try next to narrow down the problem? Or does one of the developers what to look into this using my help?

Thank you in advance!

rdb · April 29, 2020, 10:16am

Which version of Panda3D? Does it differ depending on whether VSync is on or off?

If VSync is on, it might not be working properly, since your frame rate is getting all the way up to 240 Hz. If it is off, turning it on might mitigate these issues. The fact that having other applications active fixes it too might suggest that it is related to Panda trying to render so fast that it’s “stumbling over itself”, or something along those lines.

Another thing to try is to add a time.sleep(0.001) in a task somewhere.

feuermurmel · April 29, 2020, 2:20pm

I’ve made all these observations using Panda3D 1.10.6.post2. I forgot to mention that. Should I try other versions too?

I added load_prc_file_data('sync-video', '0') (as well as load_prc_file_data('sync-video', '1')) to the top of main(), but that did not make a difference. Neither did adding time.sleep(0.001) (or time.sleep(0.01)) to the top of on_every_frame(). I did not notice a difference in how the animation looks.

Also, I don’t think I’ve seen anything near 240 Hz. That would be a value of 25 in the output. The lowest I’ve seen was around 50 (120 Hz).

There is definitely something going on related to vertical syncing. I found that Quartz Debug (another development tool for macOS) has an Enable Vertical Sync setting (which is enabled by default). Disabling this actually makes a noticeable difference. The animation starts to switch between “jerky” and “fluid” every second or so (not a regular pattern, seems random). Now adding these two lines to the top of main() results in the animation having almost no jitter anymore:

taskMgr.globalClock.setFrameRate(60)
taskMgr.globalClock.setMode(ClockObject.M_limited)

Those lines have no effect while vertical sync is enabled in Quartz Debug.

With those lines and vertical sync disabled in Quarts Debug, there is still some jitter every second or so, which can be seen in the output here (1 line is 20 frames):

 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100
 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100
 100 100 100 100 100 122 205 194 201 203 199 199 201 203 199 200 200 206 193 203
 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100
 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100
 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100
 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100
 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100
 100 100 100 100 110 200 200 206 198 197 209 193 199 200 201 203 200 197 100 100
 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100
 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100
 100 100 100 100 100 112 198 200 205 197 201 198 100 100 100 100 100 100 100 100
 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100
 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100
 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100

This is quire peculiar. Most streaks of jitter start with one frame which takes a bit longer (e.g. 122 % of 1/60 of a second) followed by 5 to 20 frames which all take twice the time after which it drop back to a very consistent time of 1/60 seconds (probably thanks to M_limited).

What can I do to narrow this down?

feuermurmel · April 29, 2020, 11:16pm

Update: I tested it with version 1.10.0 built from source (the oldest version I was able to build on my machine without looking into it too much). It exhibits the same issue on my machine. I also tested with version 1.9.4 from the distribution and it does not exhibit the problem.

rdb · April 30, 2020, 9:24am

Ah, you built from source? Not exactly sure whether this matters, but did you build against the most recent macOS SDK, or an older one?

If this is related to VSync, we may need to revisit our implementation of VSync. In 1.10.0, we are no longer using the built-in VSync implementation, because Apple introduced significant bugs to the OpenGL driver in 10.14, among which was that VSync no longer functioned (see #486. Instead, we implemented a custom implementation of VSync. It’s possible that it has bugs.

Since you are still on macOS 10.13, it might be worth trying whether going back to the built-in implementation of VSync (that we had pre-1.10.0) would fix the problem. If so, we know that the problem is in our VSync implementation. If not, we know to look elsewhere.

To switch back, edit panda/src/cocoadisplay/cocoaGraphicsStateGuardian.mm and short-circuit setup_vsync like so:

bool CocoaGraphicsStateGuardian::
setup_vsync() {
  return false;
  ...

And also finding these lines:

  GLint swap = 0;
  [_context setValues:&swap forParameter:NSOpenGLCPSwapInterval];

And replacing them with this:

  GLint swap = sync_video ? 1 : 0;
  [_context setValues:&swap forParameter:NSOpenGLCPSwapInterval];

Finally, one more question: are you using multiple monitors, or just one?

feuermurmel · April 30, 2020, 9:43am

Ah, that sounds like a lead!

I built 1.10.0 from source, I did not specify --osxtarget. The produced wheel’s name is panda3d-1.10.0-cp27-cp27m-macosx_10_13_x86_64.whl. Then I tested with 1.9.4 downloaded from the releases and it indeed does not show the problem!

I will apply the changes you suggested to a checkout of v1.10.6 and report my findings.

btw.: Which commits are the panda3d-1.10.6.post1 and panda3d-1.10.6.post2 releases on PyPI based on?

rdb · April 30, 2020, 9:52am

Our build system assigns the numbers based on the commit distance. post1 is one commit after the v1.10.6 tag on the release/1.10.x branch, post2 the second one, etc. So post2 is 783c721cefa32d56b440605f776678634a013a28.

If you are getting from Git, no reason not to just check out release/1.10.x (or master if you feel more adventurous) and get all the latest fixes.

feuermurmel · April 30, 2020, 10:45am

Ok, those changes make a big difference! I don’t notice a difference anymore whether another application is displaying an animation nor between my machines. It’s probably as good as it gets frame-rate wise.

I’m ok with either approach, installing wheels form PyPI (via install_requires in a setup.py) or building from source.

Thank you very much this far! I will gladly use the resources I have to do further testing on my machines or to test potential fixes.

rdb · May 1, 2020, 3:08pm

Thanks for offering to help out. Perhaps I need to add debugging information so we can get more information from your computer. Or perhaps @treamous or @eldee might have time to try it and see if they could reproduce the issue.

Just to rule this out as a potential source of trouble, are you using a multi-monitor set-up?

feuermurmel · May 1, 2020, 7:32pm

No. I have a single monitor connected via Display Port. Not much here, but for what it’s worth, here’s the Graphics/Displays page from System Informatio.app:

Intel HD Graphics 4000:
  Chipset Model:	Intel HD Graphics 4000
  Type:	GPU
  Bus:	Built-In
  VRAM (Dynamic, Max):	1536 MB
  Vendor:	Intel
  Device ID:	0x0166
  Revision ID:	0x0009
  Metal:	Supported, feature set macOS GPUFamily1 v3
  Displays:
    DELL P2418D:
      Resolution:	2560 x 1440 (QHD/WQHD - Wide Quad High Definition)
      UI Looks like:	2560 x 1440 @ 59 Hz
      Framebuffer Depth:	24-Bit Color (ARGB8888)
      Display Serial Number:	MY3ND96B1D6T
      Main Display:	Yes
      Mirror:	Off
      Online:	Yes
      Rotation:	Supported
      Automatically Adjust Brightness:	No
      Connection Type:	DisplayPort

eldee · May 2, 2020, 8:30am

I tried your code on two macbook, one with macOS 10.14.6 and one with macOS 10.15.1, and both using the latest Panda distribution (1.10.6) targeting SDK 10.9+.

Strangely no one displayed your problem, either with no app running at all or with several other apps running, here is the log from one of the run (it’s similar on both machines) :

(all display modules loaded.)
6249 116  86  58  82 183  18  92 129  98  83 100  96 105  96 101  97 104  98  98 103  99 101  99 101  99 101 103 104 105  86 102  92 108  93 106 102  95 112  92 101  99 104  97  99  99 101 102  99 101  96 101 103  91 106 101  99 101  93 109  98 103  96 102 100 100  94 106  98 104 103  91 106 103  90 106 101  94 100 103  97 106  98  93 108  99 100  98 102  99 105 101  97  93 107  99  99  99  95 109  97 101 105  96  99 103  98  99 103  96  94 107  99 101 106  88 110  98  96 103  99  99 102  98  99 100 102  97 101 100  99 104  91 106 105  95  93 102 104 101  98 102  99 102 100 101  92 105  99 102 102  98  93 109  97  96 103 102 100 100  99 102  99  95 106 100 103  96 100 100  94 101 106 101 100  98  97  98 106 101 101  94 104 100  94 106  96 104 100  95  99 107  99 102  98 100  92 110  98  99  95 107  93 101 100 107  89 109  94 107 110  83 106 100 102  94  98 115  90  93 103 108  97  95 105  93 108  99  95 106 100  99  93 107  95 106 102  92 100 101  99 106  91 100 102 107 100 100  95 106 101 102  89 101 106  94  98 101 102 105  98 101 104  99  97  95 100 100  98 107  99  95 106  95  96 106  94 102 107  99  95 105  99 103  98 101  94 104  95 105 102  99 101  96  98 107  94 106  91 107  95 100 100 106 100 103  93  99 100 106 100  95 105  95 104  95  99 107 100 102  93  99 102 104  96 107  93 104 101  93 105  97 105  99  96 105  99 101  95 104 101  99  91 103 100 106 100  95 100  99 106  95 105 100  95 105 102  98 102  93 107  97  95 109  99  98 101 100 102 100  98  94 106 100 103 109  87 102  92 107 100  95 105 100 101  93 101 100 105  94 109  97 100  91 104 100 106 103  91 105 100 100  94 106 100 101  99  95 106 100  91 104 114  85 100 100 106  93 101 100 104 101  99  99  94 108  92 109  99 100  98 100  99 109  92 109  84 107 100 100  94 102  97 107 101  99 100  96 104  93 107 100  94 102 106  98  94 101  99 106 100  99 103  97 101  94 101 104  96 105 100  99  94 101 106 101  99 101  90 102 100 100 102 105 100 101 100  99 100  96  99 106  95 104 101  95  96 103 106  92 110  92 106 101  92 106  95 105  94 101 101 106  90 104 104 100  94 101  99 105  95 105  91 104 105  95 105  94 107  94 100  98 103 107  93 100  99 106 100  96 104  93 104  92 104 101 108  89 104 106  91 108  94 108 100 100  90 109  95 105  96 104 100  94 102 100  96 108  93 107 101  99  92 104 103  96  95 103 107 100  99  98 104 100  95 104  95 105 100  96 104  95 105  93 102  99 105  93 109  94  96 108  95 106  94 105 100 101  91 101 102 104  97 104  97  98  96 104 105 101  93 102 105 100 101  94 106  98  94 106 101  94  97 109  88 106  98 107 100  93 109  92 102  97 102 101 105  94 101 100 105  94 105  92 110  92  98 108  97  96 107 100 100  99  92 102 107  98  92 110 101  98 100  94 106 102  89 109  99 110  92  91 108  95  96 103 106  96  99 105 102  98 101 109  85 106  98 103 100  99  95 105  94 107  99 101  93 105  94 107  93 106 100 100 100  98 103 100  92 106 101  99  93 105 101 101 100 100  99 100 100  89 104 107 100  99  92  99 111  95 105  99  98 101  99 100 101 101 100  99  91 102 108  99 100 101 100  98 102  90 104 107 100  99  94 107 100  99 100 100 100  91 104 106 101  99 100 100  99 101  99 101  99  95 106 100  93 107 100 100  99 101  98 102  99  94  97  99 110  91 109  90 110  93 106 102 100  93  98 108  95 105  99  91 102 102 108  99  91 107  96  97 108 100  94 106  90 103 106 100 100 100 100  99  92 109 100 101  99  99 101  92 107  95  98  96 106 106  95 104 100 101 100 100  99 101  92 108 100  92 108  99 101 100 100  99  94 107  95 106 100  99 101  99 101  91 108 100 100  94 133  72  93 107 100 100 100  92 103 106  91 108 101 100  94  99 106  94 107 100  92 108 100  99 101 100 100  99 101 100  91 104 106  99 101  98  90 111 100  99  93  96 102 100  96 105 107 100 100  94 106  99  96  95 110  87 113  94  98 109  99  91 110  95 106 100  99  93 100 108 100 100  98  93 108 102  93  97 110  94 106  92 108  91 101  98 100  96 107 107  90 104  94 106  99  97 100 102 101  99 119  75 106  99  98  99  99 112  93  96 105 100  95 104 102  96 101  99 102 108  91 103 100  99 101  99  99 100 101 101  96  98  99 102  99  97  99 101 104 106  99  99  97  99 106  92 105 104 100 100 101  99 100 100  93 108  96  98  98 106 101  96 105  99  97  99 104 101  92 108 100 100 100 100

What is weird is that the behaviour is exactly the same with sync-video 1 and sync-video 0 I’m wondering if Apple does not enforce the video syncing now on the apps and this causes synchronisation resonances with Panda sync system (and that could explain this jittering around 100)

I don’t have a machine with 10.13 installed so I can not check if this is present on older macOS.

eldee · May 2, 2020, 8:53am

Food for thought: I did my previous tests on my macbook using the retina screen. I tried with an external monitor (and closing the laptop to disable the retina screen), the animation started to show some stuttering! Not as dramatic as your video, but the movement was no longer smooth. The values when sometimes below 90 or above 110, which has a clear visual impact. The jittering is not constant, it comes in waves, for some time the values oscillate a lot then calm down again close to 100.

Weirdly again, the “sync-video” flag has no impact.

treamous · May 2, 2020, 5:52pm

Just dig some digging on this. I actually noticed both stuttering and tearing on my system running 10.15. Searching around a bit, I found this issue in LWJGL’s github repository, which contained a solution that ended up fixing the stuttering/tearing problem for me.

Since you’re on 10.13, that fix wouldn’t work. I do think that the root cause is similar, since we changed how we handle backing layers a bit when trying to make Panda work on 10.14. I need to get a 10.13 installation up and running so I can test this further.

eldee · May 2, 2020, 6:08pm

I’m wondering if for macOS <= 10.13 we could use instead the native vsync mechanism as it gets broken only from 10.14 onward ? Doing so will keep vsync working for all the supported versions of macOS

feuermurmel · May 2, 2020, 7:56pm

I also made the observation that sync-video generally has no influence on my machine. The only time it made a difference whas when I disabled vertical sync for the whole user session using Quartz Debug.

And cool that you could reproduce the issue! So now I’m sure that its not just my setup being weird.

At some point doing my experiments, I got the feeling that having the clock in the menu bar set to display seconds has an impact on this. There wasn’t a lot of consistency but I turned the clock on and off a few times and got the feeling that the clock made the stuttering more likely to get to a severe level (numbers ranging from 50 to 200). Maybe this helps provoke the issue. ¯\_(ツ)_/¯