Non frame-sync background tasks

Scripting Issues
1

I have a bunch of background tasks I’m using to load, generate or retrieve geoms, textures, … on demand. These coroutine-based task are spawned when parts of objects come into view or when my custom LOD mechanism decides that a higher or lower resolution texture should be used.

I have observed that these tasks are actually starting to be executed on the next frame and not sooner. This means that even if all the data is available in memory, it will only on the next frame that this will be detected by the task and the result will be visible on the next next frame. This not optimum and in some cases cause visible glitches.

Setting the sort parameter of those spawned tasks to a higher or equal value than the current task does not help either, it just offset the problem as the result will not be seen by the main task when it runs again the next frame as those tasks will be scheduled after the main task.

Creating a custom task chain which has frameSync disabled or using timeslicePriority does not help, a task is run once per frame (and I don’t want to use threads as I need to interact with the engine)

I understand that with the default tasks it’s not possible to have a taskchain that runs endlessly in the background, but with the new coroutine based task I had hoped this would be possible somehow

Stray observation, if I set the sort parameter above 46 (garbageCollectStates) the task is only run on the next frame.

2

Hmm, yes, I can see how this can be annoying. I initially decided on this behaviour because I wanted to be sure to retain predictable scheduling behaviour, but I am happy to reconsider this.

There are some ways we could deal with this, depending on what you think you need:

  1. When waking up a task, we can check whether the task chain is currently processing a task with lower or equal sort value; if so, we schedule the task to continue during the current frame to be executed in the appropriate order, otherwise wait for the next frame.

  2. We could wake up the task in “pickup” mode as opposed to regular “cont” mode. This is a mode where the task manager continues to run tasks that returned DS_pickup at the end of a frame until a given frame budget has been exceeded.

  3. We could ignore sort values altogether, and have awoken tasks run “as soon as possible”.

My initial reaction is to be inclined towards option 1, which retains predictable scheduling, but depending on the timing it’ll still be a frame later. Number 2 seemed appealing at first but on the other hand there may not be much practical difference between having a task run at the end of the current frame vs the start of the next one. I am unsure that option 3 is desirable, because the order of tasks is usually important (eg. you might get bugs if you render objects that just spawned in without first running your AI / physics update). What do you think?

Apropos, I’ve been working on a new lighter-weight scheduler based around the idea of “jobs”, which run endlessly rather than on a sorted, frame-scheduled list. However, so far I’ve been envisioning this to be used in the C++ codebase, with jobs being distributed out randomly to different threads, so this would probably not be useful to expose to Python code. But I’m not ruling out closer integration with the task system in the future.

3

I believe that option 1 is what is currently implemented by the task manager ? If I add a task with the same or higher sort value than the current task, it is executed in the same frame after the current task (or after all the tasks with lower sort value). However I noticed that means the new task will always be scheduler after the task that created the new one.

Let me give an example, if a surface patch comes into view, it needs a geom object, a heightmap and a couple of surface textures (albedo, specular, and so on). The engine does not check if any of those data exists or should be loaded or calculated, it just create a new task that will take care of that and when the task is done, the engine knows it can add the patch into the scene (during that time a temporary patch with lower quality data is shown)
The heightmap is either in the cache or must be calculated on the GPU; if it’s in the cache the task has nothing to wait on (the geom is generated on the fly and the textures are also in a cache), that means the task could do all its job before the end of the frame and add the new patch into the scene to be shown on the next frame.
If the heightmap is not in the cache, the engine schedule its rendering on the GPU (if a render buffer is free in the pool) and the calculated heightmap will be available on the next frame.

If I set the sort value of the patch task to a lower value that the engine task, it means it won’t be scheduled before the end of the frame, only at the start of the next frame. But that also means the engine task will be notified of the result also during the next frame.

With option 1, I set the sort value to the same of higher value than the engine task, the patch task will be scheduled later during the same frame slot, but if it has to wait for GPU generation it will be paused until next frame. In that case, the engine task will be executed first and the patch task after, which means the generation result will not be visible to the engine task until the next frame. (It is not usually a problem, except when the camera is moving relatively fast, which mean those two frames old data are no longer useful)

I think the option 2 with a patch task having a lower sort value that the engine task could do the trick, If I understand correctly. That would mean the patch task will always run after being created within the same frame slot, eventually block if some data must be generated or loaded, then, once the data is available, run before the engine task.
Though I would made that mode optional and keep the default behaviour by default, otherwise this would cause havok in app that relies on strict task scheduling

Option 3 scares me too, and thinking about it, is of no use to me. I still need the task to be executed in the right order. Though it might be useful for some people to have such kind of rule free task chain in parallel to more strict task chains.

4

Okay, let’s see if I understand this correctly… your engine task schedules your patch task, which then runs after the engine task, possibly awaits something, but when it wakes up again you want it to be run before the next engine run? Which currently doesn’t work because Panda forces it to always occur in the same task order?

I guess… it is legal to change a task’s sort value while it is running, which would probably work if you did it right before awaiting… would doing so solve your problem?