LilithSilver/draft_arch_scheduled.md

## draft_arch_scheduled.md

      
    Raw
  

              draft_arch_scheduled.md
            
          
    Typed this up to get a better handle on what might be necessary in an auto-dependency management thing for Arch, while bloating Arch.Core as little as humanly possible.
It's a first draft, so don't take it as gospel; feel free to change things, or to reject it entirely!
My biggest priority here was to separate things into its own assembly, so that it wouldn't hugely change Arch.Core or the existing bits of Arch.Extended.
Arch.Core

To attach a JobScheduler, call World.AttachScheduler(myScheduler)
To do a scheduled query...
JobHandle World.ScheduledQuery(in QueryDescription queryDescription, ForEach forEntity, JobHandle handle? = null); // Scheduled query
JobHandle World.ParallelQuery(in QueryDescription queryDescription, ForEach forEntity, JobHandle handle? = null, batchSize? = null); // Scheduled and parallel query
To do an inlined scheduled query...
JobHandle World.InlineScheduledQuery<T>(in QueryDescription queryDescription, JobHandle handle? = null); // Scheduled query
JobHandle World.InlineParallelQuery<T>(in QueryDescription queryDescription, JobHandle handle? = null, batchSize? = null); // Scheduled and parallel query
To actually run the handles, once you've scheduled them all:
myScheduler.Flush()
Or:
World.Scheduler.Flush();
To wait on the returned handles, just call handle.Complete().
If the user isn't using Arch.Scheduled, they must take explicit care to...

Never make structural changes on a different thread
Never make structural changes on the main thread while other queries might be running
Never make non-atomic modifications to components at the same time, or read from a component while a non-atomic operation is currently happening
Always Flush() the scheduler and Complete() the handles when necessary

Implementation notes:

We should benchmark a reasonable default for batchSize based on archetype/L1 cache/etc, but allow overriding, since we don't know how much work is being done in the user's query.
If we want to be super clear that the parallel query is also scheduled, we could do ScheduledParallelQuery and [Obsolete] the old one.
Arch.Core will need some additional hooks for ScheduledWorld (see below), but nothing impacting the public API

Arch.Scheduled

Arch.Scheduled is a new piece of Arch.Extended that lets you write parallel queries in a way that automatically tracks dependencies.
ScheduledWorld leverages the already-present QueryDescription to track dependencies. QueryDescription (see notes) has several new options...
queryDescription
    // Provides the ability to make structural changes to the query. Available for non-scheduled queries only, and causes a global sync point.
    .WithStructuralChanges()
    // Marks the provided components as read-only for the scheduling system, if they exist in the query. This is not validated in any way on Arch.Core queries!
    .WithRead<T0, T1...>()
    // Marks the provided components as writeable, if they exist in the query. Arch.Core queries use `Write` by default, unless they are otherwise specified as `WithRead`.
    .WithWrite<T0, T1...>(); 
Then, when you schedule a query with any of the regular Arch.Core query scheduling methods (ScheduledQuery etc), the ScheduledWorld overrides set up a dependency network. It is no longer necessary to provide a JobHandle to track dependencies; it automatically creates a graph based on the rule that multiple queries can read components simultaneously, but if a query writes a component, it has to come before future readers or writers.
The normal main-thread Query methods now prevent structural changes by default, Flush() the scheduler, and block the main thread until any scheduled dependencies have been completed.
Structural change methods, when run outside a query, will each Flush() the scheduler and block the main thread until all scheduled queries are completed.
At the end of a frame (or whatever desired period before operations begin repeating themselves), call ScheduledWorld.CompleteAllDependencies(). This will Flush() any un-flushed jobs and Complete() each handle scheduled so far.
Nested queries

Nested queries need some special consideration. They must be treated as an extension of their parent query, or terrible things happen. Imagine if you run a query that has its parent as a dependency -- you get deadlocked!
As a result, ScheduledWorld treats nested queries as an extension of their parent query. They can only run if the parent QueryDescription is the same or larger in scope to theirs. For example, if one query writes Alpha and reads Beta and Gamma, it can only run child queries that write (or read) Alpha, and read Beta and Gamma.
This special case is the main use-case for QueryDescription.WithWrite<...>(): It allows specifying writable components that might not be present on the parent query's entities, but are necessary for child queries.
New error handling

ScheduledWorld introduces several new restrictions for dealing with queries. The following are explicitly prohibited:

Not allowed: Scheduling other queries from within any query
Not allowed: Running main-thread queries within another query, unless their QueryDescription has compatible dependency restrictions with the enclosing query

If the parent is scheduled, the children will run synchronously on the thread that is running the parent.
(How do we validate this implementation-wise? not sure.)


Not allowed: Making structural changes in a different thread, or without WithStructuralChanges() on the QueryDescription

I.e. main-thread queries must specifically opt-in to structural changes.
Use an ECB if you want to delay the sync point until later.


Hooking into the automatic system

Maybe you want to run a custom scheduled job, before or alongside your query. You can override the default dependency behavior by providing a JobHandle to the query, just like before. However, to ensure your query can work with automatic component dependencies, you need to grab this handle with your Entity's queryDescription and use it somewhere in your dependency chain:
JobHandle GetDependency(in QueryDescription description)
For example, if you wanted to run a custom InitArrayJob job prior to running a query to fill the array:
var customHandle = world.Scheduler.Schedule(new InitArrayJob());
var queryDescription = new QueryDescription(...);
var dependency = world.Scheduler.CombineDependencies(new[] { customHandle, scheduledWorld.GetDependency(queryDescription) });
scheduledWorld.ScheduledQuery(queryDescription, (...) => { ... }, dependency);
If you don't use the handle, your query might run at the same time as a conflicting query, so be careful when overriding the default behavior!
Implementation notes:

For the QueryDescription API, it would require the ability to extend QueryDescription somehow, but I'd like to avoid bloating Arch.Core. I'm not sure how.

One option is to add the methods in Arch.Core but make them internal, and use InternalsVisibleTo to make extension methods for them in Arch.Scheduled.

That might not be possible though -- does InternalsVisibleTo work across Nuget packages? I've never tried. If this works it's my favourite solution for sure.


Another option is to create a custom QueryDescription and force the user to use separate overrides on ScheduledWorld. That seems a bit confusing, though.
The last option is to have the API present in Arch.Core, but without any purpose. Not ideal at all.


For ScheduledWorld to override World, there would need to be some manner of method abstraction, but we definitely can't use virtual methods if we want to keep things speedy.

One option is a hook system. Wherever we need a hook for Arch.Scheduled, add a special protected Action? on World, that is normally null. So, the only overhead in Arch.Core is checking to see if that hook is null. Then, ScheduledWorld actually sets those hooks:

Action<QueryDescription>? BeforeQuery, so ScheduledWorld can Complete() any needed dependencies and disallow structural changes.
Action<QueryDescription>? AfterQuery, so ScheduledWorld can stop tracking nested queries and re-enable structural changes.
Action<QueryDescription>? BeforeScheduledQuery, so ScheduledWorld can do validation on a query before it has been scheduled (i.e. disallow structural changes)
Action<QueryDescription, JobHandle>? AfterScheduledQuery, so ScheduledWorld can track the produced handle in the dependency graph
Action? BeforeStructuralChange, so ScheduledWorld can prevent a non-specified structural change if within a query, and Complete() all dependencies outside of one.


Another solution is to make ScheduledWorld a wrapper around a private World, mirroring each method as necessary, but that's a lot more work. It is, however, the most efficient solution: everything can be inlined!

Maybe there's a source-generator thing we could do, to automatically create wrappers? Seems hard, but it might be necessary, especially considering Accessors.g.cs generating a bunch of World methods. Not to mention the many many World extension methods in both Arch and Arch.Extended... yikes.


Arch.System/Arch.System.SourceGenerator

Tag a query with [Scheduled] or [Parallel] to tell it to schedule on execution instead of immediately running. It causes the generated query method to return a JobHandle.
Use [WithStructuralChanges] on a non-scheduled [Query] to cause it to create a global sync-point when run, and allow structural changes from within the query.
in vs. ref on the query method parameters is equivalent to specifying WithRead() or WithWrite() on the QueryDescription, and comes with strict validation on struct components! For class components, in doesn't prevent you from modifying the class data, so make sure to either avoid writing an in class, or make your classes concurrent.
However, if your source-generated query references nested queries, you can tag a query with [Read<T0, ...>] and [Write<T0, ...>] to allow the nested query to run on those components, without having the parent component require them.