Typed this up to get a better handle on what might be necessary in an auto-dependency management thing for Arch, while bloating Arch.Core as little as humanly possible.
It's a first draft, so don't take it as gospel; feel free to change things, or to reject it entirely!
My biggest priority here was to separate things into its own assembly, so that it wouldn't hugely change Arch.Core or the existing bits of Arch.Extended.
To attach a JobScheduler, call World.AttachScheduler(myScheduler)
To do a scheduled query...
JobHandle World.ScheduledQuery(in QueryDescription queryDescription, ForEach forEntity, JobHandle handle? = null); // Scheduled query
JobHandle World.ParallelQuery(in QueryDescription queryDescription, ForEach forEntity, JobHandle handle? = null, batchSize? = null); // Scheduled and parallel query
To do an inlined scheduled query...
JobHandle World.InlineScheduledQuery<T>(in QueryDescription queryDescription, JobHandle handle? = null); // Scheduled query
JobHandle World.InlineParallelQuery<T>(in QueryDescription queryDescription, JobHandle handle? = null, batchSize? = null); // Scheduled and parallel query
To actually run the handles, once you've scheduled them all:
myScheduler.Flush()
Or:
World.Scheduler.Flush();
To wait on the returned handles, just call handle.Complete()
.
If the user isn't using Arch.Scheduled, they must take explicit care to...
- Never make structural changes on a different thread
- Never make structural changes on the main thread while other queries might be running
- Never make non-atomic modifications to components at the same time, or read from a component while a non-atomic operation is currently happening
- Always
Flush()
the scheduler andComplete()
the handles when necessary
Implementation notes:
- We should benchmark a reasonable default for batchSize based on archetype/L1 cache/etc, but allow overriding, since we don't know how much work is being done in the user's query.
- If we want to be super clear that the parallel query is also scheduled, we could do
ScheduledParallelQuery
and[Obsolete]
the old one. - Arch.Core will need some additional hooks for
ScheduledWorld
(see below), but nothing impacting the public API
Arch.Scheduled is a new piece of Arch.Extended that lets you write parallel queries in a way that automatically tracks dependencies.
ScheduledWorld
leverages the already-present QueryDescription
to track dependencies. QueryDescription
(see notes) has several new options...
queryDescription
// Provides the ability to make structural changes to the query. Available for non-scheduled queries only, and causes a global sync point.
.WithStructuralChanges()
// Marks the provided components as read-only for the scheduling system, if they exist in the query. This is not validated in any way on Arch.Core queries!
.WithRead<T0, T1...>()
// Marks the provided components as writeable, if they exist in the query. Arch.Core queries use `Write` by default, unless they are otherwise specified as `WithRead`.
.WithWrite<T0, T1...>();
Then, when you schedule a query with any of the regular Arch.Core query scheduling methods (ScheduledQuery
etc), the ScheduledWorld
overrides set up a dependency network. It is no longer necessary to provide a JobHandle
to track dependencies; it automatically creates a graph based on the rule that multiple queries can read components simultaneously, but if a query writes a component, it has to come before future readers or writers.
The normal main-thread Query
methods now prevent structural changes by default, Flush()
the scheduler, and block the main thread until any scheduled dependencies have been completed.
Structural change methods, when run outside a query, will each Flush()
the scheduler and block the main thread until all scheduled queries are completed.
At the end of a frame (or whatever desired period before operations begin repeating themselves), call ScheduledWorld.CompleteAllDependencies()
. This will Flush()
any un-flushed jobs and Complete()
each handle scheduled so far.
Nested queries need some special consideration. They must be treated as an extension of their parent query, or terrible things happen. Imagine if you run a query that has its parent as a dependency -- you get deadlocked!
As a result, ScheduledWorld
treats nested queries as an extension of their parent query. They can only run if the parent QueryDescription
is the same or larger in scope to theirs. For example, if one query writes Alpha
and reads Beta
and Gamma
, it can only run child queries that write (or read) Alpha
, and read Beta
and Gamma
.
This special case is the main use-case for QueryDescription.WithWrite<...>()
: It allows specifying writable components that might not be present on the parent query's entities, but are necessary for child queries.
ScheduledWorld
introduces several new restrictions for dealing with queries. The following are explicitly prohibited:
- Not allowed: Scheduling other queries from within any query
- Not allowed: Running main-thread queries within another query, unless their
QueryDescription
has compatible dependency restrictions with the enclosing query- If the parent is scheduled, the children will run synchronously on the thread that is running the parent.
- (How do we validate this implementation-wise? not sure.)
- Not allowed: Making structural changes in a different thread, or without
WithStructuralChanges()
on theQueryDescription
- I.e. main-thread queries must specifically opt-in to structural changes.
- Use an ECB if you want to delay the sync point until later.
Maybe you want to run a custom scheduled job, before or alongside your query. You can override the default dependency behavior by providing a JobHandle
to the query, just like before. However, to ensure your query can work with automatic component dependencies, you need to grab this handle with your Entity's queryDescription
and use it somewhere in your dependency chain:
JobHandle GetDependency(in QueryDescription description)
For example, if you wanted to run a custom InitArrayJob
job prior to running a query to fill the array:
var customHandle = world.Scheduler.Schedule(new InitArrayJob());
var queryDescription = new QueryDescription(...);
var dependency = world.Scheduler.CombineDependencies(new[] { customHandle, scheduledWorld.GetDependency(queryDescription) });
scheduledWorld.ScheduledQuery(queryDescription, (...) => { ... }, dependency);
If you don't use the handle, your query might run at the same time as a conflicting query, so be careful when overriding the default behavior!
Implementation notes:
- For the
QueryDescription
API, it would require the ability to extendQueryDescription
somehow, but I'd like to avoid bloating Arch.Core. I'm not sure how.- One option is to add the methods in Arch.Core but make them internal, and use
InternalsVisibleTo
to make extension methods for them in Arch.Scheduled.- That might not be possible though -- does InternalsVisibleTo work across Nuget packages? I've never tried. If this works it's my favourite solution for sure.
- Another option is to create a custom
QueryDescription
and force the user to use separate overrides onScheduledWorld
. That seems a bit confusing, though. - The last option is to have the API present in Arch.Core, but without any purpose. Not ideal at all.
- One option is to add the methods in Arch.Core but make them internal, and use
- For
ScheduledWorld
to overrideWorld
, there would need to be some manner of method abstraction, but we definitely can't usevirtual
methods if we want to keep things speedy.- One option is a hook system. Wherever we need a hook for Arch.Scheduled, add a special
protected Action?
onWorld
, that is normally null. So, the only overhead inArch.Core
is checking to see if that hook is null. Then,ScheduledWorld
actually sets those hooks:Action<QueryDescription>? BeforeQuery
, soScheduledWorld
canComplete()
any needed dependencies and disallow structural changes.Action<QueryDescription>? AfterQuery
, soScheduledWorld
can stop tracking nested queries and re-enable structural changes.Action<QueryDescription>? BeforeScheduledQuery
, soScheduledWorld
can do validation on a query before it has been scheduled (i.e. disallow structural changes)Action<QueryDescription, JobHandle>? AfterScheduledQuery
, soScheduledWorld
can track the produced handle in the dependency graphAction? BeforeStructuralChange
, soScheduledWorld
can prevent a non-specified structural change if within a query, and Complete() all dependencies outside of one.
- Another solution is to make
ScheduledWorld
a wrapper around a privateWorld
, mirroring each method as necessary, but that's a lot more work. It is, however, the most efficient solution: everything can be inlined!- Maybe there's a source-generator thing we could do, to automatically create wrappers? Seems hard, but it might be necessary, especially considering
Accessors.g.cs
generating a bunch ofWorld
methods. Not to mention the many manyWorld
extension methods in both Arch and Arch.Extended... yikes.
- Maybe there's a source-generator thing we could do, to automatically create wrappers? Seems hard, but it might be necessary, especially considering
- One option is a hook system. Wherever we need a hook for Arch.Scheduled, add a special
Tag a query with [Scheduled]
or [Parallel]
to tell it to schedule on execution instead of immediately running. It causes the generated query method to return a JobHandle
.
Use [WithStructuralChanges]
on a non-scheduled [Query]
to cause it to create a global sync-point when run, and allow structural changes from within the query.
in
vs. ref
on the query method parameters is equivalent to specifying WithRead()
or WithWrite()
on the QueryDescription
, and comes with strict validation on struct components! For class components, in
doesn't prevent you from modifying the class data, so make sure to either avoid writing an in
class, or make your classes concurrent.
However, if your source-generated query references nested queries, you can tag a query with [Read<T0, ...>]
and [Write<T0, ...>]
to allow the nested query to run on those components, without having the parent component require them.
Looking fine so far! ^^
The only "problem" is the number of overrides required to realize
ScheduledWorld
.Overriding all structural change methods and all queries to make them a sync-point is incredible much work. However, integrating it directly into arch is a no-go. I wonder if there's a more decoupled approach.
What if we just let the user control the sync points without any automation? It would be a bit more "dangerous", however this way we would not need to override any methods or wrap them. We could however provide an analyzer that checks the code to provide feedback to the user, like : "You forgot a sync point there, add it!" or "You create entities, u sure that there's a sync point?".
Arch.Scheduled could just act as a source generator to generate fitting code within systems. Or we could just implement it into the system source generator directly. Or just less automation in general for Arch.Scheduled? Is there an alternative to callbacks from Arch.Core (E.g. callbacks when structural changes happen)?
Its really very complex, there must be a "clean" way without any major drawbacks.