NewNRT.md

Overview

Here are my notes from 2009, sent to sc-users:

What is needed is a new NRT mode that

• sets up a normal OSC socket for scsynth
• so awaits OSC messages and replies with OSC messages (/n_go, /n_end, /done ..., /synced, /tr, ... )
• but advances in time as fast as possible but with input from sclang

To tell the server to keep running, there would be a special /nrt_advance command:

A Routine with 1.0.wait for example will send an /nrt_advance message to scsynth with the duration of 1.0 seconds. scsynth continues to calculate audio output for at max 1.0 second. Two things could happen:

• all playing synths are still busy, no return messages are generated -> in this case, the Routine will continue with the next iteration

• a message is generated in the server -> e.g. scsynth sends an /n_end message to sclang (along with the correct logical time stamp).

Hypothetical example:

~r1 = { var synth = Synth(\default, [\freq, 333, \pan, -1, \amp, 0.5]); exprand(1.0, 10.0).wait; synth.release }.fork(NRTClock);
~r2 = { var synth = Synth(\default, [\freq, 444, \pan,  1, \amp, 0.5]); exprand(1.0, 10.0).wait; synth.release }.fork(NRTClock);

Let's say routine 1 picks a waiting time of 4.9, and routine 2 picks 1.3 seconds. At the same logical time and hence with the same bundle time, four messages will arrive at the server:

'#bundle', 0.0, [ '/s_new', \default, ... ]
'#bundle', 0.0, [ '/s_new', \default, ... ]
'#bundle', 0.0, [ '/nrt_advance', 4.9 ]
'#bundle', 0.0, [ '/nrt_advance', 1.3 ]

The next interruption for the server is at the minimum of these two advances, so 1.3 seconds. scsynth calculates an audio block corresponding to 1.3 seconds, then pauses again, and sends /nrt_advanced message back to sclang, so that sclang advances logical time at +1.3 seconds, which means that ~r1 will reissue waiting for another 4.9 - 1.3 = 3.6 seconds (probably needing to result in another ['/nrt_advance', 3.6])... and ~r2 will resume with the synth.release statement.

Thus all the threads will be properly interleaved.

Now, ['nrt_advance', 4.9] might arrive before ['nrt_advance', 1.3] so scsynth wouldn't know it needs to wait for more bundles at time 0.0. However, assuming that NRTClock uses one single PriorityQueue that handles the current suspended routines (as I guess SystemClock and AppClock do), the scheduler will of course know what is the next minimum advance time when all routines are suspended. So to correct the approach, <aNumber>.wait will not send the 'nrt_advance', but rather the scheduler as soon as it is idle, will look at the next item from the priorityqueue and send the corresponding advance message to scsynth, finally pop the item from the queue when scsynth reports back that it advanced.

Old threads:

• http://article.gmane.org/gmane.comp.audio.supercollider.devel/27755

To clarify server replies: The server must stop processing after it sends out a reply message such as n_end, until the client acklowledges it by sending another nrt_advance. The server will either put that n_end in a bundle or send a message before with the current newly advanced logical time (/nrt_advanced, 0 or 64 ).

• we could now decide whether that /n_end has logical time 0.0 or 0.0 + blockSize? also this is not so relevant in this moment, and since noone responds to /n_end in the hypothetical example, i will just skip it.
• the client will update each clock by the advanced amount
• so the client receives /nrt_advanced (let's say with a value of 0) and /n_end, but there aren't any responses. it thus has to go still 44100 (or 44100-64) frames till the next re-scheduling of the routine. hence it sends a-new an /nrt_advance, 44100
• the server then continues calculating audio-blocks, and let's say within these 44100 frames, a Dust produces a trigger. Let's say this trigger is created after 10000 frames.
• the server will send a bundle with time 10000 or a separate message /nrt_advanced, 10000 plus the corresponding /tr message.
• the client will receive this and update its NRTClock by 10000 frames (the routine is still pending). the osc-path-responder however will process the /tr message and fork a new routine (logical time 10000), hence sending a bundle with time 10000 and the new s-new command.
• and so forth. eventually the server has advanced the full 44100 frames that the client originally requested as result of a 1.wait. hence, scsynth will reply with another /nrt_advanced. let's say, there was only one Dust spike in that period. So scsynth kept processing another 34100 frames and then sends out the /nrt_advanced, 34100. this will in turn resume the main routine

A lot of details that need care, in particular

• the order in which concurrent things are processed (e.g. an /n_go happening at the same logical time as the /s_new)
• whether we use incremental times (advance-by , advanced-by) or absolute times (advance-to, advance-to). intuitively I would go for the absolute times, but we will run into a problem of integer numbers in sclang being only 32bit. (well we will run into the same problem with accumulating increments)

A refinement of how the message sending process could work:

Server-Commands:

/nrt_advance
    long - number of sample frames to advance

replies to the client with either a single message

/nrt_advanced
    long - the actual number of frames advanced

or a bundle beginning with an /nrt_advanced message.

the number of frames advanced may be smaller than requested, because within this time window, other reply messages have been generated, such as /n_end, /tr, etc. the bundle will be sent with all reply messages occuring generated after the given amount of advanced frames. the client thus will

• update the logical time of the corresponding NRTClock
• dispatch all the other messages in the bundle (e.g. /n_end might go into a NodeWatcher, /tr might go into an OSCresponderNode, ...)
• the dispatch-hooks might generate new OSC messages. it is thus suggested that the dispatch happens within the NRTClock (thisThread.clock is the NRTClock), and that all messages generated within that clock are queued in a bundle (this could be easily done with an implicit makeBundle)
• after dispatch, look into the queue of the NRTClock-scheduler, find the next item and calculate the new maxmimum number of frames to advance. it will add the appropriate /nrt_advance to the new NRT-bundle and send that out. note that the bundle-time will be "immediately".

I think it makes it easier if the server bundles the replies happening at the same time, and sends that whole bundle with an /nrt_advanced prepended. Alternatively, we could send out the /nrt_advanced as a separate message, then the reply messages one after another as plain messages, and finally an /nrt_idle message to terminate that action. so

[ bundle, <now>,
    [ /nrt_advanced, 10000 ],
    [ /tr, 1000, 0, 0.0 ]
]

versus

[ /nrt_advanced, 10000 ],
[ /tr, 1000, 0, 0.0 ]
[ /nrt_idle ]

i think the former is smarter, but the latter maybe simpler to implement in sclang which i think currently simply unpacks bundles and forgets about them.

and the client (result of dispatching process / makeBundle):

[ bundle, <now>,
    [ /s_new, "test2", 1001, 0, 1, "freq", 666.0 ]
    [ /nrt_advance, 34100 ]
]

A work-around for int32 limitation of SC's OSC interface could be to split sample frames into lower and upper words:

/nrt_advance
    int - number of sample frames to advance (upper 32bit word)
    int - number of sample frames to advance (lower 32bit word)

and

/nrt_advanced
    int - number of sample frames advanced (upper 32bit word)
    int - number of sample frames advanced (lower 32bit word)

Call Stack

In target server/scsynth/scsynth:

• scsynth_main.cpp main entry. arguments check. free switches included 'f' and 'F' (perhaps oFFline?).

• -N for old NRT aborts for NO_LIBSNDFILE. Otherwise sets options.mRealTime = false.

• options.mSharedMemoryID becomes UDP or TCP port in RT, otherwise zero

• World_New > World_OpenUDP | World_OpenTCP > World_WaitForQuit

• SC_World.cpp

• InterfaceTable_Init - this is a look up table? for OSC commands

• initializeScheduler - I think this calls into a platform specific audio driver, e.g. SC_PortAudio, SC_Jack...

• for SC_Jack.cpp, this is empty, for SC_CoreAudio.cpp, this invokes syncOSCOffsetWithTimeOfDay and others

• thus, this call would have to be skipped

Then there is two structures World (public API, accessible by plugins) and HiddenWorld (private). In World, most relevant would be mRealtime, mRunning and mDriverLock probably. SC_HiddenWorld.h contains structs for a few OSC (?) messages. It has reusable instances ("Fifos") of these. Also stores a mClientIDdict, mAudioDriver, mNRTInputFile, mNRTOutputFile (probably we would reuse the latter two).

Then it initialised the audio driver if in RT, and calls scsynth::startAsioThread() (creates some thread -- for what? processing OSC?)

Setup() and Start() are called, the class def of SC_AudioDriver is hidden in SC_CoreAudio.h. Where DriverSetup is called for Jack, I don't know. It also doesn't define Setup and Start...

World Processing

In RT, this is DriverRun.

• bufFrames = mWorld->mBufLength is block size
• driver may have its own size aka hardware block size, e.g. Jack has numSamples = NumSamplesPerCallback()
• as a result there is an inner loop with numBufs = numSamples / bufFrames
• there is a continuously incremented mBufCounter
• first copy input source (e.g. jack input or NRT input file) to input buses

There is mScheduler, a priority queue with methods NextTime and Remove. So remove events whose next-time is less than or equal to current logic time. Two world fields are updated: mSampleOffset and mSubsampleOffset. There is also an ominous mOSCincrement which is translated to oscInc that determines the time window from "now" in which events are dequeued from the scheduler.

Events are of type SC_ScheduledEvent and implement a Perform method which obviously does all the processing, such as changing the node graph.

Then there is World_Run which probably runs the DSP loop, finally the output buses are copied to the driver's output buffers.

World_NonRealTimeSynthesis

Here oscInc is actually the block size (perhaps that's also the case in RT). Instead of a scheduler Remove and Perform on the event, a function PerformOSCBundle is called. This is implemented in SC_CoreAudio.cpp!