Where precisely can Emacs take control of execution to deliver async process output to your filter-function?

emacs_async_yield_points.el

(defvar-local still-waiting nil
  "Whether we are still waiting for a chunk of process output to complete.")

(defun my-worker-1 (process)
  "Do some work and yield to PROCESS output."
  (do-some-sensitive-calculation)       
  (sit-for 1)                             ; A) output can interrupt me here
  (another-sensitive-buffer-manipulation) ; B) what about here?
  (my-worker-2 process))

(defun my-worker-2 (process)
  "Do some work and communicate with PROCESS."
  (heavy-calc1)         ; C) can these simple calc-only calls themselves be interrupted?
  (heavy-calc2)         ; D) how about *between* the two calls?  
  (process-send-string proc "I'm done!") ; E) presumably right here can be
  (my-worker-3)
  (while still-waiting
    (accept-process-output process 0.1 nil 1) ; F) obviously here
    (sit-for 0.02)))                          ; G) and here

(defun my-worker-3 ()
  "Do more work, entirely unrelated to the process."
  (do-something-entirely-unrelated-to-the-process)) ; H) but can this function or its internal calls be interrupted?

I don't understand what you mean by "less cooperative". I would say,
that actually Emacs is "more cooperative" since you can add explicit
yield points where processing can happen.

Maybe I should say "less intuitive".

In other async systems this is not possible, since all async processing only happens at the outer main
event loop.

But you can yield to that loop at will, to let other tasks run (async output arrive) at any time, in a predictable manner. Whereas Emacs may process async output at more unexpected times. Maybe it's a matter of learning to expect those and avoid the unwanted ones. But it's way less clear to me than e.g.:

do_something_heavy()
new_output = await some_output()
do_something_with(new_output) # <--- no output will arrive here
other_output = await other_output()
...

This sounds a bit weird. It should probably be sufficient if you
increase read-process-output-max? Did you try that? The default value is
far too small.

Mine is at 1048576 (on emacs-mac), but due to annoying MacOS limitations, the maximum tty line length is 1024 bytes, so we get 1024 byte blocks. Probably pipe would be much faster, but then you lose Ctrl-D and other types of tty processing. UPDATE: OMG pipes are so much faster. Blocks of output come in 65536 chunks @>1500kB/s without any special effort. Might be worth working around Ctrl-D and related. Do you know of other PTY advantages?

It may be that accept-process-output is indeed faster. But it should not be.

An example would be a %ls -lR in a directory producing about 20K of data, that takes 0.4s in a (non-emacs) terminal. For me, in a tight accept-process-output loop I can get all this data in (in 1024 byte chunks) in about 1sec. Using the outer event loop, it takes 45s-1min. I tried this same command in eshell, and i) it blocks the entire time waiting for the new prompt and ii) it takes about 6sec (though it does color-code files etc. so has more processing to do).

Update: with a pipe this takes a mere 0.7s letting the main event loop do the process polling! Why would I not just switch to a pipe?

The same holds for the GC thresholds. The Emacs developers didn't scale these values with hardware advancements such
that they are still stuck in the 90's.

I use gcmh-mode. I tend to think of GC settings as a user's discretion (maybe making recommendations). Do you actively alter it in your packages? Seems hard to do generically.

In Consult I use for example consult--async-refresh which throttles the refreshing.

Thanks, I'll take a look at this. Since most refreshing is "showing accumulating output in the buffer", I'm not sure how much throttling I should do. Do you throttle in terms of "display-altering updates per second" or total data rate of the updates? I think the former might make good sense to me. E.g.: give yourself a full second of slurping in data as fast as possible, then dump output to the shell, then repeat as needed.

This might dissatisfy some people, since in Python you can of course make arbitrary text output/input sequences at whatever rate you want, but probably better than just letting the main event loop trickle data in at 35-60kB/s (I mean, that's 1990's dialup speed!). Nobody is happy waiting 1min for a deep directory listing that takes <0.5s in a real shell.

But you can yield to that loop at will, to let other tasks run (async output arrive) at any time, in a predictable manner. Whereas Emacs may process async output at more unexpected times.

No, that's not true. It is exactly the same situation in Emacs. Note that async/await is just syntactic sugar.

do_something_heavy()
some_output(function(new_output) {
   do_something_with(new_output)
   other_output(function(other_output) {
      ...
   })
})

Vice versa you can write a macro in Emacs which performs the same async/await transformation for a block of code. Not sure how this package is called (aio?). If you write the exact same code in Emacs, "await" will also yield to the outer event loop "at will".

I am not sure where your misconception lies here. The complication in Emacs is only that you can locally start additional event loops, e.g., via sit-for. This is what makes Emacs somewhat "more cooperative" and maybe "less intuitive".

Mine is at 1048576, but due to annoying MacOS limitations, the maximum tty line length is 1024 bytes, so we get 1024 byte blocks.

Hmm, I am not sure but maybe you could disable line buffering.

I use gcmh-mode. I tend to think of GC settings as a user's discretion (maybe making recommendations). Do you actively alter it in your packages? Seems hard to do generically.

I am not a fan of gcmh-mode since it can lead to very long pauses. I use fairly conservative settings, but my threshold is still maybe 10x larger than the Emacs default. I don't recommend adjusting the gc threshold in packages. You can do it locally if you want to optimize for throughput. I do that at a few places in Consult, where a lot of allocations take place.

Thanks, I'll take a look at this. Since most refreshing is "showing accumulating output in the buffer", I'm not sure how much throttling I should do.

In Consult I use rather large delays, 0.1s maybe? We wouldn't win much by refreshing more often. The display would start to flicker and load would go up. I recommend to experiment a bit.

One more crazy idea: if I do switch to pipe interaction with the iPython process, I could also send all "hidden output" that happens behind the scenes (completion data, documentation, etc.) to stderr, attaching a separate filter-function to that. This would simplify parsing that data. I need to do some experiments to see whether this is actually faster (e.g. can iPython fill stdout and stderr separately faster than stuffing all output down one pipe). Do you have any experience with using both stdout and stderr to communicate with high-volume processes?

But you can yield to that loop at will, to let other tasks run (async output arrive) at any time, in a predictable manner. Whereas Emacs may process async output at more unexpected times.

No, that's not true. It is exactly the same situation in Emacs. Note that async/await is just syntactic sugar.

do_something_heavy()
some_output(function(new_output) {
   do_something_with(new_output)
   other_output(function(other_output) {
      ...
   })
})

Right. But in emacs, do_something_with(new_output) may itself lead to code-paths which produce process output. While this is true in Python/JS, it seems (based on admittedly limited experience) way less common. Maybe "async hygiene" is the right idea.

Vice versa you can write a macro in Emacs which performs the same async/await transformation for a block of code. Not sure how this package is called (aio?). If you write the exact same code in Emacs, "await" will also yield to the outer event loop "at will".

I am not sure where your misconception lies here. The complication in Emacs is only that you can locally start additional event loops, e.g., via sit-for. This is what makes Emacs somewhat "more cooperative" and maybe "less intuitive".

This exactly. And that various emacs internals effectively call "await process_output()" without regard for whether you are prepared to accept it at that moment. But if the list of such internals is predictable, you can work around it (and, as you say, use it to your advantage).

And that various emacs internals effectively call "await process_output()" without regard for whether you are prepared to accept it at that moment.

That's not what they do. They start a new local event loop, they are blocking. But yes, if one ensures that no such blocking functions are called, everything should be predictable.