Where precisely can Emacs take control of execution to deliver async process output to your filter-function?

emacs_async_yield_points.el

(defvar-local still-waiting nil
  "Whether we are still waiting for a chunk of process output to complete.")

(defun my-worker-1 (process)
  "Do some work and yield to PROCESS output."
  (do-some-sensitive-calculation)       
  (sit-for 1)                             ; A) output can interrupt me here
  (another-sensitive-buffer-manipulation) ; B) what about here?
  (my-worker-2 process))

(defun my-worker-2 (process)
  "Do some work and communicate with PROCESS."
  (heavy-calc1)         ; C) can these simple calc-only calls themselves be interrupted?
  (heavy-calc2)         ; D) how about *between* the two calls?  
  (process-send-string proc "I'm done!") ; E) presumably right here can be
  (my-worker-3)
  (while still-waiting
    (accept-process-output process 0.1 nil 1) ; F) obviously here
    (sit-for 0.02)))                          ; G) and here

(defun my-worker-3 ()
  "Do more work, entirely unrelated to the process."
  (do-something-entirely-unrelated-to-the-process)) ; H) but can this function or its internal calls be interrupted?

In Consult I use for example consult--async-refresh which throttles the refreshing.

Thanks, I'll take a look at this. Since most refreshing is "showing accumulating output in the buffer", I'm not sure how much throttling I should do. Do you throttle in terms of "display-altering updates per second" or total data rate of the updates? I think the former might make good sense to me. E.g.: give yourself a full second of slurping in data as fast as possible, then dump output to the shell, then repeat as needed.

This might dissatisfy some people, since in Python you can of course make arbitrary text output/input sequences at whatever rate you want, but probably better than just letting the main event loop trickle data in at 35-60kB/s (I mean, that's 1990's dialup speed!). Nobody is happy waiting 1min for a deep directory listing that takes <0.5s in a real shell.

But you can yield to that loop at will, to let other tasks run (async output arrive) at any time, in a predictable manner. Whereas Emacs may process async output at more unexpected times.

No, that's not true. It is exactly the same situation in Emacs. Note that async/await is just syntactic sugar.

do_something_heavy()
some_output(function(new_output) {
   do_something_with(new_output)
   other_output(function(other_output) {
      ...
   })
})

Vice versa you can write a macro in Emacs which performs the same async/await transformation for a block of code. Not sure how this package is called (aio?). If you write the exact same code in Emacs, "await" will also yield to the outer event loop "at will".

I am not sure where your misconception lies here. The complication in Emacs is only that you can locally start additional event loops, e.g., via sit-for. This is what makes Emacs somewhat "more cooperative" and maybe "less intuitive".

Mine is at 1048576, but due to annoying MacOS limitations, the maximum tty line length is 1024 bytes, so we get 1024 byte blocks.

Hmm, I am not sure but maybe you could disable line buffering.

I use gcmh-mode. I tend to think of GC settings as a user's discretion (maybe making recommendations). Do you actively alter it in your packages? Seems hard to do generically.

I am not a fan of gcmh-mode since it can lead to very long pauses. I use fairly conservative settings, but my threshold is still maybe 10x larger than the Emacs default. I don't recommend adjusting the gc threshold in packages. You can do it locally if you want to optimize for throughput. I do that at a few places in Consult, where a lot of allocations take place.

Thanks, I'll take a look at this. Since most refreshing is "showing accumulating output in the buffer", I'm not sure how much throttling I should do.

In Consult I use rather large delays, 0.1s maybe? We wouldn't win much by refreshing more often. The display would start to flicker and load would go up. I recommend to experiment a bit.

One more crazy idea: if I do switch to pipe interaction with the iPython process, I could also send all "hidden output" that happens behind the scenes (completion data, documentation, etc.) to stderr, attaching a separate filter-function to that. This would simplify parsing that data. I need to do some experiments to see whether this is actually faster (e.g. can iPython fill stdout and stderr separately faster than stuffing all output down one pipe). Do you have any experience with using both stdout and stderr to communicate with high-volume processes?

But you can yield to that loop at will, to let other tasks run (async output arrive) at any time, in a predictable manner. Whereas Emacs may process async output at more unexpected times.

No, that's not true. It is exactly the same situation in Emacs. Note that async/await is just syntactic sugar.

do_something_heavy()
some_output(function(new_output) {
   do_something_with(new_output)
   other_output(function(other_output) {
      ...
   })
})

Right. But in emacs, do_something_with(new_output) may itself lead to code-paths which produce process output. While this is true in Python/JS, it seems (based on admittedly limited experience) way less common. Maybe "async hygiene" is the right idea.

Vice versa you can write a macro in Emacs which performs the same async/await transformation for a block of code. Not sure how this package is called (aio?). If you write the exact same code in Emacs, "await" will also yield to the outer event loop "at will".

I am not sure where your misconception lies here. The complication in Emacs is only that you can locally start additional event loops, e.g., via sit-for. This is what makes Emacs somewhat "more cooperative" and maybe "less intuitive".

This exactly. And that various emacs internals effectively call "await process_output()" without regard for whether you are prepared to accept it at that moment. But if the list of such internals is predictable, you can work around it (and, as you say, use it to your advantage).

And that various emacs internals effectively call "await process_output()" without regard for whether you are prepared to accept it at that moment.

That's not what they do. They start a new local event loop, they are blocking. But yes, if one ensures that no such blocking functions are called, everything should be predictable.