HTMX SSE Swap Proposal

sse.md

HTMX SSE Improvements Doc

Overview

The current SSE implementation is geared towards triggering events in the DOM, requiring an HTTP request to be sent up to get new content:

  <div hx-sse="connect /event_stream">
    <div hx-get="/chatroom" hx-trigger="sse:chatter">
      ...
    </div>
  </div>

In this case, an SSE event named chatter would trigger a GET to the /chatroom URL.

This is a very limited use of the Server Sent Events API. In particular, there are two aspects of SSE that are ignored:

1. SSE events can have data (not just names) associated with them
2. SSE events do not need to have a name (i.e. they can be pure data-only messages)

I propose the following extensions to the current SSE functionality:

Swapping on Named Events

If an element wants to directly swap the data of a message into its body, I propose the following syntax:

  <div hx-sse="connect /event_stream">
    <div hx-sse="swap on chatter">
      ...
    </div>
  </div>

The swap on indicates that this element wants to listen for chatter messages and swap the body when they occur.

The hx-swap attribute can then be used to determine exactly how the swap is done, as with normal swaps.

Data-only Messages

In the case of a data-only message, the message type is message according to the SSE spec. Thus, the form swap on message should work the same as the named event implementation.

  <div hx-sse="connect /event_stream">
    <div hx-sse="swap on message">
      ...
    </div>
  </div>

Declaring and Using An SSE Endpoint On The Same Element

We should support both connecting and listening for an event (or events) in the same hx-sse declaration, by allowing comma separated values:

  <div hx-sse="connect /event_stream, swap on message">
     ...
  </div>

Summary

These two changes would be relatively modest, but I think would address the major shortcomings of the current SSE behavior without requiring a large code refactor. With these changes SSE is on equal, or perhaps greater footing than WebSockets as a way to achieve dynamic web UI within the declarative model.

Another nice aspect of this proposal is that the features are additive, which means that SSE improvments, which I very much wnat to see implemented, would not be a blocker to an htmx 1.0 release.

Links

1. MDN SSE Article - https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events
2. SSE Spec - https://html.spec.whatwg.org/multipage/server-sent-events.html
3. Initial SSE improvement PR - bigskysoftware/htmx#155
4. Second SSE improvement PR - bigskysoftware/htmx#185

Requesting feedback from @benpate @clarkevans @tomberek

Tom's Comment: The spec talks about SSE messages without an event field be defaulted to “message”. Would it be possible for “swap on message” to apply to those data-only messages? In that case there is less of an arbitrary distinction between Named and Data-Only. It also produces a smoother transition when iterating a design between no event-type and adding one.

QUESTION: Does a named event trigger onmessage as well? Probably. Need someone who uses SSE to verify

in our PoC: we simply set the default value of the event list to ["message"] and it works as-is with a addEventListener. In other words e.onmessage(...) is functionally the same as e.addEventListener("message",...) (thank: benpate for the test server, note that the documentation says to use ?eventType=Name1 when it should be ?types=Name

https://github.com/benpate/htmx/blob/pullrequest-serverSentEvents-redesign-withMultipleEvents/src/htmx.js#L905

Note: named events DO NOT trigger onmessage and in fact there is no way to express in Javascript (and therefore probably not required for us to either) "browser, listen to ALL message types, named or un-named, from this EventSource and perform this callback."

Perfect. So I think that this syntax seems like it should be complete then.

We'll need to implement the hx-sse="swap on X" functionality and then make it work as a comma separated value along side connect and we'll be good to go?

@benpate is reviewing the docs and is going to comment in a bit.

First of all, THIS WORKS GREAT, AND WE SHOULD PROBABLY GO FORWARD WITH IT EXACTLY AS WRITTEN. It does a good job of adding a new feature to the library without removing anything that already works. The "on" keyword seems superfluous to the developer side of me, but it's small and it reads like English (which is nice) so let's just rock and roll.

Multiple Event Names

I think your spec would automatically handle this, but just to put it out there, we could support swapping in the content on multiple event names by having multiple swap on statements. For example: hx-sse="connect /my-url, swap on message, swap on CustomEventName, swap on AnotherEventName"

Implementation Notes

The proof of concept included several parts

1. parsing the actual syntax in the hx- attributes
2. managing SSEs and event listeners in a connection pool
3. swapping content into DIVs.

#1 is simple and would have to be done no matter what.
#2 may not make sense now and should at least be left aside until later.
#3 was harder (for me) than I thought it would be, and was the reason behind our doSwap function that largely duplicates the existing logic in the issueAjaxRequest function. This part may be tricky, and it will be good to get @1cg 's expert eyes on it so that we minimize duplication and don't break any existing work.

One More Thing

This is probably a discussion for post-1.0, but I'll include it now just because we have some wiggle room with the syntax. The swap on syntax seems very "Hyperscript-y" to me and makes me think of future versions of both libraries that are more tightly intertwined.

Would there be any reason to use Hyperscript to accomplish this feature, or to use syntax that mirrors Hyperscript more closely? Depending on the solution, it might be a nice reason to use both libraries together. For example:

<!-- Hyperscript plug-in -->
<div hx-sse="connect /my-url" _="on sse:message swap it">

<!-- Hyperscript-like syntax in HTMX -->
<div hx-sse="connect /my-url, on message swapContent, on otherMessage call externalFunction(event)">

<!-- Or with semi-colons, because they make me happy-->
<div hx-sse="connect /my-url; on message swapContent; on otherMessage call externalFunction(event)">

Again, this is probably a discussion for another day, but I'm starting to think that HTMX and Hyperscript will have enough overlapping use cases that they'll eventually need to become a single intertwined library.

How Can I Help?

I think my two PR's show that this is possible, but they also show me ravaging HTMX's internals to do it. I'm happy to take a first pass at this, but my understanding of the complexities inside HTMX is still lacking. @1cg, if you can point me in a direction I'm happy to run as far as I can, but I don't want to get in your way while I stumble forward.

Question About Commas in HX-SSE

I've made a new branch that looks promising. Two of the tests are working well, but I've run into my first potential "gotcha."

My test server allows URLs that include commas, like this: http://sseplaceholder.openfollow.org/posts.html?types=Event1,Event2,Event3,Event4.

This is a pretty common format, and will break our comma-separated syntax when I try to do something like this hx-sse="http://sseplaceholder.openfollow.org/posts.html?types=Event1,Event2,Event3,Event4, swap on Event1".

We'll have a similar problem if we switch to semicolons or any other non-whitespace delimiter. My initial thought for idiomatic-HTMX would be a syntax like this

<div hx-swap="connect my-url-with-commas swap:Event1,Event2" />
or possibly
<div hx-swap="connect my-url-with-commas swap Event1,Event2" />

I'm open to suggestions from management :)

Comment from @tomberek on the new proposed thread:

The extraction of the handling code from Ajax and applied to all of Ajax, sse, and WebSockets would be a good thing to keep/look into. It would ensure consistent behavior and be easier to maintain.

Right now for websockets, we have separate code that performs some of the logic of the main ajax swap:

https://github.com/bigskysoftware/htmx/blob/45f3909b9c6c76fe567e583d883f9589fd3693ba/src/htmx.js#L827

This is annoying, as it doesn't offer all of the functionality of the main swap function, and could get out of sync over time.

Unfortunately the AJAX swap has ajax specific handling in it, such as history and a few places where we pass the xhr out to other functions.

We should refactor this at the xhr.onload point to share logic between ajax, web sockets and SSE

thank you to @tomberek for pointing this out.

@benpate can you URI encode the commas?

http://sseplaceholder.openfollow.org/posts.html?types=Event1Event2%2CEvent3%2CEvent4, swap on Event1

For the implementation, I think that we could take the current SSE trigger listening code:

https://github.com/bigskysoftware/htmx/blob/a63d272441506d2d443725d0f855345078f2e98a/src/htmx.js#L898

And combine that with the WebSocket code:

https://github.com/bigskysoftware/htmx/blob/a63d272441506d2d443725d0f855345078f2e98a/src/htmx.js#L822

for processing the swap on syntax here:

https://github.com/bigskysoftware/htmx/blob/a63d272441506d2d443725d0f855345078f2e98a/src/htmx.js#L881

I'd like to leave pulling the ajax swap code out and reusing it for both SSE and Web Sockets as a stand alone change, so I can make sure I get it right.

Thanks for the guidance, @1cg. I’ll use your suggestion and reimplement this using your suggested functions as a base. It should come together quickly.

For the comma delimiters, URL encoding the commas can work, but depending on the server implementation, it’s not always practical. If I, as the developer, don’t necessarily control the URLs that are being passed in to my application, then fixing them after the fact could be very difficult. What do you think of the “space delimited” suggestion?

I've just committed an update to this branch. I think it meets the spec provided by @1cg. I'm trying to avoid cluttering the HTMX repository with PRs, so just let me know when I should post it into the HTMX repo.

All of the changes from the previous commit are in the processSwap function. It removes @tomberek 's doSwap function in favor of inlining code from the WebSockets implementation. These changes are consolidated into the commented section beginning on line 846. The comments are just placeholders for now, and should probably be removed once this branch lands on dev.

I'm still concerned that commas in URLs could break this in ways that are difficult to troubleshoot. Also, (though less important for now) is that this code doesn't implement some other features that are common in other parts of HTMX -- specifically hx-settle, which would be very useful for highlighting new content that is added into the DOM. Is there an easy way to just abstract hx-settle into something we could call from here?

awesome @benpate

Can you close out the other PRs and create a new one for this?

On the commas, the problem shouldn't be on the server side, right? If you use URI encoding it would be on the client only, the server should still see commas. It looks like commas are "reserved" and should probably be avoided:

https://stackoverflow.com/questions/198606/can-i-use-commas-in-a-url

You can leave the comments in and I'll try to extract the swap code out and get it in place for this change once I have it.

Done My apologies for the sloppy commit, I keep making updates on top of the wrong branch. I believe this PR should be correct

Regarding the comma, you should avoid using it as a delimiter before you encounter the space character (or the double quote, that can be safely percent-encoded)

From https://tools.ietf.org/html/rfc3986#section-2.2, "Percent-encoding a reserved character, or decoding a percent-encoded octet that corresponds to a reserved character, will change how the URI is interpreted by most applications. Thus, characters in the reserved set are protected from normalization and are therefore safe to be used by scheme-specific and producer-specific algorithms for delimiting data subcomponents within a URI".

You should not assume that a user of your library can simply percent-encode reserved characters, such as the comma. Based upon the URI specification, reserved characters are distinguished from their percent-encoded variant. If you percent-encode a reserved character, you could be changing the meaning of the URI. For example, a search API may choose to treat , in the value part of a key-value selector as an "OR", and if an actual comma is to be matched, it could be percent encoded. In this case, "?arg=foo,bar" is not the same thing as `?arg=foo%2cbar". In my presumed example, the former means "arg is either foo or bar", in the latter, it means "arg is foo,bar" -- quite different meaning. There are search APIs in the wild that have this exact treatment of the comma in a URL. Hence, neither server nor browser infrastructure should automatically percent-encode a reserved character. Equivalently, proxies should treat a percent-encoded character as being different for caching or other purposes. For our purposes, we should not assume that percent-encoding the comma is a transparent operation, it does create a different URI.

thanks @clarkevans

let me think about it more

we use commas in other places as well, so it'd be annoying to not use them here, and I don't want to have to implement a real lexer... but I'm sure we can figure something out.

we could go full hyperscript and split on and :)

connect https://blah.blah?whatever and swap on message

Sure. I think and is fine. Alternatively, one could use double quotes, but this requires you use single quotes for the attribute... <div hx-swap='connect "my-url-with-commas" swap Event1,Event2' /> -- it's ugly. "connect <https://blah.blah?whatever,comma>; swap on message" works, but, frankly, I like and ...

Hey all.. I agree on NOT making a full Lexer. There was yet another problem with PR #193, so I'm putting together a correct one with the existing syntax while we consider and discuss commas. It will be easy to switch it out for whatever final form is decided.

If we're voting, I still like hx-swap="connect my-url-with-commas swap:eventname" best, because it seems consistent with the way that other tags already work (like hx-swap)

I do want to thank you both for working though this. I don't expect to use htmx for a few more months, but it's really encouraging to see the thoughtful work going into it. So, I'm OK with "swap:eventname" as well.

Apparently, I may have figured out how to make a clean PR, because I just posted PR#194 and it hasn't blown up, yet. Please check my work to see if it looks good?

What about:

hx-sse="connect:https://my-url-with-commas swap:eventname"

so no whitespace between the connect and the URL. That would allow us to split on whitespace.

We could update the websockets to be similar.

Sure, verb:args works. Do your verbs, connect and swap always have a known number of arguments? In which case, "connect https://... swap eventname" work fine. You split on space. Then process the stack from the front. connect would eat the next item on the stack, https://my-url-with-commas and return; swap could then be triggered next, which could eat eventname, etc. There's also nothing wrong with and, this provides you the ability to have variable number of verb arguments (provided they arn't the and keyword).

@1cg - I like the syntax you’re suggesting. Although the double colon looks a little odd, in most cases a page would be calling back to a relative path on its own server, so it’s not an issue. We’d need to be a little careful with how we parse double colons (so we don’t drop content after the (https:”), but there are already examples of this in the library, so it should be simple enough to match.

I’ll let you think on it for a bit. Just say “go” and I’ll update this PR.

@clarkevans - yes. At this time, all of the keywords have only one argument. Think of them more as named parameters, or key/value pairs. We should be able to maintain this convention, if we just pick a new keyword for any additional arguments that we add in the future.

I have pushed up changes so that we now use whitespace as the declaration separator and the first colon as the argument delimiter:

<div hx-sse="connect:/foo swap:bar">...</div>

Same for both SSE and web sockets.

I also looked at combining the swap code for at least web sockets and SSE, but they acted differently enough that I didn't want to do it at this point. Web Sockets are a pure out of band situation, since there isn't a way to signal which messages you are interested in, so it just acts differently. And when I looked at pulling out the AJAX swap, there was a large number of parameters required.

I'll take another look at it when I have a bit more time to think.