Skip to content

Instantly share code, notes, and snippets.

@egmontkob
Last active March 17, 2024 13:54
Star You must be signed in to star a gist
Save egmontkob/eb114294efbcd5adb1944c9f3cb5feda to your computer and use it in GitHub Desktop.
Hyperlinks in Terminal Emulators
@PerBothner
Copy link

I think https://www.freedesktop.org/wiki/Specifications/ would be a good home.
Perhaps https://www.freedesktop.org/wiki/Specifications/hyperlinks-in-terminals.
Or https://www.freedesktop.org/wiki/Specifications/terminals/hyperlinks, if the site supports a specification hierarchy.
Someone needs to how get access to freedesktop.org and add this document, though it would probably have to be edited a bit according to freedesktop.org's standards.

I'd avoid https://gitlab.freedesktop.org/terminal-wg. Right now I can't find any of our old discussions, which is really weird. Beyond that, it had some fundmental problems. To start with, it separated specifications into "proposals", "accepted", and "rejected". The first problem is: Accepted by who? The other problem is the link changes depending on whether something is a proposal or accepted, which leads to dead links. There was an assumption that the goal was to reach consensus and then a specification would be accepted. However, that quickly turned out to be impossible: Endless discussions, because people had different goals and understandings. Some people were and are very opinionated - myself included. Discussion of an image protocol dragged on. (There was maybe some slight movement, but very little.) It also lacked leadership: I believe it was started by Gerard Nachman, but he lacked time to guide things or keep the site updated.

I think we may need to accept that there may be multiple competing protocols in an area. It's OK to write a specification that only one terminal implements. It's OK to write a specification that gets succeeded by something better - or is just more popular.

Going forward, I think the SRFI process has been somewhat successful. Anyone is free to propose a specification. Anyone is free to ignore it. A good editor (Arthur Glecker in the case of SRFI) keeps track of the status of specifications, and (as needed) nags people. (One could argue that the main problem with the SRFI process is that it has been too successsful, in the sense of too many specifications. Plus that it is used as an incubator for R7RS-large, which I think is becoming way too big.)

@egmontkob
Copy link
Author

@lmorg

we’ve given examples of where this isn’t the case and what steps need to be included with any hyperlink specs to make it safe

And then I updated the Security section according to your requests, to which you said on Mar 26, 2019 that "I really like the way you've written that. [...] Thank you"

But, you know what. I just realized I made a big mistake. This document should have never talked about security. It doesn't belong here. This document should only state how a certain escape sequence defines a URI for each character cell, and not say a single word about what to do with that information, leaving that up entirely to the implementation.

Teaching programmers how to open a URI (or rather, how to ask another application to open a URI) should not belong to this document. I made the mistake of beginning to go down this rabbit hole whereas it's obviously impossible to gather all the relevant knowledge here, then went further down due to your request, and even though at one point you were satisfied, years later you claim again that I didn't go down that hole deep enough. No, I shouldn't have even started to go down.

You’ve [...] claimed because other unrelated systems include the same concept that somehow terminals are magically safe without any forethought.

No, I claim that terminals are fine exactly because we transferred the very model, unchanged, that is already out there at many other places.

That was the very goal of this project.

And yes, we all know, that model has some shortcoming. Yes, CSRF attacks, phishing, whatnot do exist, we all know.

What I do have a problem with is you seem to claim that, for whatever reasons, terminals are conceptually different, terminals should not have adopted this very same model, but rather came up with something different (if at all), or I don't know.

If you have a problem with hyperlinking per se, if you have a problem with web pages, emails, PDFs, Markdown files etc. being able to contain hyperlinks, because of the possible CSRF, phishing, whatnot attacks that they make possible, then this one here is not the right forum to address that. Talk to the big players, not the tiny ones. Talk to W3C or whoever, tell them what ideas you have to improve the situation. Make the web adopt those, the smaller players (including terminals) will follow, and I am going to thank you.

Or you don't have a problem with these issues on those platforms, only in terminals? Hyperlinks on web pages are fine, hyperlinks in HTML emails are fine, hyperlinks in PDF documents are fine, hyperlinks in Word, Excel documents are fine, hyperlinks in Markdown files are fine, but the exact same hyperlinks in terminals are not?? You believe that, for whatever reason, terminals should enforce some stricter security model than other hyperlink-capable software?? It's utterly ridiculous. And yes, you do see it perfectly that I am not willing to waste my time continuing any discussion in this direction.

The goal of this feature was to adopt an existing, well-known, popular feature, an industry standard if you will, into the world of terminals (and so this is what we exactly did), not to come up with something different.

(Had we come up with something different, I surely hope that terminal emulators would have been heavily opposed to implementing that, and would have looked to adapt the standard way of hyperlinking instead. Or, for example, if this document said that terminal emulators must disable this feature by default, I surely do hope terminal emulators would disregard that.)

and it was just hosted on one persons Gist

Which is exactly one Gist more than the vast majority of terminal emulator features has.

as someone who works in this field, I can promise you that statement is not correct.

Okay, not that anyone cares, but this is probably objectively verifiable by anyone. How many features do terminals have? Depends on what you call a "feature" that makes sense to discuss on its own, but I'd say it's probably at least a hundred. As of 2.5 years ago when I stopped working on terminals, I can now recall 2 of them which had a GitHub Gist set up to talk about. Did I really miss so many more Gists??

It should not be a Gist.

I agree. It should be a static page, summarizing the bare minimum of the final design.

I'm genuinely a nice guy

So am I, believe or not. So where did it go wrong? I guess my level of patience and tolerance against reoccurring BS is much lower than yours.

and refusing to have a discussion about it.

We did discuss these issues, our opinions hardly got closer to each other, I don't think you've understood my arguments, and I refuse to discuss the same ridiculuous idea (namely that hyperlinking in terminals should adhere to stricter rules than hyperlinking anywhere else) over and over and over again.

But implementations are allowed to become stricter. They might present a 10 page long legal disclaimer that you have to accept before following that link. Or refuse to implement this feature, revert if they already did. Or whatever. But this doesn't belong here. Go ahead, raise your concerns at implementations, see what they think about it.

At least the others continuing on from you are building working groups

LOLOLOLOL. You have no idea who the creators of Terminal WG are, do you?

It all started with a private email, 4 years ago on this very day, sent to gnome-terminal's main developer, asking if he had any good idea where to host the specification of a forthcoming feature, having seen that GitHub Gist turned out to be a far-less-than-ideal choice for the hyperlink feature. This conversation eventually evolved to the creation of Terminal WG.

The sender of that email? You've guessed.

Now, unfortunately, it did not go well. Nowadays it's either completely dead, or I just managed to unsubscribe, I don't know. Turned out that inviting dozens of terminal emulator developers to design something together, without any hierarchy and leadership among the guys, is a terrible idea. The comment sections there are an utter disaster even compared to this one here.

That being said, would it be a good platform to host the community effort to track an existing feature's adaption? Maybe.

@lmorg
Copy link

lmorg commented Nov 15, 2022

Okay, not that anyone cares, but this is probably objectively verifiable by anyone. How many features do terminals have? Depends on what you call a "feature" that makes sense to discuss on its own, but I'd say it's probably at least a hundred. As of 2.5 years ago when I stopped working on terminals, I can now recall 2 of them which had a GitHub Gist set up to talk about. Did I really miss so many more Gists??

The ones I’ve come across weren’t hosted on GitHub. Often mailing lists or similarly “old school” tech.

anyways, I think we’ve reached an amicable impasse. I’d like us put this saga behind and hopefully the next time we (virtually) meet it will be far more joyous for both of us :)

good luck with whatever endeavours you’re currently working on

@Alhadis
Copy link

Alhadis commented Dec 19, 2022

@egmontkob Would you mind clarifying the license of this Gist? Currently, OSC8-Adoption lacks a license, and I'd prefer to release it under the terms chosen by the original author.

You've already sent me two different links. Please come to an unambiguous agreement which one I should add to the gist. Thanks!

Sorry, I somehow missed this reply.

I don't think @ssbarnea's org is related to this Gist (and by extension, OSC8-Adoption). It appears to be related to best and common practices, whereas this Gist merely tracks support for a particular feature amongst implementors. Moreover, the link provided by @ssbarnea points to an empty repository, so there's really nothing to come to an agreement over…

@MunifTanjim
Copy link

I just found out Tmux already supports it. It got merged half an year ago. tmux/tmux#3240

Just need to build tmux from source and add this in tmux config:

set -ga terminal-features "*:hyperlinks"

@dustymabe
Copy link

Just need to build tmux from source and...

And to be clear, the only reason you need to build it from source is because there hasn't been a new release yet so major distributions won't have the new feature. Right?

@MunifTanjim
Copy link

MunifTanjim commented Dec 27, 2022

And to be clear, the only reason you need to build it from source is because there hasn't been a new release yet so major distributions won't have the new feature. Right?

Yeah, it's not included in any released version yet. The last release was on 2022-Jun-9. This feature was merged on 2022-Jun-30.

@egmontkob
Copy link
Author

egmontkob commented Dec 30, 2022

@egmontkob Would you mind clarifying the license of this Gist? Currently, OSC8-Adoption lacks a license, and I'd prefer to release it under the terms chosen by the original author.

@Alhadis There's no explicit license. It's just a freaking documentation. Whatever the legal default is. I couldn't care less as long as people play fair (e.g. don't claim that they wrote it from scratch if obviously they didn't) – and if they don't, I don't have any means of enforcing any behavior anyway, regardless of license.

I'm not a lawyer and I'm not familiar with licenses that could be applied to design work, documentation, or list of software. I know open source loves to require everything to have a license. It's almost as if if I upvoted a comment on stackexchange that upvote would require a license :D I know I'm exaggerating, but you get the point. I find this "every piece of work anyone ever does has to have a license" quite ridiculous.

However...

I'd highly appreciate, in fact if you want me to add a link to your page then I insist, that your page doesn't repeat the specification of the feature; rather, it should solely be a list of supported software (as, by the way, the name "OSC8-Adoption" implies).

We've heard a few people claiming that this feature is fundamentally broken and should be revoked. You and your co-maintainer @ppwwyyxx have expressed that you won't delete / moderate any comments, i.e. leave room for these voices. I have no idea how you're planning to maintain that page, what if the specification part would receive changes, or (I don't expect this to happen, but cannot exclude the possibility) you change the specification more or less according to the request of those voices. Whether it's just some subtle wording changes, or fundamental ones that you make to the page, I'm afraid it wouldn't be clear anymore which of these two pages is the authentic one.

Of course it's possible that some day someone will come up with an extension to OSC 8 that will somehow get widespread. It's also possible that someone will come up with an extension that won't become popular. Independently from the popularity, an extension can be a good one or a bad one. However, because I'm not going to keep an eye on the developments, I wouldn't want to link to a page that potentially allows room for any of these to happen, thereby giving my implicit approval to some future change or a takeover of the specs. Whatever change some people might promote in the future should happen without my help, implicit or explicit.

Please turn that page into a list of OSC 8 supporting software, and not more. I'd be happy to link to that, then. For the specification itself, a link pointing to this page would highly be appreciated. If you or anyone else has an idea how to change OSC 8 itself or improve its documentation (apart from tracking its adoption), that should happen elsewhere.

I hope this all makes sense. Thanks for your understanding, and for the work you're putting into this!

Moreover, the link provided by @egmontkob

There must be some misunderstanding (or typo) here, that link wasn't provided by me.

@Alhadis
Copy link

Alhadis commented Jan 31, 2023

There's no explicit license.
[…]
I'd highly appreciate [that] your page doesn't repeat the specification of the feature; rather, it should solely be a list of supported software (as, by the way, the name "OSC8-Adoption" implies).

Right, done. Removing the specification text essentially renders my earlier question moot, as the remaining material constitutes derived and original work. I'm therefore going to license the repository under Creative Commons Zero (the same terms recommended by Wikipedia).

There must be some misunderstanding (or typo) here, that link wasn't provided by me.

You're right, it was a typo. I meant to tag @ssbarnea instead; I'll amend my earlier comment…

@Alhadis
Copy link

Alhadis commented Apr 25, 2023

Nudging @egmontkob.

@egmontkob
Copy link
Author

@Alhadis Apologies, this fell through the cracks. Done. Thanks!

@walles
Copy link

walles commented May 3, 2023

Regarding pagers, terminal hyperlinks are supported by moar since v1.14.0:

https://github.com/walles/moar

@Alhadis
Copy link

Alhadis commented May 6, 2023

@arvenil
Copy link

arvenil commented Jul 18, 2023

Is there any way to detect if terminal supports this feature?

@AnonymouX47
Copy link

AnonymouX47 commented Jul 18, 2023

@arvenil, it's answered in the document.

@alvaromuir
Copy link

alvaromuir commented Aug 1, 2023

for the life of me i can't get this to work when trying to replace a value with JQ.
I wouldv'e thought this was pretty straight forward:
echo '{"link":"\x1b]8;;https://google.com\x1b\\google.com\x1b]8;;\x1b]8;;\x1b\"}' | jq

@jamie-pate
Copy link

jamie-pate commented Sep 6, 2023

Is there any way to detect if terminal supports this feature?

No, and no standard way to disable it, so any program that adds these links ends up spamming escape characters all over your output (edit: if they are not 100% compatible with all forms of escape codes)

]8;id=274247;https://ansible-lint.readthedocs.io/rules/syntax-check/\syntax-check[specific]]8;;\: Invalid options for ansible.builtin.include_role: vars

https://gist.github.com/egmontkob/eb114294efbcd5adb1944c9f3cb5feda#backward-compatibility mentions that any terminal emulator that doesn't parse it correctly should consider this a bug since it doesn't properly follow ECMA-48

@AnonymouX47
Copy link

AnonymouX47 commented Sep 6, 2023

Hey @jamie-pate.

That should be considered a bug on the part of the program (I believe ansible-lint in your case) emitting the sequence, you should report the issue over there.

As Egmont explained, it's the duty of the program emitting the sequence to detect if its output stream is connected to a terminal device before emitting such sequences... The same applies to almost any other terminal control sequence, not just this.

Next time, kindly ask nicely about things you may not be well-informed about. 😃

EDIT: For the record, I saw your previous comment.

@egmontkob
Copy link
Author

Oh, and just for the fun of it (in response to a freshly deleted comment that's archived on web.archive.org)

feels like this whole thing put the horse before the cart

The last time I checked, that's where the horse belongs 😂

@jamie-pate
Copy link

https://gist.github.com/egmontkob/eb114294efbcd5adb1944c9f3cb5feda#backward-compatibility mentions that any terminal emulator that doesn't parse it correctly should consider this a bug since it doesn't properly follow ECMA-48

I was trying to be constructive, but to clarify, I mean that the terminal emulator is faulty if it parses colors and other ECMA-48 codes, but fails to properly deal with ]8

Trying to help people identify where to report the issues with various pieces of infrastructure that this is exposing.. e.g. concourse/concourse#4318

@mintty
Copy link

mintty commented Sep 6, 2023

I mean that the terminal emulator is faulty if it parses colors and other ECMA-48 codes, but fails to properly deal with ]8

Isn't that just what the quoted section is saying?

@jamie-pate
Copy link

jamie-pate commented Sep 6, 2023

I mean that the terminal emulator is faulty if it parses colors and other ECMA-48 codes, but fails to properly deal with ]8

Isn't that just what the quoted section is saying?

I was responding to this comment, which seems to be laying the blame on the emitter (Originating devices as per ECMA-48), not the consumer (Receiving devices as per ECMA-48):

Hey @jamie-pate.

That should be considered a bug on the part of the program (I believe ansible-lint in your case) emitting the sequence, you should report the issue over there.

My conclusion is that according to #backward-compatibility the Receiving device is 100% responsible for implementing the rest of ECMA-48 even if they just wanted fancy color support and implemented it by scanning through the wikipedia article on the subject. Therefore, if you have issues, you should report issues with the Receiving device implementer.

@AnonymouX47
Copy link

@jamie-pate Just to be clear, where exactly did you copy the following from?

]8;id=274247;https://ansible-lint.readthedocs.io/rules/syntax-check/\syntax-check[specific]]8;;\: Invalid options for ansible.builtin.include_role: vars

@jamie-pate
Copy link

jamie-pate commented Sep 6, 2023

The program generating the output is ansible-lint which creates output using the rich python library

The issue shows up when running inside concourse ci

@PerBothner
Copy link

Whether or not concourse-ci renders OSC 8 links as links is optional: nice but not required.
However, it should parse OSC escape sequences and ignore ones it doesn't handle. That is a pretty basic requirement for any kind of modern terminal emulator (or wrapper like screen/tmux) that claims to be more-or-less-xterm-compatible (which almost all do). If concourse-ci only claims to implement a minimal ansi/vt-NNN-style terminal, then ansible-lint/rich should not be emitting any OSC sequences.

@AnonymouX47
Copy link

AnonymouX47 commented Sep 6, 2023

The program generating the output is ansible-lint which creates output using the rich python library

The issue shows up when running inside concourse ci

I see.

Firstly, I believe @PerBothner has given a good reply (possibly with the exception of the last sentence).

In addition... as far as I know, rich does the necessary detection. So, the issue seems to lie on the end of concourse-ci. If concourse-ci "tells" the programs it executes that their output is a terminal device, then it should gracefully behave as one.

@jamie-pate
Copy link

jamie-pate commented Sep 6, 2023

Concourse doesn't advertise any virtual terminal support, but does support 'ANSI' color sequences ^[.... but not ^]8...

Many concourse examples add TERM=xterm-color to the environment, which advertises full support of the spec..

By removing the falsely advertised terminal advertisement from my config prevents the links, but I also lose all color.

@PerBothner
Copy link

If rich sees TERM=xterm-color I think it is reasonable for it to assume it is safe to emit OSC escape sequences, especially the more common ones.

So either fix/enhance concourse to ignore OSC escape sequences (probably not that difficult if it already handles ANSI color sequences). Or change TERM to something closer to what concourse supports, such as TERM=ansi. (I don't know if TERM=ansi will allow colors, but you should be able to find something that works.) And Concourse examples should be fixed to not use TERM=xterm-color as that is too much of a lie.

@jamie-pate
Copy link

jamie-pate commented Sep 6, 2023

I don't know if TERM=ansi will allow colors

Unfortunately, this doesn't seem to be possible.

The issue is that 'supports colors' has been the dominating aspect of terminfo for so long that every library that sniffs for terminal capabilities will only check if it 'supports colors' and then give up if it doesn't. Other Control Function escape sequences have not been on the radar for quite a while for this class of non-interactive program. (edit: see this relevant code from the rich library as an example.) (ncurses-alike libraries will need more capabilities)

I agree the best way forward is that concourse's elm-ansi should be updated. This leaves me currently with the task of stripping unsupported sequences using sed and that is fine.

(edit: Actually, concourse+rich still guesses 'standardcolor' without TERM)

@stuaxo
Copy link

stuaxo commented Sep 7, 2023

Probably worth opening a ticket on concourse ci for osc8 support since it's open source.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment