Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Hyperlinks in Terminal Emulators
@evaryont

This comment has been minimized.

Copy link

commented Apr 25, 2017

This isn't made explicit, and I didn't see any concrete consensus in the discussions, how are arbitrary/non-standard URL schemes handled? From the looks of things, it's not the terminal's responsibility, they will just pass the URL to the OS's own file open routines. So if I wanted to generate a link to say, an IRC chatroom:

echo -e '\e]8;;irc://irc.freenode.net/#lobby\e\\#lobby channel\e]8;;\e\\'

All I will need to do is make sure my IRC client is registered in the system to handle irc:// links. Is that correct?

@egmontkob

This comment has been minimized.

Copy link
Owner Author

commented Apr 25, 2017

@evaryont You're right, we don't intend to go into such details. It's up to the terminal emulator to decide which schemes it supports and what action it takes on them, probably delegating this decision to some lower level OS routine.

Most terminal emulators already autodetect standard URLs appearing on the screen. On one hand, it's reasonable for them to take the same action, regardless whether e.g. "irc://..." is autodetected as a visible string, or specified explicitly via this new escape sequence. On the other hand, the new one is potentially more easily configurable by the users since they won't have to handcraft complicated regexes, so perhaps some terminal emulators will add a configuration option for this.

It might also make sense for terminal emulators to offer the possibility to override the OS-wide action because the OS-wide registry would probably prefer to open a graphical app, whereas if coming from a terminal emulator, you might want to prefer to open a terminal-based IRC client in a new window/tab of the same terminal emulator, or similarly, prefer a lynx-style browser for http/https rather than a graphical one.

This is an open ended question, leaving room for terminal emulators to come up with nice ideas.

@ssbarnea

This comment has been minimized.

Copy link

commented Jun 21, 2017

This is really cool! I found this after looking for the same thing myself (and wasting some time with the reddit thread).
Is there any ANSI "coloring" python library already implementing this? I would love to start giving hyperlinks as output. The option to display them raw is not ok.

@Foxboron

This comment has been minimized.

Copy link

commented Jul 8, 2017

Wouldn't this allow rather ugly attack vectors where the user doesn't understand what he/she is clicking?

@egmontkob

This comment has been minimized.

Copy link
Owner Author

commented Jul 12, 2017

@ssbarnea Not that I'm aware of. Thanks for spreading the news of this feature and asking others to add support!

@egmontkob

This comment has been minimized.

Copy link
Owner Author

commented Jul 12, 2017

@Foxboron I'm fairly sure there's no risk here at all. The situation shouldn't be any worse than clicking on a link on an untrusted webpage. In fact, webpages are worse. First, they can leak data via the Referer field which isn't filled out if you click from the terminal emulator. Second, they can do nasty JS tricks to alter the target just when you click on the link (sometimes this is actually used for good purposes, like to inject a redirector which hides the referring page). In the terminal emulator a malicious app can still probably overwrite the link target at any time, but it has no clue when you're about to click.

Any webpage that does harmful things (e.g. delete data stored on a server) by just visiting a link suffer from CSRF bug anyway which can be exploited from browsers (malicious websites) too.

@towolf

This comment has been minimized.

Copy link

commented Jul 19, 2017

@stefanhaustein

This comment has been minimized.

Copy link

commented Sep 3, 2017

What happens to "data:" urls or pure local (#) links? Any chance to get anything to STDIN or back to a terminal based app that doesn't involve registering a link handler with the window manager? Use case: local link handling inside a "pure" command-line app with hyperlink support.

@egmontkob

This comment has been minimized.

Copy link
Owner Author

commented Sep 3, 2017

@stefanhaustein Interpretation of different schemes (or lack thereof) is up to the terminal emulator. E.g. with "data:" URLs they might show a popup with the content, or store in a temporary file and open that. Note that with the current limit of 2083 bytes for the URL in both VTE and iTerm2, the "data:" scheme isn't quite useful. I wouldn't want to significantly increase the limit (let alone remove it altogether) because then a malicious utility could cause the terminal emulator to eat tons of memory.

Handling the link inside the existing terminal emulator is alas again something that would be truly problematic. It would raise tons of questions like what to do if an app is already running or if you're ssh'd to a remote host, and probably a whole lot more. Not to mention that if we did anything like this, that would probably introduce serious security concerns.

The closest you can do with the current design is to specify a handler that opens a new terminal window and a specific app inside that, e.g. "lynx" for "http(s)" scheme. I'm not sure if iTerm2 allows you to configure this. GNOME Terminal takes the handler app from the global GNOME-wide configuration, hence it cannot differ from what's used for graphical apps. This is something we might improve on if there's a high demand – or other VTE-based apps might implement this.

@flarn2006

This comment has been minimized.

Copy link

commented Sep 13, 2017

@egmontkob: Malicious "utilities" aren't worth worrying about. If you're executing a malicious binary, you've got much bigger problems than a terminal emulator eating up memory. However, catting a text file could be a problem, as people expect text files to be safe to display. This is actually similar to an issue MS-DOS had with the ANSI.SYS driver, only that was much worse, as that allowed random text files to change what the keys on your keyboard do—for instance, making a common letter instead type a command that formats your hard drive and press Enter.

@egmontkob

This comment has been minimized.

Copy link
Owner Author

commented Sep 13, 2017

@flarn2006 Thanks for the clarification, I indeed did not clearly distinguish these two. A malicious binary can also be a compromised binary on a remote system that you ssh into, in which case it's like locally cating a maliciously constructed text file (plus all the damage being done over there on the compromised system). As far as the terminal emulator is concerned, it's only the incoming byte stream that matters.

@hdon

This comment has been minimized.

Copy link

commented Oct 10, 2017

A note to any maintainers of terminal emulators out there: please include a prominent option to scale back this feature. I've been using terminal emulators for 20 years and one behavior I find really annoying is clicking even what is obviously a URL and then having it come up in my browser. Sometimes I'm just trying to select text...

(Oh, and rectangle select is a nice feature, too, guys!)

@mofux

This comment has been minimized.

Copy link

commented Nov 28, 2017

For reference, xterm.js is tracking this feature request here: xtermjs/xterm.js#1134

@egmontkob

This comment has been minimized.

Copy link
Owner Author

commented Nov 28, 2017

@mofux Thanks, added.

@devkev

This comment has been minimized.

Copy link

commented Nov 30, 2017

To follow up on @Foxboron's earlier comment, it would be great if this document included a requirement for terminals to prominently inform the user of the actual URL before they click on it, eg. on hover (similar to web browsers). iTerm2 does this, but I'm not sure about other terminals.

This isn't just a convenience, it's important to prevent attacks like '\e]8;;http://malicious.com/\e\\http://safe.com/\e]8;;\e\\' (ie. the user thinks they're clicking a link to safe.com, but they're actually going to malicious.com), which is why I think it should be mandatory.

@aisamanra

This comment has been minimized.

Copy link

commented Nov 30, 2017

Given that our Matterhorn project was linked to up above, I figure I should note here that our newest Matterhorn release—version 40400.0.0—produces hyperlink escape sequences for all links which appear in the chat. (Matterhorn is built on top of the vty and brick libraries, which are medium- and high-level abstractions over terminals, respectively, and both of these libraries added support for hyperlink escape sequences in October.)

@egmontkob

This comment has been minimized.

Copy link
Owner Author

commented Dec 1, 2017

@aisamanra: Thanks, added.

@egmontkob

This comment has been minimized.

Copy link
Owner Author

commented Dec 1, 2017

@devkev: gnome-terminal shows the target as a tooltip (appears about half a second after you stop moving the mouse). Not sure about tilix and I cannot test it right now.

I'm not sure I fully share your concerns. It's a quite complex scenario and I'm wondering if the terminal emulator is the right way to address this, I'm honestly not sure.

Certain utilities use this feature for linkifying what they autodetect as links anyways, just to keep it working across linebreaks or so. As far as I understand, the aforementioned Matterhorn is such an example: it cannot be tricked by a remote user to show URLs where the actual target differs from the visible text. Other utilities, like ls also always produce trustable output.

Another use case is where the visible URL and the target are significantly different. Let's say a site literally called malicious.com looks totally like Facebook, phishing for your data hoping that you'd enter your credentials there. In this case this malicious URL will also be visible in your browser's URL bar, and I'm not entirely sure how much the terminal emulator also showing this URL would help you notice what's going on.

Yet another use case is where the actual URL looks damn similar to the one it's phishing, let's say faceboook.com (with three o's) or some Unicode ideographs. In this case again the terminal emulator showing the target probably wouldn't help you too much in catching the phishing attempt. Also note that this does not only hold for such OSC 8 explicit hyperlinks, the current feature of autodetecting URLs in pretty much every graphical terminal emulator also suffers from this issue, it isn't addressed there and I'm not aware of anyone ever having complained about it.

To tell a malicious site, it's probably a better approach to have some centrally maintained database of known such sites, and check the URL against that. And that's what modern browsers do when you open the URL. The same functionality could be repeated elsewhere too: In utilities that present links to the users (let's say, terminal based e-mail clients); in the URL-opening framework of popular graphical desktops (e.g. whatever happens behind the scenes after a gtk_show_uri()); or indeed in terminal emulators. I'm not sure what the best place for this is.

Given what browsers already do, terminal emulators here could be another line of defense, but would definitely not be the only one, nor the strongest one. As such, I wouldn't make it a requirement. I'd say terminal emulators are recommended to show the target URL, but it's also up to the users to choose an emulator accordingly (or file feature requests, PRs) if this feature is important for them. Obviously there's also room for further improvements, e.g. some terminal emulators might want download some malware list and check the URL against it, or might highlight the domain name (as browsers do in the URL bar) or might want to warn the user in case non-ASCII letters are present in the domain name, etc.

So I'd suggest that we bring these issues to the attention of terminal emulator developers, and recommend them to at least somehow conveniently let users figure out the URL before they activate it. Does this sound reasonable to you?

@jtdaugherty

This comment has been minimized.

Copy link

commented Dec 2, 2017

I'll also note that soon Vty will not do hyperlinking by default because of garbage output with older versions of VTE. An API call will be required to opt in and enable it. For more info: jtdaugherty/vty#137

@devkev

This comment has been minimized.

Copy link

commented Dec 4, 2017

@egmontkob, I think you may have missed the point. This isn't about trying to detect malicious sites, or spoofing attempts. It's about users being able to make an informed choice (about clicking on a link). Without the ability to discover the actual url, arbitrary-text links introduce an ambiguity that prevents users from being able to trust any links presented to them. This makes the linking feature effectively useless, and a security risk.

Suppose you have a terminal which:

  • auto links urls
  • displays arbitrary-text links to urls (ie. this spec)
  • does not have any way for users to discover the actual url of these links

Now you (the user) see the linkified text https://www.google.com/ somewhere on your screen. If you click on this, will the Google homepage be loaded in your browser? You have absolutely no way of knowing. Clicking the link could open literally any url at all, including urls that you (the user) would never have chosen to click if you had been properly informed.

This is why it's vital for users to be able to discover the actual url of any link. It's the same reason that web browsers and email clients show users the actual url. It applies to any software that presents arbitrary text hyperlinks.

@devkev

This comment has been minimized.

Copy link

commented Dec 4, 2017

Also, terminology's ticket tracking this feature is https://phab.enlightenment.org/T6329

@egmontkob

This comment has been minimized.

Copy link
Owner Author

commented Dec 4, 2017

@devkev I understand your usability concerns here, but I don't understand how it's a security or privacy concern, and as such, I'd leave it up to authors of terminal emulators, and leave it up to users to choose a terminal emulator meeting their needs.

As I've already mentioned in this comment, this problem is also present and is more serious on webpages. E.g. search for something in Google and hover over the results. In the bottom corner of the browser, you'll see that the link points let's say to Wikipedia. The moment you click on it (before the browser begins to follow it) the link target is changed to some Google-owned redirector. This is a proof that it can be done on webpages, and hence probably can be done for malicious purposes as well. At least in terminal emulators, the target cannot be altered just when you click on it.

@egmontkob

This comment has been minimized.

Copy link
Owner Author

commented Dec 4, 2017

@devkev I've added the Terminology link, thanks. (Honestly, I don't expect any terminal emulator that refused true color support to implement this one, and hence I myself didn't bother filing feature requests against them.)

@vapier

This comment has been minimized.

Copy link

commented Feb 26, 2018

i've implemented this in hterm (commit) and released it in hterm-1.76. couldn't see any way of updating the gist or sending an update request directly.

@egmontkob

This comment has been minimized.

Copy link
Owner Author

commented Feb 28, 2018

@vapier Thanks a lot, it's great to hear this feature is gaining popularity. I've updated the page. (I've no clue either how to do collaborative stuff on gists.)

@1Hyena

This comment has been minimized.

Copy link

commented Mar 11, 2018

Just learned about this! This is awesome, it has enormous implications to text-based gaming (MUDs). Is it possible to send a predefined string to stdin when user clicks on a link? That would make Text User Interfaces s so much more convenient to use.

@egmontkob

This comment has been minimized.

Copy link
Owner Author

commented Mar 13, 2018

@1Hyena It would probably be a quite straightforward change to e.g. gnome-terminal to handle special URIs like send-text:foobar. There's no standard for such protocol (scheme) though, and more importantly, this feature wouldn't be accepted mainstream due to the serious security implications.

@modest

This comment has been minimized.

Copy link

commented Apr 7, 2018

It seems critical to include some spec guidance on schema whitelists and security. The reality on most platforms is that many apps register insecure URI schema handlers for private usage. (My Mac has over 140 of these protocols registered.) These apps did not consider the security or privacy implications of the schema being abused by a third party. A malicious URL can cause these apps to take actions without user confirmation, crash the process, execute arbitrary code, login as a different user, or repeatedly call the police.

For this reason, all major web browsers have whitelists for URL schemas that are allowed without prompting the user (usually just http:, https:, and ftp:). For all other URL schemas, the user is prompted after clicking the link and must explicitly allow the schema handler to open. This prevents users from being tricked into clicking on links that abuse poorly-secured schema handlers from installed apps.

Example of a security prompt in Chrome when clicking a skype: link
screen shot 2018-04-06 at 22 19 44

@egmontkob:

I'm fairly sure there's no risk here at all. The situation shouldn't be any worse than clicking on a link on an untrusted webpage.

It is true that this feature is no more dangerous than a web browser, but only if implemented with a schema whitelist like a web browser. Otherwise, this exposes terminals to a dangerous and widespread category of vulnerabilities.

@gsemet

This comment has been minimized.

Copy link

commented Apr 14, 2018

Guake has been updated and should support this now :)

@nickl-

This comment has been minimized.

Copy link

commented May 1, 2018

Testing the waters to see if the community has an interest to explore and define standards for currently off-spec but terminal hypertext link related/useful uri schemes. URI schemes and OS specific scheme handlers for terminal hypertext links functionality directed back towards the shell session.

See nickl-/term-hyperlinks and create an issue with your comments and suggestions.

@haya14busa

This comment has been minimized.

Copy link

commented May 4, 2018

FWIW, I write a shell script which support tmux too. https://github.com/haya14busa/hyperlink

@PerBothner

This comment has been minimized.

Copy link

commented May 10, 2018

DomTerm version 1.0.2 now supports this escape sequence.

DomTerm also has special handling of file: URLs of the form file:/filename#position=line or file:/filename#position=line:column. Clicking on such a link brings up an editor at the specified position. See here and here. These links can be created explicitly (using the \e]8 escape sequence or DomTerm's more general sequence to embed HTML), or implicit (based on matching the pattern FILENAME:LINE: or FILENAME:LINE:COLUMN: - as produced by compiler error messages). This may worth considering as an extension of the specification.

@egmontkob

This comment has been minimized.

Copy link
Owner Author

commented May 10, 2018

@modest Thanks a lot, I believe you raised an important problem.

In an ideal world, I guess we could just sit back and blame those URL handler apps that do harmful stuff upon clicking on their URLs, couldn't we? We're far from this, and as such, sure it would be desirable for terminal emulators to ask for confirmation.

I don't know if certain operating systems or desktop environments provide any support (framework) for this, or it needs to be implemented from scratch in terminal emulators. Ideally, I guess anyone who provides a method to open a URL (e.g. GTK+'s gtk_show_uri()) should give us some help here. In practice I'm afraid they don't.

@egmontkob

This comment has been minimized.

Copy link
Owner Author

commented May 10, 2018

@PerBothner Thanks for the heads-up!

I actually looked around a while ago and was surprised to see that there wasn't a common notation for jumping to a certain position. It would sure be nice to see a standard emerging. Compiler errors, as you mention, is a perfect use case. Another one could be the filename as printed by grep in front of every matching line.

(As far as gnome-terminal is concerned, there's nothing we could do in gnome-terminal itself though, handling the position should go into GTK+.)

I can see a couple of issues, though.

A local file, denoted by the file: scheme, could still have a MIME type, probably autodetected based on extension or file contents magic. And that MIME type could be e.g. text/html or any other format that does have the concept of internal positions to jump to. So, we have a conflict since these two features don't have a separate namespace. Should a file:/foo.html#position=5 jump to the fifth line, or to the HTML tag having the ID of "position=5"? (But let's take one step back: Who decides whether to open this file with a plain text viewer as raw text, or with a HTML viewer as rendered HTML? Anyone adding that "#position=5" probably expects that it'll be opened as a text file. So shouldn't we also add the possibility to force opening with a plain text viewer; e.g. by specifying a forced MIME type?)

I find the choice of the word "position=" quite unfortunate, as it's not clearly extendable to other use cases, e.g. byte offset in a text file, page number within a PDF, time offset within a video etc. They're all positions, taking a unit of a different dimension as their argument. Probably "line=" and "column=" would be better, although I'm not sure what should be the separator between them (is another '#' okay?).

For plain text files let's assume that "line" is a clear story (let's for now put aside all those confusions that CR vs. LF vs. CR-LF can cause). "column" is more problematic. What's the encoding of the text file? Do we count bytes, or characters, or width in terminal (including double-wide characters), or Unicode codepoints, or what exactly? How to handle TAB? What about BiDi, is it logical or visual column?

Would be nice to have some standard (maybe even an RFC) about this. I'm afraid DomTerm isn't a strong enough reference everyone would agree to follow.

@PerBothner

This comment has been minimized.

Copy link

commented May 29, 2018

"Should a file:/foo.html#position=5 jump to the fifth line, or to the HTML tag having the ID of "position=5"?"

The character '=' is not valid for the id attribute in either XML and HTML 4. It is allowed by HTML 5, but discouraged. Some user agents may be able to go to check if there is an ID with value "position=5", but if they can't it is reasonable to ignore that possibility.

"Probably "line=" and "column=" would be better,"

That doesn't handle ranges: START_LINE:START_COLUMN to END_LINE:END_COLUMN. For an error messages or grep output it can be useful to not only navigate to a position, but also to high-light a range.

"although I'm not sure what should be the separator between them (is another '#' okay?)"

I don't believe so. Using '&' seems reasonable - it matches the convention for "query string" (starting with '?'). Basically, the server handles the '?' part; the browser handles the '#' part.

"So shouldn't we also add the possibility to force opening with a plain text viewer; e.g. by specifying a forced MIME type?"

The format can be extended. For example foo.ext#position=10:12&mime-type=text/plain

"find the choice of the word "position=" quite unfortunate"

I'm not disagreeing. Just keep in mind that the encoding should support ranges.

""column" is more problematic. What's the encoding of the text file? Do we count bytes, or characters, or width in terminal (including double-wide characters), or Unicode codepoints, or what exactly? How to handle TAB? What about BiDi, is it logical or visual column?"

The Language Server Protocol counts 16-bit characters. I.e. Unicode characters not in the BMP (Basic Multilingual Plane) count 2 as they use a surrogate pair. Nice and simple for Java or JavaScript but not very semantic and somewhat error-prone (since people will often skip dealing with it.)

In LSP both line and column numbers are zero-based, not one-based. (DomTerm #position values are one-based, though.)

LSP does specify 1 for with TAB and double-wide characters (in the BMP), which I think is a reasonable choice.

Note it is possible to pick one convention - and allow future extensions. A URL could possible encode positions in more than one way:

foo.java#position=3:10&lsp-position=2:12

Some user-agents could handle #position; some could handle #lsp-position; some could handle both. Some producers (compilers etc) could emit one or the other or both. If a handler sees #lsp-position but can only handle #position it can convert to the former by adding 1. That might not be 100% right but it would usually be good enough.

An RFC could be nice, I agree.

@egmontkob

This comment has been minimized.

Copy link
Owner Author

commented May 31, 2018

The character '=' is not valid for the id attribute

Still, it's a pretty ugly design if the semantics of a keyword depends on whether the value contains such a sign or not. But it might still be the best we can come up with.

That doesn't handle ranges: START_LINE:START_COLUMN to END_LINE:END_COLUMN. For an error messages or grep output it can be useful to not only navigate to a position, but also to high-light a range.

Is this really needed, or is it an overkill? If really needed, don't we need multiple ranges too? E.g. grep can find multiple matches within a row. I'd rather say it's an overkill for now, but the chosen design should be forward compatible with such possible improvements.

Language Server Protocol

Never heard of that; will need to take a closer look.

counts 16-bit characters. I.e. Unicode characters not in the BMP (Basic Multilingual Plane) count 2 as they use a surrogate pair [...] not very semantic and somewhat error-prone

They use a surrogate pair in UTF-16, the most brain-damaged encoding of Unicode, combining the disadvantages of other representation formats (UCS-4 and UTF-8). This encoding should have died out a long time ago. Luckily AFAICT it's mostly used internally by some legacy stuff that originated in the very early days Unicode, like Java. (Or is it still a common thing in Windows?) Especially in the Unix world where data exchange happens primarily in UTF-8, I can't imagine any valid reason for non-BMP to be handled differently from BMP.

I'd say it's a TERRIBLE thing, freaking error-prone, and gets an absolutely "no go" vote from me.

LSP does specify 1 for with TAB and double-wide characters (in the BMP), which I think is a reasonable choice.

Agree. We need to encode logical positions within the text file, rather than visual positions. The editor that opens the file will know the tab width it chooses and will place the cursor in its corresponding visual position, meaning that the indicated column number will probably be quite different, it's its internal business. Tools like gcc -ftabstop=... will need to take care to generate different column numbers for the URL than for the visual column number they print based on their assumption of tab width.

Note it is possible to pick one convention - and allow future extensions.

Future extensions: yes. Future alternatives: no.

If a handler sees #lsp-position but can only handle #position it can convert to the former by adding 1.

(s/former/latter I guess.)

If a handler recognizes the keyword #lsp-position and knows to add 1, then it already supports it (at least to some basic degree), doesn't it? What if it doesn't recognize this keyword at all?

There has to be a minimum base that's commonly used everywhere, and there could go extensions on top of that, that's okay. It's NOT okay to have competing alternatives, i.e. if an emitter may emit either format X or Y, and a consumer might understand only one of X and Y.

@sigalrm

This comment has been minimized.

Copy link

commented Jun 2, 2018

Hello, TermySequence is my new terminal multiplexer which supports this escape sequence from the initial release. Thanks for doing this specification work.

@stuaxo

This comment has been minimized.

Copy link

commented Jun 4, 2018

This is great, I've been wanting something like this forever.

We need a way that the shell can be aware a hyperlink has been clicked - it would be good if there was an easy way that clicking a directory, bash could cd into it if properly setup.

@egmontkob

This comment has been minimized.

Copy link
Owner Author

commented Jun 4, 2018

@stuaxo This – or doing anything inside the same terminal, e.g. opening a text-based browser or e-mail client for an http(s) or mailto link – would need an enormous amount of design and work, and is outside of the goals of this project. There would either need to be a separate communication channel between the terminal emulator and the shell solely for the purposes of such new features, or the terminal emulator would need to completely understand what's happening inside (something like iTerm2's shell integration, but probably even more thorough), would need to know whether the shell is sitting idle at the prompt and no partial command has been entered (which probably requires explicit support from the shell), and in that case it could feed the necessary faked keystrokes to type that cd command for you. And then make sure it's free of security issues, and race conditions due to the full duplex asynchronous nature of the communication channel between the shell and the terminal (probably the synthesized cd command would need to be enclosed in special symbols similarly to bracketed paste). It's unlikely for anything like this to happen. Explicit hyperlinks are useful for opening certain resources externally to the current terminal emulator (either in a separate graphical app, or in a newly launched terminal emulator).

@egmontkob

This comment has been minimized.

Copy link
Owner Author

commented Jun 4, 2018

@stuaxo Thinking about it a bit more, something along the lines of how bracketed paste technically works could work here too. If a certain mode is enabled then clicked hyperlinks would be sent to the app enclosed between some special marks. (If the mode is disabled, which would be the default, then nothing would be sent.) It would be the shell deciding what to do (e.g. cd if it's a file: URL with the local hostname), rather than the terminal emulator deciding to fake type a cd command. It could even change the directory when the command line is not empty (similarly to e.g. how ^X^V is handled in bash). The shell could even be configured to launch certain apps on clicking some types of URLs – the problem being though that it'd only work whenever the shell doesn't have a foreground job running.

In order to move forward, I guess we'd need buy-in from at least one popular shell, plus ideally a few other (non-shell) use cases in mind as well. I can't really think of the latter kind; non-shell apps probably aren't interested in URLs back in the scrollback, and if a fullscreen app is interested about URLs emitted by itself then it can more easily do its own mouse handling. So I'm still afraid it'd be a marginal feature which may not be worth the efforts.

@ssbarnea

This comment has been minimized.

Copy link

commented Jun 5, 2018

@egmontkob See tartley/colorama#134 for adding hyperlink support in colorama, the most popular ANSI python library. Implementing this there will open it to lots of tools. Some support would be needed to get the buy in, I already made a pull request.

PS. I think that it could be very useful for the the "ANSI" cause to have a proper gitlab project (maybe even org) that tracks/documents improvements and support of different extensions on various applications. Using a personal gist does not send the best signal (may give the impression that there is only one supporter behind).

@egmontkob

This comment has been minimized.

Copy link
Owner Author

commented Jun 5, 2018

@ssbarnea If anyone's willing to create and maintain a documentation of all the terminal emulator extensions out there, it would be highly appreciated. It won't be me.

@stuaxo

This comment has been minimized.

Copy link

commented Jun 5, 2018

@egmontkob maybe it's a matter of getting some shells to understand file:/// for local directories - zsh already can change to a directory if you enter a path without cd

@egmontkob

This comment has been minimized.

Copy link
Owner Author

commented Jun 6, 2018

@stuaxo Having to parse a file:/// URI is not the problematic part here at all.

zsh already can change to a directory if you enter a path without cd

bash too. And it's absolutely irrelevant here.

@egmontkob

This comment has been minimized.

Copy link
Owner Author

commented Sep 29, 2018

I've modified the page to prefer the ECMA standard ST (\e\\) over the nonstandard BEL (\a) terminator. I took the liberty of editing others' comments too to use the preferred form.

@egmontkob

This comment has been minimized.

Copy link
Owner Author

commented Oct 15, 2018

(I don't normally do this, but after long hesitation I decided to remove a few comments from here: a discussion that got way too long and hardly contributed anything to this topic.

The poster insisted that this whole feature is conceptually broken: the terminal emulator should itself match its onscreen contents against configurable regexes and combine them with other externally gathered information (such as looking around in /proc) to recognize filenames and decide what action to take. The poster ignored all the problems and limitations that this approach had which were pretty much the reasons for designing the explicit hyperlink feature.

In my opinion, its only valuable contribution was to point out that when using containers, chroots etc., filenames produced by tools running inside these containers might become broken when trying to open them from the emulator. This is indeed true, however, I find it a way less typical use case than the ones OSC 8 fixes. Moreover, if the terminal emulator is able to somehow figure out the context of the utilities running inside, e.g. how the filenames need to be mapped (as the poster claimed it should), then it can also perform this mapping on the OSC 8 hyperlink targets just as it would do them for regex matches. So, in my opinion, no feature is lost or degraded by introducing explicit hyperlinks, not even with containers in the game.

How a terminal emulator should detect the app's container, chroot etc. and how it should decide the desired mapping of filenames is outside of the scope of this feature and this document.

Should anyone be interested, the deleted discussion was archived by the well known Internet Archive Wayback Machine.)

@sindresorhus

This comment has been minimized.

Copy link

commented Nov 13, 2018

@egmontkob Could you consider moving this to a GitHub/GitLab repo instead? That way people could do PRs to update/improve it. And we would get notifications on new comments.

@egmontkob

This comment has been minimized.

Copy link
Owner Author

commented Nov 13, 2018

@sindresorhus I also feel the pain from these limitations. In maybe a week or two I'll post a brand new design document for terminal emulators (unrelated to this one), and I have already decided to have a proper repo. I'm yet to evaluate whether to go with public GitHub, public GitLab, or GNOME's GitLab. Moving this one too is a good idea!

@drudru

This comment has been minimized.

Copy link

commented Dec 3, 2018

Hi - I was recommended that I check this out. This is a good idea. Thx for working on specing this out. I am implementing this feature right now.

I have a question about the encodings. Do you want to allow space characters and other punctuation in the Params section?

URIs are easier since that is standardized nicely. (No whitespace, etc.)

Also, it would be great if you defined a regex to match this particular OSC code.

@egmontkob

This comment has been minimized.

Copy link
Owner Author

commented Dec 3, 2018

@drudru According to ECMA-48 § 8.3.39, the valid characters within OSC are 0x08..0x0D and 0x20..0x7E. I'm tempted to say that the valid ones for parameters should be the printable ones, that is, 0x20..0x7E, except for the defined separators 0x3A : and 0x3B ;. I don't see a reason for deliberately further excluding let's say the space character from parameters. Does this sound reasonable?

If really required, a formal specification should use ABNF or something similar, not a regex. I don't wish to create one, but if someone does, I'll be happy to add to the page.

A regex can define whether a string matches or not, but – as opposed to ABNF – cannot tell how to break it down to components if it matches. Regexes depend on all various circumstances, like the exact engine interpreting them, or the current locale affecting intervals or equivalence classes.

I cannot see how a robust and fast implementation could use regexes to match against a stream of incoming data, rather than some state machine. For example, it's pretty easy to create a (faulty) regex where the match depends on the timing of the arriving data, which is unacceptable, and I'm not aware of a generic protection against this problem. If I were to write a parser for a terminal emulator, I sure wouldn't risk accidentally having such a regex here and there. Also, the time required gets multipiled by the length of the match if the input arrives byte by byte, e.g. if the regex matches in linear time O(n) then continuously matching against the stream will consume O(n^2) of CPU, etc. Of course it's also multiplied by the number of regexes you define.

Implementations are quietly assumed to already support OSC in general, for setting the window title (where space is definitely allowed) and such, and are expected to hook up recognizing this new numeric entry and handling the subparameters within that already existing framework. I find it unlikely (although sure possible) that a terminal emulator (or other utility) would newly implement parsing OSC for the sake of this feature. I don't want to go into this business as part of this spec, so let's just leave it here in the comments that catching and parsing the entire OSC using a regex is really not a great approach. (If you catch the entire OSC by other means and then parse its parameters using a regex, that's probably fine.)

@drudru

This comment has been minimized.

Copy link

commented Dec 3, 2018

@egmontkob - wrt to 'space', that is very reasonable. I was thinking about it this morning over coffee before I read this thread and came to the same conclusion. It is basically the same pattern for other sequences (CSI for example).

Also, agreed on the use of regex vs. ABNF.

tldr - thanks, it all makes sense, and I have what I need to move forward. If you want to learn more about my implementation and its needs, feel free to read further if you have time.

Just to provide more information on my use case, I am the author of the Ansi-Up javascript library. I am tracking the implementation with this issue: Ansi-Up Issue #51.

In my implementation, my goal is to process the stream and immediately return some HTML that best represents the stream at that point. I only process CSI SGR sequences. This is ideal for viewing the live streaming of log output in a web page. For example, the result of a compiler or a daemon process.

The major difference between my implementation and the use-cases mentioned above is that I don't expect to go back and patch the DOM when I receive new input. This is quite different from the use cases mentioned above. When I implemented 'linkification' in the past, I was relying on developers having enough luck to avoid receiving a partial 'packet'. I became painfully aware of the edge-cases and ended up dropping the feature until I was sent this spec.

In order to implement this feature, I have two choices. I could buffer or not buffer the stream until I receive a hyperlink terminator. I am currently leaning towards buffering even though it is more complex.

The hard part of this implementation is that I need to know whether the pattern matches, could match, or is an illegal pattern. If I can do all of this in a regex, then I get a performance win in the browser. Also, by properly constraining the pattern (no spaces, whitespace here, etc.), I can avoid the horrible degenerate case of someone making a mistake with the pattern and creating a screen-full, single URL. This last piece is important to me.

One other thing to note is that I will be whitelisting URI prefixes. This will eliminate 'javascript:' and 'data:' prefixes.

Also, I would also vote for moving this to a github project.

Thanks again for your spec.

@drudru

This comment has been minimized.

Copy link

commented Dec 5, 2018

@egmontkob - quick note - I decided in my implementation to forbid newlines in the entire sequence. I felt that it prevents any degenerate sequences from preventing my library from working.

@egmontkob

This comment has been minimized.

Copy link
Owner Author

commented Dec 5, 2018

When I implemented 'linkification' in the past, I was relying on developers having enough luck to avoid receiving a partial 'packet'. I became painfully aware of the edge-cases and ended up dropping the feature until I was sent this spec.

Yeah, on-the-fly autodetected linkification when the data can stall at any time is incompatible with the requirement of flushing the data as soon as it arrives.

I could buffer or not buffer the stream until I receive a hyperlink terminator.

Are you talking about the OSC 8 terminator \e\\ or \a, pretty much mapping to HTML's > (there are two of them in a complete hyperlink), or about the hyperlink terminator \e]8;;\e\\ corresponding to HTML's </a>? (I do get confused by them a lot.)

OSC 8's syntax pretty straightforwardly maps to HTML. The moment you see a \e]8; you know it's related to links. You can ignore the IDs. The moment you see the first character of a URL, you can tell it's an opening tag. Since you'd blacklist (or whitelist) some protocols, you should probably buffer up until the first colon, that is, to see the protocol (and potentially bail out if the protocol is longer than a small threshold). Once you encounter a http:, you can already emit <a href="http: and switch to a mode where any printable input is just copied to your HTML output (taking care to escape <, &, ', " to be sure). On the OSC 8 terminator \e\\ or \a you'd just finish the opening tag by outputting ">.

Or you can of course batch up until the entire opening tag is seen, that probably simplifies a lot, and doesn't degrade the user experience at all (a partially printed opening HTML tag doesn't help users see the data any faster). I wouldn't do any batching with the visible link string, the one between the link's opening entire OSC 8 sequence and the closing entire OSC 8.

@stuaxo

This comment has been minimized.

Copy link

commented Dec 12, 2018

Quick RFC for positioning, build on githubs idea of an L line number anchor.

This uses L to denote lines and optionally C for columns.
Lines and columns are 1 based.

It has the advantage of being short, which may look less weird in the context of ANSI.

Examples:

Open at line 10
file:///home/stu/document.txt#L10

Open, positioning cursor at line 20, column 20:
file:///home/stu/document.txt#L10C20

Ranges should open the editor with a range selected, if supported

Select line 10 to 20
#L10-L20

Select from line 10, five characters in, to line 12, 20 characters
#L10C5-L12C20

@egmontkob

This comment has been minimized.

Copy link
Owner Author

commented Dec 13, 2018

#L10 is probably the most common practice. I've expressed earlier my slight worries about a clash with the HTML tag with the ID of L10 in case the file is detected to be an HTML one. I've also posted dilemmas how to exactly interpret the column. But these are minor issues, and we can live with some imperfection here.

There's a major problem here, though, namely: how would terminal emulators implement such a specification? I'm not familiar with macOS (e.g. iTerm2), so I'll take Linux, GNOME and gnome-terminal as an example.

What happens currently is: gnome-terminal just simply passes the URI to gtk_show_uri() and that's all that it cares about it. The rest of the story is exactly the same as for other URLs opened from anywhere else in GNOME. The MIME type is somehow guessed, then ~/.config/mimeapps.list is consulted (falling back to I-don't-know-what if there's no such file) to locate a .desktop file, then that desktop file is opened, a command line is contsructed by using the given URI (if %u is present) or by converting it to a local file (if %f) and then it's launched. Chances are that I've missed a step or two.

I definitely do not intend to bypass all this infrastructure and come up with something custom one-off solution (disobeying the user's desktop-wide preferences) in gnome-terminal.

Where would the line number be parsed and interpreted? Should all text editors be required to recognize the file: scheme and the #L10 syntax, and as part of that, perform URI-decoding on their command line parameters? E.g. should Vim be modified not to copy the file first, as it does, but to open it at place? What about Emacs, Joe, and presumably a whole lot more that do not recognize file: and just treats it as a local path?

Or shall the desktop spec be extended by new macro(s), let's say %l for the line and something for the column (note: %c is taken), and then whoever is responsible for substituting these macros should interpret the #L10 URI fragment and convert accordingly? Probably a resonable approach.

All this needs cooperation from plenty of parties, probably beginning with a discussion on freedesktop mailing list, followed by a new version of the desktop file specs, followed by implementation, and editors starting to ship corresponding desktop files using these macros or interpreting the URL fragments themselves.

I'm not entirely certain what the best design would be. I'd love to see this happening, I'm happy to cime in in discussions if someone drives it, but I don't have capacity to drive this. And if I just added #L10 to this specification, it would take us nowhere, it presumably wouldn't be sufficient to push all these changes through, and I couldn't implement it in gnome-terminal either.

A much higher level agreement is required on the URI syntax for local files at a given line; that is, an agreement across OSes and desktops and kinds of products (not just terminal emulators). This needs to be followed by implementation in various desktops (such as GNOME). And once it's done, there'll be nothing left to do by terminal emulators, it'll just work.

@langpavel

This comment has been minimized.

Copy link

commented Dec 27, 2018

For node users there is terminal-link package. And there is discussion on chalk

@koconder

This comment has been minimized.

Copy link

commented Jan 14, 2019

Just tested on iTerm 3.2.6 and not working

@mintty

This comment has been minimized.

Copy link

commented Jan 22, 2019

Mintty now supports hyperlink attributes (just uploaded, hovering to follow).

@mintty

This comment has been minimized.

Copy link

commented Jan 23, 2019

Mintty hyperlink attribute support is now complete.

@lmorg

This comment has been minimized.

Copy link

commented Feb 6, 2019

I'm a little concerned about how genuine security concerns (such as the following: https://gist.github.com/egmontkob/eb114294efbcd5adb1944c9f3cb5feda#gistcomment-2404153) have been completely ignored.

I've written some thoughts on this in a related discussion (kovidgoyal/kitty#68 (comment)) but essentially I seriously question just how much convenience this brings beyond a few niche use cases that cannot already be served with other workflows (such as right or shift clicking a URL) that still preserve the WYSIWYG nature of terminal output.

We already suffer from a whole plethora of CSFR, phishing attacks and so on and so forth inside web browsers - let alone a bloat of dubious URL schemas - so the last thing we need to do is make one of our most sensitive and important administrative tools subject of doubt when working with references such as URLs.

That all said, I'm not saying hyperlinks have no place in terminals per se - just that we need to seriously think about how to do this properly rather than jumping in feet first because it seems hip and cool.

@lmorg

This comment has been minimized.

Copy link

commented Feb 6, 2019

Taken from another discussion, but posting here because the author of that project didn't want the discussion on his issue tracker (kovidgoyal/kitty#68 (comment))

Security aspects have been discussed in the comments section of the feature's gist, I recommend to you guys to go through them.

I already had done (that's how I found this issue) and those concerns weren't addressed at all. In fact I even replied on there stating exactly that.

The only pending concern I find valid is that some OSes allow various apps to register (unsafe) handler apps for various protocols. A solution could be to limit support in Kitty for a few well-known protocols, e.g. http, https, ftp and file.

Just because you don't find other security concerns valid that doesn't mean you should simply ignore them when others do raise legitimate points.

Equally so from a browser. If a malicious HTTP(S) request can be crafted, your website is buggy as it doesn't do proper authentication, CSRF etc. protection.

That's just an acknowledgement that there is a problem. You're not addressing a solution to the problem with your hyperlinks proposal.

It's not strictly speaking a "need", it's a new convenience feature. Sure you can live without it, just as much you can live without curly and colored underlines, raw keyboard reporting, image support and all those other new features newly supported by Kitty or whichever other terminal.

Again, you're not actually addressing what problems this "convenience feature" solves. I've exampled other solutions that offer the same level of convenience but with additional protections against abuse, whereas you're just sweeping any concerns people raise under the carpet (and by your own admission, I'm not the only person to raise these points). Quite frankly, that's not an acceptable way to hold this discussions. Please meet us half way and talk about why you want this. THEN we might be able to find a common middle ground that works for all parties.

First, Kitty might offer a config option to disable hyperlinks. Then you can just entirely disable this feature for yourself. Second, apps (e.g. mutt), if they implement explicit hyperlinks, might also offer a way to disable them. This way you can disable them for apps that handle untrusted data.

That's a reasonable compromise however I'd argue it should be disabled by default. That way you can want individuals about the dangers of using it when they come to enable it.

I use this feature quite frequently. 5 seconds indeed sounds like a good estimate on average (counting for cases where copy-pasting the filename doesn't work due to space characters in it; tab completion stopping at unexpected places where I have to locate the next letter of the filename to be typed; and cases when I mix up the utility's name and type "evince" instead of "geeqie" or the other way around which does happen to me quite frequently). Then multiply this 5 seconds by maybe 10-20 times a day, for me.

The problems you're describing there can easily (and I mean easily) be solved by better command line tooling. Better auto-completion scripts, or a better $SHELL. You could also solve that problem with the terminal emulator detecting spaces in it's clipboard and auto escape it. Or even solve it just be typing a quotation mark before pasting your output. The issue you're describing here is really more of a problem with the user not learning to use their command line environment effectively. Now if they cannot be trusted to a quotation key before pasting random data in, then I certainly wouldn't trust them to click random hyperlinks without doing due diligence first!

In gnome-terminal, a single click doesn't follow the link, it selects text. You need to Ctrl+click, or right-click and choose the corresponding menu entry to open the link.

Given you can open a URL in quite a few terminals - even those that don't support OSC 8 - by doing exactly the same thing, I have to question what the bloody point of OSC 8 is. Just put file: schema in front of the file name and be done with it. You could even write a wrapper script around ls to do this for you. Then you have all parties happy with both security and usability concerns covered.

The feature's specification lists quite a few current and possible future use cases.

I'm sure it does but that doesn't help our discussion here and now where I've listed some use cases and the problems they create.

What's more, you're the one opening issues in every terminal emulators issue tracker requesting this feature so it might be nice if you then didn't expect everyone else to do the donkey work researching the background to this afterwards (nor does it help how you repeatedly sidestep every security-related point people bring up just because you happen not to care about it).

I really don't like how this is being pushed through - if feels like a pet project of yours that you're urging everyone should adopt without any security review. And your responses on here whenever anyone does audit the potential ramifications is essentially just "that doesn't concern me so I'll ignore it". I'm really not happy with how you're going about this and the last thing I want is my terminal to become yet another platform vulnerable to social engineering just because a small minority of hackers things "oooh this will be cool. I could save myself 5 seconds of time". So I urge you to actually investigate:

  1. what problems are you trying to solve?
  2. can those problems already be solved using existing tooling?
  3. what protections can you bring to mitigate any security concerns?

Thus far you're not answering any of those questions.

@lmorg

This comment has been minimized.

Copy link

commented Feb 6, 2019

reply to kovidgoyal/kitty#68 (comment)

kitty currently requires one or more configurable keys to be pressed while clicking a URL for it to be opened.

So then why do we need dedicated hyperlink support when Kitty already open URLs?

Well kitty supports Linux and macOS and at least on macOS this concern doesn't apply because there are no Macs with touch screens. I'm not sure about Linux as I have never tried to use a touch screen with Linux before.

Linux does support touch screens. But the point is also valid for other terminal emulators as well

I'm pretty sure kitty doesn't work with a screen reader as it uses OpenGL to render stuff. Other terminals that explicitly support screen readers will probably work much better for this sort of thing.

So the same argument but for a different terminal which @egmontkob has also requested this feature on (he's been contacting a great number of developers asking this to be incorporated).

@egmontkob

This comment has been minimized.

Copy link
Owner Author

commented Feb 6, 2019

I'm a little concerned about how genuine security concerns (such as the following: https://gist.github.com/egmontkob/eb114294efbcd5adb1944c9f3cb5feda#gistcomment-2404153) have been completely ignored.

I wouldn't say it's been completely ignored. I responded in a followup comment. Indeed I should've added this to the spec itself, as something implementations should watch out for. It's on my mental todo list.

I also wouldn't say "such as", I believe this is the only security point so far that we've found valid.

make one of our most sensitive and important administrative tools subject of doubt

If you're worried about opening a URL having unexpected, bad consequences, e.g. due to lack of proper CSRF prevention, then it's those websites that should be subject to doubt, not the terminal.

[CSRF] That's just an acknowledgement that there is a problem. You're not addressing a solution to the problem with your hyperlinks proposal.

No. That's the acknowledgement that the problem is elsewhere. Seeing a problem already existing somewhere else, on broken websites, can't be a valid reason for us not to move forward.

We have put a lot of thinking into the security aspects of this issue. One of the principles was: if something can already happen from a browser, and the very same thing can happen from now on from a terminal, that just can't be a bug or a security issue. A security issue would be something that we open up newly and is not yet seen while browsing the web.

If you question this principle, if you're worried about this feature exposing more surface for social engineering attacks, you should just not use this feature, it's that simple. If, after almost 2 years of its design and implementation is some terminals you want to revert all this work, it's clearly not going to happen, we're just wasting our time arguing about it.

If you find a security bug that's a new one according to the principle I've just mentioned, sure let us know.

Please meet us half way and talk about why you want this.

This document mentions plenty of places where this feature can be useful, convenient, potentially speed up one's work.

The world of terminal emulators continuously sees improvements made here and there. Sure you can go back to to a hardware monochrome terminal of decades ago and get your work done from that. Or you can, for each and every single improvement, question "why" that happened. Because it made terminals better for some people. Maybe not for you, but that alone can't be a reason for you to question the feature's usefulness.

The problems you're describing there can easily (and I mean easily) be solved by better command line tooling. Better auto-completion scripts, or a better $SHELL.

Show us please how you can do it better!

Given you can open a URL in quite a few terminals - even those that don't support OSC 8 - by doing exactly the same thing, I have to question what the bloody point of OSC 8 is.

This is a solid proof to me that you have not familiarized yourself enough with our work, and the possible use cases we (and others) have come up with.

I really don't like how this is being pushed through - if feels like a pet project of yours that you're urging everyone should adopt without any security review.

What I really don't like is how casually you're just waving the security flag, without any concrete issue you could come up with (except pointing to that single one so far that someone else found, and I've responded to). Seems to me we've already put a whole lot more into evaluating the security aspects than you have.

the last thing I want is my terminal to become yet another platform vulnerable to social engineering

It's not the terminal that's vulnerable, it's your careless websites. And those websites are already vulnerable, and any other untrusted site you visit can get you into exploring those vulnerabilities. There's nothing new here.

Since it's not the terminal emulator actually fetching the remote site but it delegates this task to a browser, you also get the phishing prevention and stuff that generally a browser provides.

Of course you're always free to use whichever terminal emulator of your choice, which doesn't support this feature or allows you to disable.

That all said, I'm not saying hyperlinks have no place in terminals per se - just that we need to seriously think about how to do this properly rather than jumping in feet first because it seems hip and cool.

Care to actually help, rather than blaming what we've done so far? Because in the latter one I'm not interested at all.

@lmorg

This comment has been minimized.

Copy link

commented Feb 6, 2019

If you're worried about opening a URL having unexpected, bad consequences, e.g. due to lack of proper CSRF prevention, then it's those websites that should be subject to doubt, not the terminal.

You cannot expect every website ever written to be secure. This is why I and others have repeatedly quoted the follow: https://en.wikipedia.org/wiki/Defense_in_depth_%28computing%29

No. That's the acknowledgement that the problem is elsewhere. Seeing a problem already existing somewhere else, on broken websites, can't be a valid reason for us not to move forward.

But if the problem exists elsewhere that implements your proposal than ergo the problem also exists with your proposed implementation.

We have put a lot of thinking into the security aspects of this issue.

I completely disagree. You're responses demonstrate this. Here's another example of how little thought you've put into this: you haven't even standardised what URL schema's should be supported nor dictated any rules on other non-standard extensions to existing URL schema's (such as line numbers) thus even what little you have outlined could differ from one hyperlink supporting terminal to another. The whole thing feels like a pet project someone rushed through rather than something that's been through a proper review. Which is fine if you're also building your own terminal emulator to support this. But when you're spamming every terminal developer out there to support your made up standard, things get a little more serious.

Now I'm not saying we shouldn't have hyperlinks per se but if we are going to agree on a standard then lets get this designed properly.

If you find a security bug that's a new one according to the principle I've just mentioned, sure let us know.

I have done. You've even acknowledged it's a problem on the web. ;)

This document mentions plenty of places where this feature can be useful, convenient, potentially speed up one's work.

Yes, I've addressed them. Debunked them. Found security issues with some of them. But ultimately that list is just "oooh this would be cool" rather than trying to solve actual real world problems. Which is fine if it wasn't for the fact that you're introducing new security concerns for everyone just for the sake of building yourself a new toy.

The world of terminal emulators continuously sees improvements made here and there. Sure you can go back to to a hardware monochrome terminal of decades ago and get your work done from that. Or you can, for each and every single improvement, question "why" that happened. Because it made terminals better for some people. Maybe not for you, but that alone can't be a reason for you to question the feature's usefulness.

I'm actually all for progress. I've written tools to render images and web pages to the terminal (agnostic of which terminal emulator you use too). I'm a heavy user of colours. I've even written my own readline API and $SHELL because I thought "these tools are good, but I could write something that better serves modern computer usage". However one fundamental principle of the terminal is WYSIWYG - what you see is what you get. Hyperlinks make sense on the web but they're an anti-pattern for how terminals work. If you want or need that kind of UI then we have one already (several in fact).

Show us please how you can do it better!

I already did and you edited that part out of the quote! Another idea is maybe use an interactive ls-like tool where you can select the file (terminals do have mouse support too it doesn't have to be a keyboard driven tool) the file and that tool opens $EDITOR for you.

I've written my own UNIX shell and have hotkeys where if you have a file in the command line (eg some-command -config /etc/command.conf and hit [ctrl]+e it opens that file in $EDITOR. My shell also passes filenames around as JSON arrays to get around problems with white space and other characters. The drawback is I've not followed POSIX standards for my shell so it's not like you're just using Bash or similar. However Bash, Zsh and other shells have pretty sophisticated auto-completion engines. You could even use hotkeys to do partial matches of parameters instead of having to type the first few chars and hit [TAB] repeatedly. Not to mention that you could just put quotation marks around a file name.

This is a solid proof to me that you have not familiarized yourself enough with our work, and the possible use cases we (and others) have come up with.

It's funny that because every time I've asked you to expand on your work you refuse to - simply telling me to read the docs (which I clearly have). If you feel I'm missed some fundamental part of your proposal then you're making it impossibly hard to get educated on it but ignoring my every request for clarification.

What I really don't like is how casually you're just waving the security flag, without any concrete issue you could come up with (except pointing to that single one so far that someone else found, and I've responded to). Seems to me we've already put a whole lot more into evaluating the security aspects than you have.

Are you serious!?! I've listed a multitude of concerns that you've not even addressed (and keep avoiding answering):

  • Hyperlinks break the WYSIWYG nature of terminals - actual address isn't clear to the user without additional user intervention which then undermines the convenience factor of hyperlinks
  • They will cause issues for accessibility users
  • There is no formal spec on how terminals will handle URLs - which means even you can have incompatibilities between two terminals that both technically support this "spec"
  • There isn't even a guideline on which protocol schema's you should support

Isn't that enough problems to be getting started with?

It's not the terminal that's vulnerable, it's your careless websites. And those websites are already vulnerable, and any other untrusted site you visit can get you into exploring those vulnerabilities. There's nothing new here.

...or your PC if a terminal doesn't whitelist URL schema's. Or your terminal itself if that happens to have it's own URL schema. You're awfully confident for someone that has a lot of undefined behaviour in the proposal. :)

Also this: https://en.wikipedia.org/wiki/Defense_in_depth_%28computing%29 (you cannot fix every website but you can mitigate it by giving users the tools to evaluate which links they open before they open it)

Of course you're always free to use whichever terminal emulator of your choice, which doesn't support this feature or allows you to disable.

That argument works both ways: you're free to fork any terminal emulator and add this feature for yourself rather than badgering every terminal project out there to implement this on behalf of everyone else - most of whom never required it in the first place.

However I'm actually trying to be a bit more reasonable and saying "lets find a common middle ground that fixes the problems you're trying to solve without introducing anti-patterns into the terminal". Thus far you're just repeatedly ignoring me, saying "you don't understand - go read the docs" instead of actually talking to me. This is VERY frustrating.

Care to actually help, rather than blaming what we've done so far? Because in the latter one I'm not interested at all.

I'm trying! I keep asking you what problems you're trying to solve and you keep "stonewalling" me.

@drudru

This comment has been minimized.

Copy link

commented Feb 6, 2019

Hi @egmontbob

2 things:

  1. If the text for the anchor is blank, should we just use the URL?
  2. Can we move this to a github project? That way we can use the issue tracker.
@lmorg

This comment has been minimized.

Copy link

commented Feb 13, 2019

...and here lies the problem. @egmontkob has this idea they want the community to adopt in everyone else's code base yet is completely unwilling to work with the community in any way apart from his own. Including the repeated requests for the "specification" (if you can call it that) to be moved into a proper repository (it's insane to manage this in a gist) and any requests for further discussions that others have made when they've raised potential areas of exploit. Then @egmontkob wonders why people like myself comment about how this feels more like a pet project than a serious specification that's had community input and rigorous thought put in it. For example @egmontkob been submitting feature requests to a dozen terminal emulators before half the details have been decided upon which has meant @egmontkob has then had to raise follow up requests to the early adopters to fix the support they had literally just added.

A concept like this could work really well but I'm honestly a little scared at how badly - inconsistently and insecurely - this will be implemented while it's under their control.

I know this probably comes off a little hard / negative and I'm sorry for that. If this were just one developer hacking their own terminal emulator / CLI tools then I'd be totally fine with it. However this is someone actively going out and requesting the default behaviour of everyone else's tools be changed. I'm not cool with that if it hasn't followed any due diligence - which this hasn't.

@egmontkob

This comment has been minimized.

Copy link
Owner Author

commented Feb 21, 2019

@drudru

  1. If the text for the anchor is blank, none of the character cells of your terminal emulator will end up having this URL. It's like as if you changed the color to red and then back to the default, none of the cells would end up being red. Obviously the terminal emulation behavior cannot change from what it is now; if an OSC 8 with a non-empty URL is immediately followed by an OSC 8 with an empty one, there's no reasonable way we could make the URL automatically appear printed on the screen. That is, the result is the same as if this pair of OSC 8 was never received. Does this make sense, does this sufficiently answer your question?
@egmontkob

This comment has been minimized.

Copy link
Owner Author

commented Feb 21, 2019

@lmorg

egmontkob has this idea they want the community to adopt in everyone else's code base

No, I don't have this idea, I filed an FYI against multiple terminal emulators, it's up to them to decide what they do. (We first talked in Kitty's bugreport, and you might have noticed, I didn't say a single word about the author's likely decision of not implementing this feature – by the way because he himself thinks it's just not worth it, and not because he shares your security concerns. I'm absolutely fine with his decision of not implementing this feature.) If you take a look at the issue trackers of various terminal emulators from the last couple of years, you'll notice that I'm pretty much by far the most active person in trying to coordinate the work and making sure we at least don't conflict, or preferably go somewhat in the same direction; these issues were a part of this effort, due to the lack of a more appropriate forum for announcing such a new feature.

yet is completely unwilling to work with the community in any way apart from his own.

Whereas you seem to define "community" as your personal opinion, backed up with the great feedback of 1 particular person earlier. Guess what, this specification was discussed among several people, I just happen to be the one who wrote this page and takes a look at it every once in a while. Seeing many great engineers on my side, and you as an "opponent", I firmly reject the claim that I'm "unwilling to work with the community". That's what I'm doing all the time, I just happen to have an opinion that matches the one of many other people I've successfully worked with, just not yours.

Including the repeated requests for the "specification" [...] to be moved into a proper repository

Wow, "repeated" emphasized, where the second one was about a week earlier than your comment.

Guess what. In the mean time I've been working on another specification that literally took me months of work, way more than the hyperlink one. Plus, as a collaboration between various emulators, we set up and announced a group on freedesktop.org specifically for terminal emulator extensions and specifications; the goal being to have a proper place rather than ad-hoc github/gitlab/whatever choices. This was a prerequisite for moving this specification. Moving this one is still on my TODO list, I just haven't gotten there.

@lmorg It's not a secret that in the last 5+ years I volunteered quite a lot into improving terminal emulation. You're free to thank it to me, you're free to use my work without thanking, you're free to avoid it altogether, you're free to constructively critize, you're free to disagree with my decisions. You're even free to non-constructively critize, which, in turn, I'm free to ignore. But when you say things like I'm "unwilling to" do something which I told I would do, I just haven't gotten around to, then you cross a line which I might perhaps (not sure) tolerate from the moment you start to pay for my work, but not until then.

I know this probably comes off a little hard / negative and I'm sorry for that.

No, your attitude that accompanies your arguments comes off as hardly acceptable to me.

If this were just one developer hacking their own terminal emulator / CLI tools then I'd be totally fine with it.

What's the definition of "just one developer"? The emulator I'm working on (not my "own" in the sense of having ownership) has an estimated usage share of about 50% among Linux terminal emulators for most of my work. Two of us from this terminal – the main developer and me – took part in designing this feature. And another one which we co-designed hyperlinks with is probably the second most used terminal emulator for macOS, presumably the most popular among power users. You like it or not, we aren't just some random developers you could ignore when it comes to terminal emulators. As such, I seriously doubt you'd be fine if it was just us implementing this feature, without filing FYIs for others.

However this is someone actively going out and requesting the default behaviour of everyone else's tools be changed.

Feel free to go ahead and convince terminal emulator developers not to implement this feature, or to disable it by default, or to add a confirming dialog before opening such a URL, or whatever. Feel free to let them know what your concerns are. At this point this is the only thing I can recommend to you.

@lmorg

This comment has been minimized.

Copy link

commented Mar 9, 2019

I emphatically disagree with the points you make about how you've listened to my concerns, that there have only been two of us who have raised them, nor that I consider my opinion the voice of the community. None of that is true.

however

I do agree that my attitude has become less than civil and for that I offer my sincerest apologies. I do also sympathise with your reasons for going silent (both with regards to your frustration with me as well as yourself having other priorities).

I do still feel this specification is grossly under-developed to be ready for mass implementation and I cannot withdraw my concern that it has already been pushed onto so many terminal emulators prematurely. However I definitely lost the argument the moment I lowered the tone (I allowed my own frustrations get the better of me).

I hope you don't hold my poor exchange here against me should we ever meet again.

It's not a secret that in the last 5+ years I volunteered quite a lot into improving terminal emulation.

You'll have to excuse my ignorance there. I've not used nor contributed to any of the projects you've worked on so this was the first time I've come across your name. For what it's worth, I've invested a great many years into similar goals - albeit on less widely used tools. So this is clearly a topic that we're both highly passionate about.

@kamazee

This comment has been minimized.

Copy link

commented Mar 11, 2019

Until then, almost all terminal emulators ignore the OSC sequences they don't know about, so emitting OSC 8 will result in the anchor text not becoming a hyperlink, and the target URI getting lost. (Unfortunately VTE prior to 0.48.2 is an exception, it prints the URI plus some garbage too.)

Unfortunately, JetBrains-JediTerm emulator is another exception; it's used in JetBrains IDEs (IntelliJ, PHPStorm, etc.) in built-in terminal. Being given the sequence from the example, it shows nothing:
screen shot 2019-03-11 at 14 16 20. However, it works OK it a \a is used as a string terminator. There is an issue for that.

@glensc

This comment has been minimized.

Copy link

commented Mar 25, 2019

@egmontkob: echo -e is not portable, use printf in your example:

$ printf '\e]8;;http://example.com\e\\This is a link\e]8;;\e\\\n'

at least 3.2.57(1)-release (macOS) prints \e verbatim here.

image

@egmontkob

This comment has been minimized.

Copy link
Owner Author

commented Mar 26, 2019

@modest @lmorg I've added a section about security, I hope you find that sufficient.

@glensc Thanks, fixed.

@egmontkob

This comment has been minimized.

Copy link
Owner Author

commented Mar 26, 2019

Back to the topic of opening the file at a particular line:

There's something obvious we've missed so far. We tried to squeeze the line number into the URL.

Another possibility: The OSC 8 escape sequence has room for arbitrary parameters outside of the URL (such as id as it's defined currently). We could place the line number here.

Advantage: We don't need to have any RFC or any kind of common agreement on denoting the line number in the URL.

This still doesn't solve the problem of desktop integration, though, in case the terminal emulator wishes to build on existing freedesktop intfastructure or such for launching the app. But if a terminal decides to have its own user preference for the apps to be launched, based on MIME type and whatnot, then it could easily integrate passing the line number.

@lmorg

This comment has been minimized.

Copy link

commented Mar 26, 2019

@egmontkob

I've added a section about security, I hope you find that sufficient.

I really like the way you've written that. Sets out of the concerns clearly with potential remedies and does so without diluting the specification.

Thank you

@devkev

This comment has been minimized.

Copy link

commented May 8, 2019

I'm glad that you've added some recommendations for mitigations against deceptive links. However, I do take issue with the upfront declaration.

... we believe security isn't a thing to worry about.

Even though it's likely not what you're trying to say, this projects a terrible attitude. Security must always be given serious consideration. Please consider at the very least rewording this.

This feature doesn't introduce anything that's not already present while browsing the web. [emphasis added]

You can't appeal to the problem's existence in some other context as a justification for its existence in this context, precisely because they are different contexts. A terminal is not a web browser. Unless in a program like lynx, one does not browse the web in a terminal. "Using a terminal" is not the same as "browsing the web". These two activities have different risk profiles, and users' expectations are very different.

For example, in a web browser I am alert to security issues like deceptive links and even relatively mundane things like inefficient JS bogging down my system. I have various extensions and behavioural strategies to help guard against these things. But when I'm in my terminal I'm not thinking about cookies or CSRF. When I run some script that outputs a URL in (what looks like) plain text, I'm not wondering if there's some unseen aspect of the text that means it's not trustworthy. By contrast, on the web I'm aware that there is basically no such thing as "plain text", and I need to be careful because virtually everything could have malicious hidden behaviours.

I'm not saying terminals don't have security issues. I'm saying they have different issues than web browsers. Unfortunately the attitude here conflates these two very different environments. This is a problem not just because of the issues themselves, but because users aren't expecting them and so will not guard against them.

In terms of this specification, would you consider increasing the strength of the various "might", "should", and "could" recommendations in the final two paragraphs? Ideally the spec will require implementors to include at least some of these mitigations - because implementations which don't are insecure by design (which is to say, they are unavoidably vulnerable to these attacks, whereas implementations with mitigations, or without the feature at all (which is historically every terminal emulator), are not), which obstructs any security-minded user from the benefits of this useful feature.

@egmontkob

This comment has been minimized.

Copy link
Owner Author

commented May 9, 2019

Please consider at the very least rewording this.

You're probably right. You might have noticed that I'm not a good technical writer, not even a native English speaker, and don't necessarily feel subtle differences in wording. I didn't mean to say security can be ignored. I meant to say that we firmly believe this feature has no security aspects to be worried of. Do you have any concrete suggestion for the wording?

And by "we firmly believe" I mean at least the 3 developers who designed this feature together: myself with some web development and web security experience, and two other great software developers who are presumably also aware of web development and security basics; but probably none of us are true experts in the topic.

[...] different contexts [...] "Using a terminal" is not the same as "browsing the web". These two activities have different risk profiles, and users' expectations are very different.

No, it's exactly the opposite. The URL is not opened in the terminal but in a browser, and from that moment on it's only the latter one (the browser) that matters.

But when I'm in my terminal I'm not thinking about cookies or CSRF.

That's okay. That's absolutely okay because you don't have to – in fact, you should not; if you had to, we'd be in trouble. The website's creator has to, independently of this new feature of some terminals.


Let's take a closer look at what changed in the terminals with this feature.

Most terminals autodetect onscreen URLs and offer to quickly open them in a browser. When and where did this feature appear? Was security considered? Did anyone consider the risks due to homograph attacks? Did anyone consider the implications of invisible text (either using the SGR 8 attribute, or via setting identical foreground and background colors) in the terminal, thus the visible URL not being the same as the one to be opened? If so, where is such a study available?

(By the way, apart from evaluating the design, has anyone ever evaluated the security aspects of particular implementations? That is, assured that a particular terminal emulator's URL matching logic (or any other feature) cannot be tricked into crashing the emulator, executing arbitrary code, or just taking overly long time?)

The new feature makes it possible to display an arbitrary text instead of the link's target. But the target can still be seen before opening; in fact, it can be seen more reliably, without the risk of having invisible characters. The rest is also pretty much the same as before: the terminal can open a webpage in your preferred browser, using Ctrl+click or some similarly complex (that is: not single click) user action. Nothing fundamentally new.


Let's take a closer look how this feature relates to many other features of your desktop.

You can have a standalone e-mail client, viewing an incoming HTML message with a hyperlink. You click on it, it opens in your browser. You can open a DOC file in a word processor and have hyperlinks in them with misleading visible text. You can have a PDF opened in a PDF reader, have links in them, they also open in your browser. I'm pretty sure the list goes on. Don't you think the creators of these software evaluated the security aspects? Is any such document available somewhere publicly? Are you okay with these features, or do you plan to speak up (or have you already) against?

I'm quite certain that about 20 years ago one could fire up a word processor, a PDF viewer, a browser, but they weren't connected this way, you couldn't get the former ones open a URL. (If you could, just go back a little bit further in time.) At one point this feature appeared. People got used to it, people changed their expectations about these software, their knowledge about their capabilities. And that's okay.

Let's step out of the desktop world. You might type in a URL you saw on a flyer. You might scan a QR code from your mobile and open that. On my Android phone an app can open an arbitrary URL in the browser, without requiring any special permissions. Don't you think these features were evaluated by security experts?

What happened now? The most that you could say is that the terminal also joined these apps and other URL sources. Why would this be a problem? The world is changing; document editors/viewers have changed and users got used to it, maybe now the terminal changes and users will get used to. But to be fair, terminals have joined these kinds of apps a long time ago, now only some minor cosmetics changed: the target is not the piece of text you click on, but is shown elsewhere.


All these existing practices prove that making the web secure is the responsibility of the browser and the server (in case of well behaved sites; or the browser only to detect deceptive servers). Sure there's a lot more to this: layers and components they build on and additional infrastructure (TLS, CAs, caches/proxies, DNS, routing; the maintained list of malicious sites etc.). But one thing is pretty clear: it's not the responsibility of whoever has a link and decides that it should be opened. To translate it to our context: With the introduction of this hyperlink feature, security is not the terminal's responsibility, but is already supposed to being taken care of elsewhere.

That being said, multiple lines of defense is nice to have, and accordingly, terminals clearly show you upfront what's about to happen.


increasing the strength of the various "might", "should", and "could" recommendations in the final two paragraphs? Ideally the spec will require [...]

Quite the opposite.

We are not some authority, we are not going to give out certificates to terminals who implement this specification properly. So there'd be no use for it. What if I change a "should" to a "must", and a certain implementation just skips that bit? There's nothing we could do about it. So it's pointless. In that case you should speak up in the bugtracker of that particular software.

In fact, since apparently no matter how I try to maintain this page it's not good for some folks, I'm tempted to reduce it to its bare minimum: just stating what gnome-terminal does (and promises to do in the future), and nothing more. No usage ideas, no list of supporting apps, no security considerations, and no user feedback forum. This feature has been designed and implemented 2 years ago by a few of us; it's me alone who wrote up and maintain this page and try to address the comments. I have no capacity to try to defend the design against some vague unspecific waving of the security flag.

@teto

This comment has been minimized.

Copy link

commented May 27, 2019

Random comment: Would love to see this in neovim. With LSP taking off (thanks coc.nvim), I finally have proper documentation in neovim but urls take way too much space.

@egmontkob

This comment has been minimized.

Copy link
Owner Author

commented May 27, 2019

@teto Please follow up with nvim developers about how you imagine this to be implemented.

Note that this specification does not specify a document format. This specificatoin specifies a communication format between the terminal and other apps. Sure you can store such escape sequences in a file, but then it'll only be useful if you "cat" that file to a terminal, not if you view it in various text editors, browsers etc.

If you have some documentation in plain text or some similarly simple format, and you're worried about URLs being too long in the file, this specification does not help you out. You might adopt the convention of saying "[1]" and such in the text flow, and place the actual URLs in the footer. Or you might want to look at lightweight formatting standards (and corresponding tools) such as Markdown.

@teto

This comment has been minimized.

Copy link

commented May 28, 2019

For instance in markdown documents, I would like to use conceal to replace [Documentation](file:///nix/store/h657gfx9qsw7x6hirzw7s6xcyisgwjvz-ghc-8.6. 4-doc/share/doc/ghc/html/libraries/base-4.12.0.0/Control-Concurrent-MVar. html#t:MVar) with Documentation and still have the terminal recognize it's a link. I believe it's a valid usecase right ? It wouldn't be saved into the file ofc

@egmontkob

This comment has been minimized.

Copy link
Owner Author

commented May 28, 2019

This sounds like a great use case. I'm not familiar with (n)vim to tell whether it's already possible via some config of Conceal, or needs a new feature to be implemented by the developers first. I suggest that you ask around on vim forums, or contact its developers if it depends on a new feature.

@teto

This comment has been minimized.

Copy link

commented May 28, 2019

oh yeah it's possible with conceal. I would like to implement it for neovim but sadly lacks the time. I might open a ticket instead (until I get more time :) )

@devkev

This comment has been minimized.

Copy link

commented Jul 2, 2019

I'm going to try one last time.

Consider 3 different terminal implementations:

  • TermA: implements only automatic URL linkification, and none of this spec. This is the baseline.
  • TermB: a fork of TermA which adds support for this spec, but without any mitigations.
  • TermC: a fork of TermB which adds the mitigations.

TermB is inherently insecure by design, because it permits the following situation (which TermA and TermC are robust against).

Normal:

  • Some software's installer uses the following command to output a simple message in the terminal:
    echo 'Thanks for installing! For info on how to configure, go to https://docs.example.com/'
    
  • Behaviour:
    • TermA: users see a link to https://docs.example.com/ in their terminal. When clicked their browser loads the docs.
    • TermB: users see a link to https://docs.example.com/ in their terminal. When clicked their browser loads the docs.
    • TermC: users see a link to https://docs.example.com/ in their terminal. When clicked their browser loads the docs.

Attack:

  • An attacker manages to publish a rogue version of the software. The installer now runs:
    printf 'Thanks for installing! For info on how to configure, go to \e]8;;http://install-0day-rootkit-and-then-redirect-to-docs.com/\e\\https://docs.example.com/\e]8;;\e\\\n'
    
  • Behaviour:
    • TermA: users see a link to https://docs.example.com/. When clicked their browser loads the docs. The attack has failed.
    • TermB: users see a link to https://docs.example.com/. When clicked their browser loads http://install-0day-rootkit-and-then-redirect-to-docs.com/, which installs malware then redirects to the legitimate docs site. The user is likely unaware that their system has been infected. The attack has succeeded.
    • TermC: users see a link to https://docs.example.com/, but with some visual indication that this is not where the link goes to. When the user hovers over the link they immediately see that the actual URL is suspicious. The user does not click the link. If they click by accident or in a rush, a confirmation popup asks if they are sure they want to load the alternate URL, at which point the user notices the problem and cancels. The attack has failed.

Conclusions:

Users of TermB are vulnerable to attacks that users of TermA and TermC are not vulnerable to.

Whenever TermB users see a link that looks like a URL (a "URL link"), when considering clicking it they need to be concerned about this vulnerability (otherwise they are unknowningly playing Russian Roulette). Users of TermA and TermC do not have this concern (in TermA, URL links can only go to that URL, while in TermC users are fully informed about URL links), they can click URL links with confidence about what their browser will load.

To put it another way, the introduction of this spec without mitigations (aka TermB) has introduced additional security concerns for the user. The introduction of the mitigations (aka TermC) removes these security concerns.

This is why the mitigations are essential, and why in their absence the spec is insecure.

In the same way that this spec tries to be a common point of reference for the feature itself, so that each terminal implementation doesn't do something different --- so too it ought to mandate these essential mitigations, so that each terminal implementation doens't implement some different (possibly ineffective) set of mitigations, including possibly none at all (aka TermB) because the spec incorrectly asserts that no new security concerns are introduced.

@cben

This comment has been minimized.

Copy link

commented Jul 2, 2019

An attacker manages to publish a rogue version of the software. The installer now runs: printf ...

So the user already executed rogue code, game over!

However, relevant attack scenarios are cat README, curl example.com/README.txt etc where one expects that merely reading some untrusted content can't harm them (and doesn't, until they click...)

@devkev

This comment has been minimized.

Copy link

commented Jul 2, 2019

So the user already executed rogue code, game over!

Not quite, because the user may have executed that code on a different system, e.g. a remote host via ssh, or in a throwaway virtual machine. This vulnerability additionally exposes their local workstation to attack. Normally there is no way for code running inside an ssh session to influence the ssh client system in any way (assuming no port/agent/X11 forwarding).

where one expects that merely reading some untrusted content can't harm them (and doesn't, until they click...)

Yes, this is a better demonstration - in TermB you can no longer trust what used to be trustworthy (specifically, that URL links are "just text" and clicking will take you to the expected location).

@XVilka

This comment has been minimized.

Copy link

commented Jul 4, 2019

@egmontkob what about converting this gist into the repo too? Like true colors one? And we can think about adding tests to the tests repo.

@egmontkob

This comment has been minimized.

Copy link
Owner Author

commented Jul 9, 2019

[devkev] When clicked their browser loads http://install-0day-rootkit-and-then-redirect-to-docs.com/, which installs malware then redirects to the legitimate docs site.

Tell me again, how can visiting a malicious site compromise your system? It can't. Or if it can, isn't that a bug in the browser, rather than in the terminal? Or if you still insist we're talking about a design bug in the terminal, then isn't it also a design bug in all other kinds of applications that let you open a URL without showing it to you upfront, including PDF readers, email clients, web browsers (e.g. on mobile devices), social media apps etc.?

Seriously, if I click on a link on my mobile phone's browser, email clilent, social media client etc., it opens without telling me upfront what that URL is. It's good enough for major mobile OS / browser / social media / etc. vendors that do a thorough security review of their products. So again, why is the terminal so much different that it's not okay here?

Just as having to show the URL upfront isn't part of HTTP / HTML / etc. specifications, it doesn't belong to this specification either. Even if it belonged here, there was no way for us to enforce it (including for touchscreens, mobile apps etc.). It's here as a recommendation, and if you have problem with a particular implementation not respecting this recommendation, you should talk to that implementation. (In the mean time, I'd love to see the security reports you send to various mobile browsers, email clients, social media apps, PDF readers etc. about this same issue.)

The questions I asked were rhetorical ones. I do want this thread to end right here and right now, for good.

@egmontkob

This comment has been minimized.

Copy link
Owner Author

commented Jul 9, 2019

[XVilka] what about converting this gist into the repo too?

Now that we get email notifications about comments, I have no plans to convert it to a repo. In fact, I'm tempted to convert it to a static page without the possibility of commenting. If I get one more comment questioning the security of the feature in general, without pointing out an actual bug in this very specification (not elsewhere!), I'm probably going to do this.

@matthewbauer

This comment has been minimized.

Copy link

commented Aug 15, 2019

Emacs users can try my package for handling this:

https://github.com/matthewbauer/comint-hyperlink

@stuaxo

This comment has been minimized.

Copy link

commented Aug 27, 2019

Back to the topic of opening the file at a particular line:

There's something obvious we've missed so far. We tried to squeeze the line number into the URL.

Another possibility: The OSC 8 escape sequence has room for arbitrary parameters outside of the URL (such as id as it's defined currently). We could place the line number here.

+1 on this. It would be good if this worked with ranges from the beginning.

For integration on the other side, we could work out the best way to go from there, with some mechanism to map these to launch parameters for whichever text editor you want to launch.

@cben

This comment has been minimized.

Copy link

commented Aug 28, 2019

There are already integration mechanisms to launch programs passing them the URL. Terminal passes URL to, say, xdg-open, which routes it to assigned application.
If line number becomes a separate field in the escape sequence, you'd need an out-of-band way to pass it to editors...

There exists RFC 5147 defining URL fragments for text/plain: #line=100, ranges #line=100,200 and some other addressing ways.
I'm not sure what, if anything, implements it ☹️. Chrome and Firefox don't. Still, IMHO it's a sensible convention 👍.

@stuaxo

This comment has been minimized.

Copy link

commented Aug 28, 2019

This sounds good to me. Big +1 on using RFC 5147 as its a preexisting standard.

@egmontkob

This comment has been minimized.

Copy link
Owner Author

commented Aug 30, 2019

@cben Thanks a lot for pointing to this RFC, I wasn't aware of this. Probably it is the way to go. Although...

It has a really unusual way of representing lines. The basic primitive is "line position" (I'd call it "line boundary"), and built on top of this it defines "line range" between two such points. So what you would normally call "line 10" of a text file becomes "#line=9,10". I'm a bit afraid it's prone to be implemented incorrectly by various tools, but we can always file bugs to get them fixed.

The document correctly points out that the definition of "character", and (if we allow non-ASCII-comatible encodings, such as UTF-16) even the definition of "line" depends on the "encoding" (sometimes loosely called "charset"). Yet, it makes the mistake of this piece of information not being a first class citizen, it's a part of some integrity check information. Integrity check is not only needless in many cases, but mandating it is straight harmful for us for various practical reasons. E.g. a grep reasonably wants to emit all the matches as it encounters them without having seen the entire file yet, thus not being able to tell the file's length or checksum; although for the overall expected behavior of this ecosystem grep should pass on the encoding it assumed for the file (presumably the process's locale), and the opening application should also respect it.

(As an absolutely side note, this RFC allows the use of only one particular checksum algorithm, one that is no longer considered safe.)

It's unclear to me what is the exact definition of "character". How do combining accents, all those weird stuff happening in Arabic, Devanagari, Thai etc. scripts, various control characters etc. count? A reasonable assumption is probably "codepoints according to the encoding". What about UTF-16 surrogate pairs? What about encoding errors? To make it even more confusing, 4.1 says "Independent of the number of bytes or characters used to represent a line ending, each line ending MUST be counted as one single character." So is CR+LF to be counted as 1? Also it's a faulty recursive definition of multiple characters becoming one character, such a loose wording shouldn't be present in RFCs.

At this point, I only see this standard suitable for us if we limit its use to ASCII-compatible encodings and single line / line ranges only. An updated version of this document that addresses the above concerns would be required to move forward with encodings (there needs to be a way to specify the encoding without any other piece of additional information) and with character positions (depends on the encoding, plus also needs a clear definition around combining accents, special control chars, surrogates, encoding errors, CR+LFs, whatnot).

@stuaxo

This comment has been minimized.

Copy link

commented Aug 30, 2019

Thanks for reading this in detail, I have to admit to skimming the part about ranges as it seemed complex.

So I'm not as keen on the RFC any more.

It could be worth starting a new page to collect information on existing solutions to see what they do + their pros and cons.
(for instance, vscode has it's own flavour of urls, then there are also selection schemes used for github, bitbucket and probably gitlab).

I'm definitely persuaded on the advantage of the line/selection info being inline now though.

You're probably right, separating the location from the selection makes sense + we could start with just the location part of this RFC.

@davidmalcolm

This comment has been minimized.

Copy link

commented Oct 10, 2019

Thanks for your work on this.

I've just added support for this to GCC (for the future release of GCC 10), so that the [-Wname-of-option] text that we emit at the end of each compilation warning becomes a hyperlink to GCC's documentation for that warning.

@egmontkob

This comment has been minimized.

Copy link
Owner Author

commented Oct 10, 2019

@davidmalcolm It's lovely to hear it!

In the mean time I'm somewhat disappointed that it's not done for filenames. Half of the story is the concerns raised about the hostname part, which ideally should've been raised here and discussed properly, rather than rejecting it because someone had concerns. The other half of the story is that this feature would be really useful if we figured out how to handle line numbers, which we haven't yet.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.