EmbeddedLanguageLSPExpansion.md

Expanding LSP to Support Embedded Languages

This document details a solution to enable embedded languages to provide fully reusable, reliable and high quality tooling experiences in the form of language servers (LS) through expansion of the language server protocol (LSP).

Table of Contents

• Problem Background
• Solutions
  • Historical solutions
  • Proposed solution
• Examples and scenarios
  • Document State
  • Language Feature
  • Diagnostics
  • External Interactions
• LSP Spec Expansion
• Open Questions

Problem Background

An embedded language is a language that hosts other languages inside of it. Today there are an increasing number of embedded languages, a few examples to bring to light are: HTML (CSS / JavaScript), PHP (HTML), Visual Basic (XML), React/JSX (HTML) and Razor (HTML / C#). As tooling ecosystems grow towards supporting the latest LSP, embedded language tooling has been left to fend for itself. In its current form, embedded language tooling often re-hosts innerworkings of their embedded languages or rely on platform dependent implementations to provide quality embedded langauge editing experiences for developers. As LSP currently exists it defines contracts that are appealing to embedded language tooling implementations but lacks the tools or guidance to hook experiences together.

VSCode has been the front runner in giving embedded languages a model to follow; however, even their guidance results in a platform dependent model that puts a significant burden on embedded language client implementations to build out complex document state, synchronization & request forwarding mechanisms.

To summarize, the current LSP spec & guidance result in embedded languages needing to build out highly complex, platform dependent infrastructure to enable any cross-platform, highly functional experiences.

Solutions

Historical Solutions

Note: This section is not necessary to understand the embedded language LSP spec expansion, it's just historical information to better understand motivations for the proposed solution below.

There are two pre-existing approaches to building out an embedded language LSP solution. One which re-hosts all sub-languages in a singular language server and another which splits its logic into two components, a re-usable language server and a platform specific component that understands how to intercept and delegate requests for common features, i.e. completion, hover etc.

The Re-Hosting Approach

Re-hosting was the first approach at having an embedded language LSP compliant experience. In its infancy embedded languages would re-host all sub-languages underneath them in order to provide tooling experiences. For instance, in PHP it'd re-host HTML, CSS and JavaScript language services (or even sometimes language servers) to enable language specific experiences. This enabled PHP to provide HTML completions when a user would type < inside of a document.

So lets analyze what handling a feature in this model might look like for an index.php document with content (pipe denotes the cursor):

<div>
    <?php echo '<strong>Hello World!</strong>'; ?>
    |
</div>

1. User types < and the client (i.e. VSCode) sends a textDocument/completion request to PHP Language Server (PHPLS) at line 2 (0 indexed), character 5 (< now exists in the buffer). We'll represent this with (2, 5)
2. In PHPLS' completion handler it detects which language is applicable at (2, 5). It determines that two languages should return completions, HTML and PHP. HTML for the various p, strong etc. elements and PHP for ?php.
3. PHPLS updates its virtual document for the current HTML content that corresponds to index.php, lets call this index.php.html:

   <div>
       <
   </div>

   NOTE: The <?php ... > piece is ommitted because technically that line isn't relevant to HTML.
4. PHPLS finds out where (2, 5) exists in index.php.html. Results in (1, 5)
5. PHPLS hands over its index.php.html document content to an HTML language service and asks it for completions at (1, 5).
6. The HTML language service responds with [p, strong, ...]
7. PHPLS then aggregates HTML's completions with its own and returns an entire list of completions [p, strong, ?php, ...]
8. Client presents the combined completion list.

Throughout the above flow there are several concerns that were glazed over which make the re-hosting approach troublesome. Here are some of the drawbacks of re-hosting:

1. All sub-language interactions are invisible to client extensions. For instance, what if there was an emmet extension on the client and the user was expecting to be able to use it?
2. Settings don't translate. Embedded languages have their own set of settings on how things format, which things get offered in completion etc. Re-hosting can't utilize pre-existing settings so it requires users to re-implement every setting.
3. Language detection results in highly coupled flows. In the above example, what would have happened if the user was in a <script> block? The host language needs to understand every sub-language underneath it OR that sub-langauge needs to also re-host all of its inner workings so the correct requests can make it to corresponding language services.
4. Dependent on sub-language architecture. If the PHP language server was written in PHP but the HTML language service was written in NodeJS, how do they communicate? This problem gets exacerbated when more sub-langauges come into the mix and then they all need to work in a cross-platform manner.
5. Updates become difficult and introduce fragility. What happens when a new version of HTML or CSS or JavaScript comes out? You now need to update the underlying language services and ensure all interactions with them work as they did in the past.

	Supported
Platform Agnostic	✔️ ❔
Flexible	❌
Maintainable	❌ ❔
Future proof	❌
Feature Rich	✔️ ❔
Extension Friendly	❌
External Server Interaction	❌

The ❔ indicates that there's some uncertainty based on the various embedded language requirements. Aka, not all embedded language services may be cross-plat or be feature rich and although highly unlikely, some hosted languages may actually be maintainable if they never need or want to update.

The Delegation Approach

A delegation / request forwarding approach quickly followed the re-hosting model in an effort to solve some of the support matrix flaws that re-hosting introduced. This approach LSP compliant approach consists of two pieces:

1. A delegation component which is typically in-proc to the corresponding platform to enable request forwarding, language synchronization and aggregation of sub-language results.
2. A language server with knowledge of its immediate sub-languages. For instance a PHP language server in this model would know which portions of its document were HTML portions but not the CSS/JS portions.

Both of these components work together to orchestrate the end-to-end experience users experience when working in the embedded language. Lets analyze what handling a feature in this model might look like for an index.php document with content (pipe denotes the cursor):

<div>
    <?php echo '<strong>Hello World!</strong>'; ?>
    |
</div>

1. User types < and the client (i.e. VSCode) dispatches two textDocument/completion requests at line 2 (0 indexed), character 5 (< now exists in the buffer). We'll represent this with (2, 5)

   1. One to the PHP Language Server (PHPLS)
   2. A second to the delegation component. We'll denote this as the PHP Delegation Component (PHPDC).

2. In the PHPLS' completion handler it determines that it should provide a PHP domain specific completion: ?php. It returns it.

3. In the PHPDC completion handler it authors a custom LSP request to ask PHPLS what immediate sub-language exists at (2, 5) and at what location. It responds with "HTML" at (1, 5)

4. PHPDC ensures that an HTML document exists on the client exists to represent the HTML interactions for index.php. PHPDC assigns it an addressable uri index.php.html

5. PHPDC uses custom LSP requests to acquire the virtual HTML content from PHPLS for index.php. PHPLS returns:

   <div>
       <
   </div>

   NOTE: The <?php ... > piece is ommitted because technically that line isn't relevant to HTML.

   1. PHPDC updates the index.php.html with the received content.
   2. The client sees a document content update to index.php.html and notifies all HTML language servers of the updated content

6. PHPDC constructs a new textDocument/completion request pointing to index.php.html at (1, 5) and asks the client to query all associated language servers and for the client to aggregate the results. Active HTML language servers respond with [p, strong, ...]

7. PHPDC receives the completion result and returns it as its own

8. Client receives [ ?php ] from PHPLS and [p, strong, ...] from PHPDC, combines the two completion lists and then presents the result.

NOTE: Fast forward past the delegation approaches initial unvieling there have been variants that help further reduce the complexity of various interactions. Those variants do things like provide a single delegation language server that's platform dependent and tries to play the ultimate role of delegator (even for the top-level language, in this case PHP). These routes have solidified the delegation approach as an extremely strong alternative to the re-hosting approach but unfortunately still have some of the drawbacks listed below.

Throughout the above flow there are several concerns that were glazed over which make the delegation approach difficult. Here are some of the drawbacks of delegation:

1. Document content updates leave opportunity for de-synchronization. Since the document source of truth is coming from PHPLS yet embedded language document content is updated via requests or notifications there is not a single source of truth to manage document updates in a way that both language servers can play nicely. Given that we only have one buffer representing the embedded language this type of synchronization becomes paramount and to combat this the delegation approach will ocasionally throw away requests if document versions have diverged significantly to protect the user.
2. PHPDC requires that the platform it's built on has two capabilities:
   1. Ability to create readonly, hidden documents
   2. Programatic LSP invocation. Aka, the ability to query for "completions" or "hover" etc. for a document.
3. PHPDC has to run in-proc to delegate requests binding it to the corresponding clients architecture (i.e. PHPDC would be C# for Visual Studio or JavaScript in VSCode)
4. PHPDC ultimately is meant to be unintelligent plumbing for the platform but is innately complex.

	Supported
Platform Agnostic	❌ ❔
Flexible	❌
Maintainable	✔️
Future proof	✔️
Feature Rich	✔️ ❔
Extension Friendly	✔️
External Server Interaction	❌

The ❔ indicates that there's some wiggle room. PHPLS is platform agnostic but PHPDC is not and due to the complexity of PHPDC the feature richness does have some restrictions due to the need to throw out requests occasionally.

Proposed Solution

In all embedded language solutions there has always been the idea of querying sub-language tooling capabilities by creating hidden documents that contain language relevant data; we call these hidden documents "virtual text documents". For instance, lets take the following HTML scenario:

<html>
    <head>
        <script type="text/javascript" src="site.js"></script>
        <script>
            console.log('The current site name is: ' + siteName);
        </script>
    </head>
</html>

This document will typically result in several virtual text documents to represent the JavaScript, CSS and any other sub-language that may exist in it. In this example you could imagine the JavaScript virtual text document could look something like:

// From site.js
var siteName = "My SiteName"

// From host file
    console.log('The current site name is: ' + siteName);

This way when you're typing in the second <script> block you get all the JavaScript completion items from site.js (e.g. siteName) even though they aren't directly present in the HTML file.

As you can imagine these virtual text documents drive nearly every embedded language interaction but aren't well defined in the LSP landscape. Therefore, the first step to building a end-to-end solution for embedded languages in LSP is to define what it means to work with a virtual text document. These are the three focal points when doing anything with virtual text documents:

1. Managing virtual text document state
2. Querying virtual text document data
3. External virtual text document interactions

Manging Virtual Text Document State

Virtual document state (i.e. what content should the JavaScript hidden document have in an HTML scenario) is managed via workspace/applyEdit requests from server -> client. Ultimately it enables a server to create, edit or delete a virtual text document when it sees fit.

• Example
• Spec

Querying Virtual Text Document Data

Once virtual text documents are available the next logical step is to ask those virtual text documents for information. For instance, when a user hovers over a portion of an HTML document it's the job of the "host" language, i.e. HTML, to potentially delegate/query that hover request to the appropriate virtual text document (JS, CSS etc.). Virtual text document data can be queried via requests from the server to the client for commonly known LSP features like textDocument/completion, textDocument/diagnostic etc. This method effectively allows a server to treat a client as another language server where the responsibility of the client is to delegate and translate the request to all applicable language servers in their version of LSP and then aggregate the responses together.

• Completion Example
• Diagnostics Example
• Spec

External Virtual Text Document Interactions

When language servers get in the business of creating virtual text documents they then have to worry about what it means for other language servers to take those documents into consideration when returning results. For instance, what happens if you or someone else tries navigating to or even editing a virtual text document (it's hidden)? What happens if an operation like rename is performed on a non-virtual text document that should have host document reactions? Answering these questions results in two to three top level LSP types that can externally interact with virtual text documents: WorkspaceEdit and Location/LocationLinks which are represented in the go-to-X, find all references, rename and workspace edit applications requests.

When a client identifies that a location or workspace edit applies to virtual text documents from an external server (a server that didn't create the virtual text document) it's the clients' responsibility to send translation requests to the language server that owns the virtual text documents. This translation request gives the language server the opportunity to filter, remap or add edits/locations prior to applying the final result.

• Examples
• Spec

Conclusion

By standardizing what it means for embedded languages to create virtual text documents, query their data and control external interactions with them, embedded langauges become a first-class citizen in LSP enabling them to provide feature rich, reliable and reusable tooling experiences.

	Supported
Platform Agnostic	✔️
Flexible	✔️
Maintainable	✔️
Future proof	✔️
Feature Rich	✔️
Extension Friendly	✔️
External Server Interaction	✔️

Examples and Scenarios

Document State Example

When the host language (i.e. HTML, Razor etc.) changes it can "update" its corresponding embedded language representations via workspace/applyEdit requests from server -> client. This path enables all embedded document state management to transfer across platforms, be visible to extensions and most of all be standardized.

• Opening is done via CreateFile workspace edits with a virtual property set to true. Spec
• Changing is done via normal workspace edits with document changes. Spec
• Closing is done via DeleteFile workspace edits. Spec

Here's an example in Razor (C# / HTML are sub-languages) where a user opens a file, types an @ and then closes the file (@ transitions into C#). It represents what happens for the C# embedded language (excludes the HTML embedded language for simplicity).

Important: For all the details on virtual document state management check out the full spec below.

Language Feature Example (Completion)

A host language server can present embedded language features by delegating to well-defined contracts on the client to forward / delegate requests to embedded language documents. This way the host language server can present the responses as a combined result from the originating language server. This path enables language servers to support embedded language interactions in a cross-plat, extension friendly way while simultaneously eliminating synchronization complexity/limitations.

Here's an example of a user in an open Razor document typing @ to get C# compeltions (@ transitions into C#):

Important: For all the details on virtual document language features check out the full spec below.

Diagnostics Example

A host language server can provide embedded language diagnostics by delegating to the document diagnostic endpoint on the client to ask embedded language servers for sets of diagnostics.

Here's an example of a user in an open Razor document having just typed the @ character (invalid on its own in Razor). Typically this produces two diagnostisc, one from Razor saying you need content after the @ and one from C# about missing C# content; however, in the example below only one is returned because Razor filters out the C# diagnostic in favor of the Razor one:

Important: For all the details on virtual document diagnostics check out the full spec below.

External Interactions Examples

A host language server can serve as a translator for WorkspaceEdits and Location/LocationLinks that are pointed towards virtual text documents it owns. Common scenarios include:

• User renames a symbol in a non-virtual text document that happens to exist in a virtual text document.
• User finds references on a symbol that also is used in a virtual text document.

In both of the above the host language server typically wants to either remap the result to a location in the host text document, throw it out completely or add additional results.

Remapping Interaction Examples

This section contains examples for when a host language server wants to remap edits or locations that are pointed towards virtual text documents.

In these two examples there are two top-level files:

1. Person.cs which contains a class for a Person object in C# syntax:

   public class Person
   {
       public string Name { get; set; }
   }

2. Users.razor which renders a list of people in Razor syntax:

   @foreach (var person in People)
   {
       <p>Name: @person.Name</p>
   }

And a C# virtual text document to represent the C# for Users.razor, Users.razor.cs:

public partial class Users
{
    public void Render(RenderTreeBuilder __builder)
    {
        foreach (var person in People)
        {
            __builder.Add(person.Name)
        }
    }
}

Rename Interaction Example

User attempts to rename the Name property of Person to FirstName via Person.cs:

Find References Interaction Example

User attempts to find references of the Name property of Person via Person.cs:

Note: Same flow applies if LocationLinks are returned. Instead it uses the translate/locationLink endpoint

Additive Interaction Example

This section is meant to detail what it takes to perform more complex operations where an edit would be added to a result for an embedded langauge when initiated on a non-virtual text document.

In this examples there are three top-level files:

1. Address.cs which contains a partial class for a Address class in C# syntax:

   public partial class Address
   {
       // A [Parameter] makes it so if someone tries to utilize the "Address" object 
       // in HTML they can via <Address StreetName="Broadway" />
       [Parameter] public string StreetName { get; set; }
   }

2. Address.razor which utilizes the StreetName property from the Address.cs file. The Razor syntax always generates its C# classes as partial classes so that code can be written in C# or Razor and utilized in either:

   <p>Street name: @StreetName</p>

3. Company.razor which uses Address.razor custom component:

   <Address StreetName="Pine Street" />

And a __SymbolHelper.cs C# virtual document that the Razor language server uses to aid in understanding when external symbol interactions occur. It writes down all of its symbols in a mappable way so that when they're changed via a rename, or reference etc. it knows how to map those interactions to other languages like HTML:

public class __SymbolHelpers
{
    public void __HelperMethod()
    {
        var v1 = typeof(Address); // Maps to Address.razor and usages of <|Address| .../> 
        var v2 = nameof(Address.StreetName); // Maps to <Address |StreetName|="..." />
        var v3 = typeof(Company); // Maps to Company.razor and usages of <|Company| .../>
    }
}

And of course there's C# virtual documents to represent Address.razor and Company.razor's C# but their content is not relevant for this example.

User attempts to rename StreetName to Street via Address.cs with the expectation that it will not only rename the C# representation but also the HTML representation in Company.razor:

The interesting result of this would modify Company.razor to be:

<Address Street="Pine Street" />

Note how the HTML attribute StreetName changed to Street even though the rename was initiated on a C# property

Virtual Text Document Support LSP Spec Expansion

Virtual Text Document State Management

Virtual document state is managed via workspace/applyEdit requests. The VirtualTextDocumentClientCapabilities define client capabilities the editor provides in relation to managing virtual text document state.

Client Capability:

• property name (optional): workspace.workspaceEdit.virtualTextDocument
• property type: VirtualTextDocumentClientCapabilities defined as follows:

/**
 * Client capabilities specific to virtual text documents
 */
export interface VirtualTextDocumentClientCapabilities {
    /**
     * Whether the client supports renaming virtual documents
     */
    rename?: boolean;
}

Creating Virtual Text Documents

Client Capability:

• property name (optional): workspace.workspaceEdit.virtualTextDocument
• property type: VirtualTextDocumentClientCapabilities

Virtual documents are opened / created via a workspace/applyEdit request with a CreateFile document change that has virtual set to true:

Example:

{
    documentChanges: {
        [
            {
                kind: "create",
                uri: "file:///some/path/that/doesnotexist.cs",
                options: {
                    overwrite: true,

                    // This is the magic portion of the CreateFile object 
                    // that indicates that the created file should be 
                    // virtual
                    virtual: true
                }
            }
        ]
    }
}

Creating a virtual text document results in the client issueing a textDocument/didOpen to all other applicable language servers. That textDocument/didOpen request has its DidOpenTextDocumentParams.virtual property set to true.

Virtual text documents have a few characteristics that set them apart from normal text documents:

1. Viewed as normal documents to other language servers
2. Never written to disk and shouldn't be shown in the file explorer.
3. Can only be edited by the server that created them indirectly via workspace edits.
4. Immediately trigger textDocument/didOpen requests upon creation
5. Trigger textDocument/didClose requests when deleted
6. If the server that created them disapears all virtual documents created from said server get deleted/closed

Trying to create virtual text documents with DocumentUris that already exist without overwrite: true will result in failure to apply workspace edits.

Changing Virtual Text Documents

Client Capability:

• property name (optional): workspace.workspaceEdit.virtualTextDocument
• property type: VirtualTextDocumentClientCapabilities

Virtual documents can be changed by the server that created them via a workspace/applyEdit request with a corresponding changes or documentChanges. This results in the client following standard textDocument/didChange handling which results in it notifying all appolicable language servers of the change.

Example:

{
    changes: {
        "file:///some/path/that/doesnotexist.cs": [
            {
                range: {
                    start: {
                        line: 0,
                        character: 1
                    },
                    end: {
                        line: 0,
                        character: 4
                    }
                },
                newText: "Hello"
            }
        ]
    }
}

If the edit is provided via documentChanges the version property of the provided textDocument should not be provided. Edits to closed or unowned virtual text documents result in failure to apply workspace edits.

Attempting to edit unowned or non-existent text documents results in failure to apply worskpace edits.

Deleting a Virtual Text Document

Client Capability:

• property name (optional): workspace.workspaceEdit.virtualTextDocument
• property type: VirtualTextDocumentClientCapabilities

Virtual documents are closed / deleted via workspace/applyEdit request with a DeleteFile document change:

Example:

{
    documentChanges: {
        [
            {
                kind: "delete",
                uri: "file:///some/path/that/doesnotexist.cs"
            }
        ]
    }
}

Deleting virtual text documents gets translated into textDocument/didClose notification to all other applicable language servers.

Deleting virtual text documents that are unopened or unowned result in failure to apply workspace edits.

Querying Document Data

Virtual text document data can be queried via requests to the client for commonly known LSP features. If a client supports data querying for a language feature its client capability will have a queryable property set to true.

For instance, if completion can be re-queried its CompletionClientCapabilities will have queryable set to true. It can then be queried by the server performing a JSONRPC request to textDocument/completion with a valid CompletionParamsobject.

export interface CompletionClientCapabilities {
    ......

    /**
     * Indicates whether the client supports server -> client requests for 
     * the textDocument/completion request.
     */
    queryable?: boolean
}

Diagnostics

Virtual text document diagnostics have many implications to them. For instance, when languages interchange within the same line or construct do all diagnostics still make sense? Do their ranges map directly 1-to-1 to another document location? In many languages diagnostics don't all translate to a top level document and their ranges position and length get modified. To account for this, virtual document diagnostics rely on LSP's pull based diagnostic approach. Aka the ability for the client to request diagnostics for a document (client -> server) and also the ability for a server to request diagnostics for a virtual text document (sever -> client).

Workspace diagnostics are not supported for virtual documents and therefore shouldn't be displayed by the client or provided from the language server.

Feature	Server -> Client Method	Parameters	Response	VSCode Command
Document Diagnostics	textDocument/diagnostics	TBD	TBD	(New) vscode.executeDocumentDiagnosticsProvider

As for textDocument/publishDiagnostics notifications from server -> client. These aren't fully supported by virtual text documents. However, a client can choose to throw out Diagnostics that point at a virtual text document or convert a PublishDiagnosticsParams into a Location[] or LocationLink[] and perform a request to the translate/locations or translate/locationLinks endpoint to get accurate diagnostics. Keep in mind that the locations/locationLinks endpoints can remove locations so if the list that's passed in does not match the list size that was originally authored the client will have to individually translate each location / location link.

Language Features

Querying language features can be done on any text document (not just virtual text documents). It is the job of the client to handle a server -> client language feature request by:

1. Forwarding the request to all applicable language servers
   • This requires the client to translate the request into a compatible LSP version for each language server. If it cannot it will not query that specific language server.
2. Aggregate the results and return them to the requesting language server

Below is the complete list of all supported queryable (they have a queryable client capability) language features with their corresponding parameters and return types.

Feature	Server -> Client Method	Parameters	Response (nullable)	VSCode Command
Completion	textDocument/completion	(CompletionParams)	CompletionList	vscode.executeCompletionItemProvider
Completion Resolve	completionItem/resolve	(DocumentUri, CompletionItem)	CompletionItem	(New) vscode.executeCompletionResolve
Hover	textDocument/hover	(HoverParams)	Hover	vscode.executeHoverProvider
Signature Help	textDocument/signatureHelp	(SignatureHelpParams)	SignatureHelp	vscode.executeSignatureHelpProvider
Goto Declaration	textDocument/declaration	(DeclarationParams)	Location[] or LocationLink[]	vscode.executeDeclarationProvider
Goto Definition	textDocument/definition	(DefinitionParams)	Location[] or LocationLink[]	vscode.executeDefinitionProvider
Goto Type Definition	textDocument/typeDefinition	(TypeDefinitionParams)	Location[] or LocationLink[]	vscode.executeTypeDefinitionProvider
Goto Implementation	textDocument/implementation	(ImplementationParams)	Location[] or LocationLink[]	vscode.executeImplementationProvider
Find References	textDocument/references	(ReferenceParams)	LocationLink[]	vscode.executeReferenceProvider
Document Highlight	textDocument/documentHighlight	(DocumentHighlightParams)	DocumentHighlight[]	vscode.executeDocumentHighlights
Document Symbols	textDocument/documentSymbol	(DocumentSymbolParams)	DocumentSymbol[]	vscode.executeDocumentSymbolProvider !!
Code Action	textDocument/codeAction	(CodeActionParams)	Command[] or CodeAction[]	vscode.executeCodeActionProvider
Code Action Resolve	codeAction/resolve	(DocumentUri, CodeAction)	CodeAction	(New) vscode.executeCodeActionResolve
Code Lens	codeAction/codeLens	(CodeLensParams)	CodeLens[]	vscode.executeCodeLensProvider
Code Lens Resolve	codeLens/resolve	(DocumentUri, CodeLens)	CodeLens	(New) vscode.executeCodeLensResolve
Document Link	textDocument/documentLink	(DocumentLinkParams)	DocumentLink[]	vscode.executeLinkProvider
Document Link Resolve	documentLink/resolve	(DocumentUri, DocumentLink)	DocumentLink	(New) vscode.executeLinkResolve
Document Color	textDocument/documentColor	(DocumentColorParams)	ColorInformation[]	vscode.executeDocumentColorProvider
Color Presentation	textDocument/colorPresentation	(ColorPresentationParams)	ColorPresentation[]	vscode.executeColorPresentationProvider
Document Formatting	textDocument/formatting	(DocumentFormattingParams)	TextEdit[]	vscode.executeFormatDocumentProvider
Document Range Formatting	textDocument/rangeFormatting	(DocumentRangeFormattingParams)	TextEdit[]	vscode.executeFormatRangeProvider
Document on Type Formatting	textDocument/onTypeFormatting	(DocumentOnTypeFormattingParams)	TextEdit[]	vscode.executeFormatOnTypeProvider
Rename	textDocument/rename	(RenameParams)	WorkspaceEdit	vscode.executeDocumentRenameProvider
Prepare Rename	textDocument/prepareRename	(PrepareRenameParams)	{ range: Range, placeholder: string }	(New) vscode.executePrepareRenameProvider
Folding Range	textDocument/foldingRange	(FoldingRangeParams)	FoldingRange[]	(New) vscode.executeFoldingRangeProvider
Selection Range	textDocument/foldingRange	(SelectionRangeParams)	SelectionRange[]	vscode.executeSelectionRangeProvider
Prepare Call Hierarchy	textDocument/prepareCallHierarchy	(CallHierarchyPrepareParams)	CallHierarchyItem[]	vscode.prepareCallHierarchy
Hierarchy Incoming Calls	callHierarchy/incomingCalls	(CallHierarchyIncomingCallsParams)	CallHierarchyIncomingCall[]	vscode.provideIncomingCalls
Hierarchy Outgoing Calls	callHierarchy/outgoingCalls	(CallHierarchyOutgoingCallsParams)	CallHierarchyOutgoingCall[]	vscode.provideOutgoingCalls
Semantic Tokens	textDocument/semanticTokens/full	(SemanticTokensParams)	SemanticTokens	vscode.provideDocumentSemanticTokens
Semantic Tokens Range	textDocument/semanticTokens/range	(SemanticTokensRangeParams)	SemanticTokens	vscode.provideDocumentRangeSemanticTokens
Semantic Tokens Legend	(New) textDocument/semanticTokens/legend	(SemanticTokensRangeParams)	SemanticTokens	vscode.provideDocumentSemanticTokensLegend
Semantic Tokens Legend Range	(New) textDocument/semanticTokens/legend/range	(DocumentUri, Range)	SemanticTokens	vscode.provideDocumentRangeSemanticTokensLegend !!
Linked Editing Range	textDocument/linkedEditingRange	(LinkedEditingRangeParams)	LinkedEditingRanges	(New) vscode.provideLinkedEditingRanges
Monikers	textDocument/moniker	(MonikerParams)	Moniker[]	(New) vscode.provideMoniker

External Virtual Text Document Interactions

External virtual text document interactions can occur from several requests. The data types that are deemed "externally interactable" are: WorkspaceEdit, Location and LocationLink. The current methods that utilize these types are:

Feature	Method	Type	Direction
Goto Declaration	textDocument/declaration	Location[] or LocationLink[]	client -> server
Goto Definition	textDocument/definition	Location[] or LocationLink[]	client -> server
Goto Type Definition	textDocument/typeDefinition	Location[] or LocationLink[]	client -> server
Goto Implementation	textDocument/implementation	Location[] or LocationLink[]	client -> server
Find References	textDocument/references	LocationLink[]	client -> server
Rename	textDocument/rename	WorkspaceEdit	client -> server
Apply Workspace Edit	workspace/applyEdit	WorkspaceEdit	server -> client

When a client gets a response or a request (workspace/applyEdit) from one of these supported methods that implicate a virtual text document it's the clients responsibility to lookup that virtual text document's owner and perform translation requests prior to applying the results.

Server Capability:

• property name (optional): workspace.virtualTextDocument
• property type: VirtualTextDocumentServerCapabilities defined as follows:

/**
 * Server capabilities specific to virtual text documents
 */
export interface VirtualTextDocumentServerCapabilities {
    /**
     * Whether the client supports translating externally sourced WorkspaceEdits on owned virtual text documents
     */
    workspaceEditOptions?: VirtualTextDocumentWorkspaceEditOptions;

    /**
     * Whether the client supports translating externally sourced Locations on owned virtual text documents
     */
    locationOptions?: VirtualTextDocumentLocationOptions;

    /**
     * Whether the client supports translating externally sourced LocationLinks on owned virtual text documents
     */
    locationLinkOptions?: VirtualTextDocumentLocationLinkOptions;
}

/**
 * Server capabilities specific to externally sourced WorkspaceEdit handling for virtual text documents
 */
export interface VirtualTextDocumentWorkspaceEditOptions {
}

/**
 * Server capabilities specific to externally sourced Location translation handling for virtual text documents
 */
export interface VirtualTextDocumentLocationOptions {
}

/**
 * Server capabilities specific to externally sourced LocationLink translation handling for virtual text documents
 */
export interface VirtualTextDocumentLocationLinkOptions {
}

WorkspaceEdit Translation Request

Server Capability:

• property name (optional): workspace.virtualTextDocument.workspaceEditOptions
• property type: VirtualTextDocumentWorkspaceEditOptions

Request:

• method: translate/workspaceEdit
• params: TranslateWorkspaceEditParams defined as follows:

interface TranslateWorkspaceEditParams {
    /**
     * The workspace edit to translate.
     */
    workspaceEdit: WorkspaceEdit;

    /**
     * The method that returned the provided workspace edit i.e. "textDocument/rename"
     */
    methodSource?: string;
}

Response:

• result: WorkspaceEdit

Locations Translation Request

Server Capability:

• property name (optional): workspace.virtualTextDocument.locationOptions
• property type: VirtualTextDocumentLocationOptions

Request:

• method: translate/locations
• params: TranslateLocationsParams defined as follows:

interface TranslateLocationsParams {
    /**
     * The locations to translate.
     */
    locations: Location[];

    /**
     * The method that returned the provided locations i.e. "textDocument/definition"
     */
    methodSource?: string;
}

Response:

• result: Location[]

Location Links Translation Request

Server Capability:

• property name (optional): workspace.virtualTextDocument.locationLinkOptions
• property type: VirtualTextDocumentLocationLinkOptions

Request:

• method: translate/locationLinks
• params: TranslateLocationLinksParams defined as follows:

interface TranslateLocationLinksParams {
    /**
     * The location links to translate.
     */
    locationLinks: LocationLink[];

    /**
     * The method that returned the provided location links i.e. "textDocument/reference"
     */
    methodSource?: string;
}

Response:

• result: LocationLink[]

Open questions

• Is workspace/applyEdit the right method to manage document state?
  • Initially I built out a mechanism for managing document state via custom virtualTextDocument/open/change/close requests (sub-spec link) where the open was a request and change and close were notifications. After deliberating pre-existing models for manipulating document content in the workspace (workspace/applyEdit) I fell back to the proven route; however, if the dynamic of open virtual text documents allow for change/close to be notifications that could be a highly beneficial approach.
• Should the top-level language server be controlling document versions after edit? Even in my spec variant I choose to not have it author the versions because technically the server should be controlling all of the document update requests which get put into queues ensuring that past or future sub-language versions don't really matter; servers always operate on the "latest".
• Is it reasonable for a client to make requests to translate locations and edits based off of the result of a previous request like textDocument/rename?
  • I considered having the translation be server initiated however I uncovered two problems with that approach:
    1. Servers would have to be virtual text document aware for any sort of translations to occur. This felt excessively restrictive given the sheer number of pre-existing language servers.
    2. If a server initiated a translate request could it actually provide reasonable information for others to react to? Aka, the act of renaming a symbol in one language is difficult for a language like Razor to understand. For instance in the above example regarding additive external interactions I found that having a symbol helper class was the only way (without private APIs) for Razor to properly understand when an interesting symbol interaction occurred making it less meaningful for a server to even initiate the request. Try to do the additive external interactions example with a server initiated flow but no symbol helper C# file. I at least quickly ran into issues where the server would be doing a translate/renameOperation and trying to pass opaque parameters that don't necessarily mean that the server is renaming a pertinant symbol that the host language cares about.
• Why does VSCode not provide a platform API to delegate resolve based requests like completionItem/resolve? I imagine the intent was because VSCode doesn't know which server cares about the completionItem, is this solveable? I found this thread where the completion item provider APIs can take in a number of items to auto-resolve; however, this is less than ideal. In VS all language servers get asked to resolve a completion item and most no-op if they can't do anything or don't recognize it.
• What in the world is the vscode.provideDocumentRangeSemanticTokensLegend VSCode command?
• lol why'd I even write this section? Everything is an open question 🎉 🤣 🎉

One area of complexity that I didn't see (or didn't recognize 😅) is how to handle server initiated messages to the client that originate from an embedded language server. We've hit two of these so far in our scenarios, push-based textDocument/publishDiagnostics and workspace/applyEdit. Quick hypothetical examples:

• In index.php.html.css (i.e. CSS virtual document within an HTML virtual document within a PHP document), the CSS language server identifies an error and publishes a diagnostic for it. This needs to make multiple hops to get remapped twice to get back to the original coordinate space in the non-virtual PHP document.
• In index.php.html, an HTML server recognizes a <script> block and provides a code action to extract that to its own file (e.g. "Extract script to new file <...>.js"). From the HTML server's point of view, it should issue a WorkspaceEdit containing two parts: CreateFile to create index.php.html.js, and TextDocumentEdit to edit the content (populate the new file).
  • The file name is probably incorrect and may collide with a virtual document of the same name (e.g. above you hinted that this could represente the <script> block plus any additional JS content present in the HTML virtual document). This may confuse the client. (Obviously it depends on the implementation of the virtual document naming...)
  • If the file is somehow correctly created with the (presumably correct) index.php.js name, then TextDocumentEdit also needs to be redirected to that new document name.
• workspace/applyEdit might also happen on any other code action which cannot immediately calculate an edit, e.g. a code action returns a Command to be invoked, which can asynchronously process the edit and respond with a workspace/applyEdit against the virtual document. That edit should, like the publishDiagnostics example, be remapped multiple times so that it can be applied as an edit on the non-virtual index.php document; more specifically, each Change or TextDocumentChange in the edit needs to be processed and remapped as needed on an individual basis (they may also pertain to documents - virtual or not - other than index.php(.*)).

Our mitigation for these so far has been to place a middle-man in the client which is aware of the delegation nature of these servers, and can intercede to apply the redirection when necessary. It's yet another complex gear in the machine that makes this challenging to work today.

Another thought: the behavior here ought not to make assumptions about how the virtual documents are structured. Your examples above assume that multiple embedded regions are mapping into the same file, however I imagine that would not be the case in some cases. VB.NET XML Literals spring to mind, as XML documents have XML declarations (<?xml version="1.0" ?>) or possibly other XML processing instructions specific to that fragment.

One area of complexity that I didn't see (or didn't recognize 😅) is how to handle server initiated messages to the client that originate from an embedded language server. We've hit two of these so far in our scenarios, push-based textDocument/publishDiagnostics and workspace/applyEdit

Oh definitely, it's a pretty big doc after all 😄. I tried to cover the workspace/applyEdit type scenarios in External Interactions part of the solution section and for textDocument/pushDiagnostics I felt that virtual text documents should primarily be using pull based diagnostics (coming in the next official LSP spec) but did add a nod to how they could be handled in the spec for diagnostics handling. Do you feel like it's worth adding some clarity to those?

• The file name is probably incorrect and may collide with a virtual document of the same name (e.g. above you hinted that this could represente the <script> block plus any additional JS content present in the HTML virtual document). This may confuse the client.
• If the file is somehow correctly created with the (presumably correct) index.php.js name, then TextDocumentEdit also needs to be redirected to that new document name.

Very interesting example. Only way I can imagine a fully supported way of handling that is for a host language server to try and apply fuzzy path matching to workspace edits it receives back and re-map filepaths that look like virtual text documents to the top level? Not ideal but it'd technically work; otherwise you'd probably need to throw out the code action.

Another thought: the behavior here ought not to make assumptions about how the virtual documents are structured. Your examples above assume that multiple embedded regions are mapping into the same file, however I imagine that would not be the case in some cases. VB.NET XML Literals spring to mind, as XML documents have XML declarations () or possibly other XML processing instructions specific to that fragment.

Hmm, I'm not sure I entirely follow. Are you referring to situations where in HTML you have a more then one language playing? Aka JS/CSS in the same line with attributes?

I assume you had something different in mind from the __SymbolHelpers in the additive interaction example right?

---

Just realized I can't add reactions to gist comments..... what is this GitHub!!! Love the feedback @jimmylewis ❤️ !

It might be nice to actually show what JS in HTML in Razor looks like, and the message complexity adding that third level would add

It seems a bit burdensome to add a queryable field to all of the supported messages. Is this intentional to require host support to be explicit per message type?

Why not allow the virtual document open message to pass over the initial text?

What's the purpose of the allied:true VS->LS response? Is it to ensure that the didOpen/didChange notification was sent out before the LS could send another applyEdit request?

In the interactions exapmples, why send over Person.cs entries to be translated by razor? Would it only be able to handle the *.razor.cs ones that it created?

One area of complexity that I didn't see (or didn't recognize 😅) is how to handle server initiated messages to the client that originate from an embedded language server. We've hit two of these so far in our scenarios, push-based textDocument/publishDiagnostics and workspace/applyEdit

Oh definitely, it's a pretty big doc after all 😄. I tried to cover the workspace/applyEdit type scenarios in External Interactions part of the solution section and for textDocument/pushDiagnostics I felt that virtual text documents should primarily be using pull based diagnostics (coming in the next official LSP spec) but did add a nod to how they could be handled in the spec for diagnostics handling. Do you feel like it's worth adding some clarity to those?

Ah I think I see it now, I might have glossed over that last night. Right now we're doing that in the delegating server pattern or with the middle-man interception, and you're proposing that the client should take on the roll of knowing "when" an LSP message needs to be remapped, but the "how" it gets remapped is still owned by the LSP for the real document. This is an interesting idea, and I like it, because right now with the middle-man approach we have to apply heuristics to try and determine this in a relatively stateless way (e.g. look at incoming message, and try to determine if both (a) the document looks like a virtual document and (b) if it were a virtual document, we can identify a mapping back to a parent document).

I think to augment this, maybe using workspace/applyEdit is not the right way to handle virtual document lifecycle events because it is ambiguous as to whether it's being used for real edits or for virtual document upkeep. I.e. are you creating a new "real document" named index.razor.cs or are you creating a virtual document? What if there were instead dedicated events for those, so the client would be able to easily track which documents are virtual, and then it automatically knows which messages require translation? E.g.

Real Document LSP (e.g. Razor LSP)                                  LS Client (e.g. VS Code, etc)                                  Embedded LSP (HTML, C#, etc)
             <------------------------ textDocument/didOpen ------------------------    
             ------------------------ textDocument/openVirtualDocument ------------>
                                                                                     ----------------------------- textDocument/didOpen ---------------->


             <---------------------- textDocument/didChange ------------------------
             ---------------------- textDocument/updateVirtualDocument ------------>
                                                                                     ----------------------------- textDocument/didChange ---------------->


                                                                                     <---------------------------------- workspace/applyEdit -----------------
                                                 (because this originated from a virtual document, I know it needs to be translated, 
                                                  and I know which language server is resposible for creating it)
             <---------------------- translate/workspaceEdit ------------------------
             ---------------------- translate/workspaceEdit (response) ------------>
             <---------------------- textDocument/didChange ------------------------
             ---------------------- textDocument/updateVirtualDocument ------------>
                                                                                     ----------------------------- textDocument/didChange ---------------->


             <---------------------- textDocument/didClose------------------------
             ---------------------- textDocument/closeVirtualDocument ------------>
                                                                                     ----------------------------- textDocument/didClose ----------------->

Although even with this I wonder if there will be other complexities in managing workspace edits. What if my embedded LS creates a workspace edit with changes targeting multiple virtual documents that map back to different types of real files (i.e. the Real Document LSP doesn't know all of the translation requirements)? Does the client need to peek into the WorkspaceEdit and split it out into multiple translation requests to different servers?

• The file name is probably incorrect and may collide with a virtual document of the same name (e.g. above you hinted that this could represente the <script> block plus any additional JS content present in the HTML virtual document). This may confuse the client.
• If the file is somehow correctly created with the (presumably correct) index.php.js name, then TextDocumentEdit also needs to be redirected to that new document name.

Very interesting example. Only way I can imagine a fully supported way of handling that is for a host language server to try and apply fuzzy path matching to workspace edits it receives back and re-map filepaths that look like virtual text documents to the top level? Not ideal but it'd technically work; otherwise you'd probably need to throw out the code action.

I think using a dedicated virtualDocument set of messages might address this ambiguity? Throwing out the code action seems like a really poor experience (technically, you're throwing out the workspace edit created by invoking the code action), because you wouldn't know to throw it out until after the user has invoked it; from their perspective, "this code action never works, argh! 😡"

Another thought: the behavior here ought not to make assumptions about how the virtual documents are structured. Your examples above assume that multiple embedded regions are mapping into the same file, however I imagine that would not be the case in some cases. VB.NET XML Literals spring to mind, as XML documents have XML declarations () or possibly other XML processing instructions specific to that fragment.

Hmm, I'm not sure I entirely follow. Are you referring to situations where in HTML you have a more then one language playing? Aka JS/CSS in the same line with attributes?

I assume you had something different in mind from the __SymbolHelpers in the additive interaction example right?

I'm thinking of something like this:

Dim jimmylewis As XDocument = 
    <?xml version="1.0"?>
    <contact>
      <name>Jimmy Lewis</name>
      <phone type="home">206-555-0144</phone>
      <phone type="work">425-555-0145</phone>
    </contact>
Dim taylormullen As XDocument = 
    <?xml version="1.0"?>
    <contact>
      <name>Taylor Mullen</name>
      <phone type="home">206-555-0144</phone>
      <phone type="work">425-555-0145</phone>
    </contact>

You can't concatenate/join these into a single XML document because of the <?xml version="1.0"?> lines (each of which could have other additional attributes). Something needs to determine that I need separate AddressBook.vb.jimmylewis.xml and AddressBook.vb.taylormullen.xml documents. Maybe that's just an implementation detail left up to the top level LSP and it's really a non-issue....

@ToddGrun responses:

It might be nice to actually show what JS in HTML in Razor looks like, and the message complexity adding that third level would add

I'm glad you said that. I thought about adding the more complex example but in the end convinced myself not to probably out of laziness 😄

It seems a bit burdensome to add a queryable field to all of the supported messages. Is this intentional to require host support to be explicit per message type?

So it'd only be in the client capabilities of each feature but admittedly yes, it'd be a lot. I didn't want to have it so piecemeal initially but after looking at the supported VSCode command APIs I saw gaps that VSCode had which led me to the conclusion that it may be unrealistic to expect clients to implement every server->client LSP request.

Why not allow the virtual document open message to pass over the initial text?

Only because it utilizes workspace/applyEdit to manage virtual document state. I initially built out a sub-spec which utilized custom virtual document open/change/close events but then fell back to workspace/applyEdit after realizing the size of the spec expansion + a precedent for modifying workspace state. That being said it's definitely an open question I also had. I added some more context in the first Open question section bullet.

What's the purpose of the allied:true VS->LS response? Is it to ensure that the didOpen/didChange notification was sent out before the LS could send another applyEdit request?

It's actually an implementation detail of the workspace/applyEdit request. I don't love it and in the sub-spec I built out I actually didn't have it for the change/close events. Because workspace edits can technically fail to apply they need to indicate if they were able to apply them. Check out the sub-spec though, I take a different approach and I plan to talk to some VSCode folks on if that can be the route forward 😄

In the interactions exapmples, why send over Person.cs entries to be translated by razor? Would it only be able to handle the *.razor.cs ones that it created?

So it's definitely a point for debate! My intent with it is that if a sub-language truly can't map a workspace edit and wants to be extra protective of the scenarios they can throw out the entire workspace edit. Aka, sometimes it's better do nothing then do the wrong thing? I can be convinced otherwise!

@jimmylewis responses:

I think to augment this, maybe using workspace/applyEdit is not the right way to handle virtual document lifecycle events because it is ambiguous as to whether it's being used for real edits or for virtual document upkeep. I.e. are you creating a new "real document" named index.razor.cs or are you creating a virtual document? What if there were instead dedicated events for those, so the client would be able to easily track which documents are virtual, and then it automatically knows which messages require translation? E.g.

We all think a lot alike! I initially built out a sub-spec which utilized custom virtual document open/change/close events but then fell back to workspace/applyEdit after realizing the size of the spec expansion + a precedent for modifying workspace state. That being said it's definitely an open question I also had. I added some more context in the first Open question section bullet.

Oh and holy hell you're a BEAST for using markdown for your diagram! Haha super impressed 👏

What if my embedded LS creates a workspace edit with changes targeting multiple virtual documents that map back to different types of real files (i.e. the Real Document LSP doesn't know all of the translation requirements)?

Ya good point, I imagine the client would have to ask all virtual document owned servers in aggregate. Aka, ask server 1 => server 2 => respond. Technically the servers shouldn't know another file is a virtual document so they'd either leave it alone or discard the entire edit.

I think using a dedicated virtualDocument set of messages might address this ambiguity?

It's an interesting decision point. Honestly, the only reason why I didn't was for LSP spec maintainability. Aka, when completion or hover etc. gets updated being able to re-use any spec updates as much as possible would be ideal. One thing I don't quite understand is how would a dedicated set of virtual document messages fix this though?

You can't concatenate/join these into a single XML document because of the lines (each of which could have other additional attributes).

Ahh, in this model the host language server (HTML) would only get notified about completions etc. for the HTML document. It's up to it to determine which virtual document that location maps to. In this case it could technically build out two separate XML docs for each of those pieces and then choose to perform a textDocument/completion request on the appropriate one.

Maybe that's just an implementation detail left up to the top level LSP and it's really a non-issue....

If I read your comment correctly I think so!