Skip to content

Instantly share code, notes, and snippets.

@StephenWeatherford

StephenWeatherford/f12-proposal.md Secret

Last active October 10, 2023 23:25
Show Gist options
  • Save StephenWeatherford/723d1a8ac530cfd4ba734d6aef0c545a to your computer and use it in GitHub Desktop.
Save StephenWeatherford/723d1a8ac530cfd4ba734d6aef0c545a to your computer and use it in GitHub Desktop.
Download ZIP
F12 Bicep Sources Proposal
Raw
f12-proposal.md

Problem statement

Current: Pressing F12 on the link opens the compiled main.json file, not the original bicep files (multiple bicep files compile to a single JSON file).

Desired: Pressing F12 should show the original Bicep source files

Tracked in Azure/bicep#5407

We've recently moved to using an internal container registry for our Bicep modules. It's really great. However, when you command+click / hit "Go to Definition" you are presented with ARM templates, rather than Bicep code.

ARM templates are less readable than Bicep. The code you see when you hit "Go to Definition", whilst being the built output of the module being consumed, it is not the beautifully readable Bicep.

Example:

module cogito 'br/public:ai/cognitiveservices:1.0.3' = {
  name: 'sum'
}

Pressing F12 on the link opens the compiled main.json file, not the original bicep files (multiple bicep files compile to a single JSON file).

Planned breaking changes

  • Breaking changes coming to support publishing sources and other future features and to fix a possible compatibility issue in our ACR manifest.

Planned behavior: Breaking change will affect version v-minus-2, i.e.:

  • Current version (v0.20) or earlier: Will get an error trying to use a module published with v0.22
  • v0.21 (Early September): Can use modules published with v0.22 (but no new functionality)
  • v0.22 (Early October): Breaking changes made affecting v0.20.

Opt in

Sources will only be published if a developer opts in to it when publishing (e.g. --publishSource or similar flag).

Finding and accessing source code

User Scenarios

1. Browsing/viewing sources inside VS Code

  • P1: User can see individual source files and follow links to navigate between them
  • P1: User can copy/paste into their own projects as desired (just like any other sources shown in VS Code - following the license is responsibility of end user)
  • P1: This should be the experience for public Bicep registry, others can opt in

2. Browsing sources in the original repo inside the browser

  • P1: Allow developer to specify a link to the source location (e.g. github repo). We can then take the user to that location when directed (main branch, current code)
  • P1: This should be the experience for public Bicep registry, others can opt in
  • Possible future: Take user directly to a particular source file at the same version from which it was compiled

3. Contributing back to source code (file bugs, fix/contribute code)

  • P1: Same as #2 P1 - Take user directly to the repo
  • P1: This should be the experience for public Bicep registry, others can opt in
  • Possible future: Integrate some of this experience more directly into VS Code (e.g., file issues, clone, create/open branches)
  • QUESTION: How show link to user?

How store/locate source code?

Two options:

1. Store source directly with the module in ACR

Advantages

  • Simple
  • Reliable.
  • Sources are guaranteed to be available, even without network connection (once restored)
  • Simple access modulel - anyone able to consume a module will be able to see its sources (if published)
  • No need to worry about authentication to different hosts
  • Simple and easy to understand
  • Source is guaranteed to match the module, regardless of versions or updates

2. Store links to source with the module

Advantages

  • Can control access to the module and the module's source separately
  • Faster restoration (and maybe publish), sources don't need to be downloaded with restore

Disadvantages

  • Exposes the ACR or other location of where the source is stored, which can be sensitive information.
  • To avoid this, we would need to store the source in a separate location (presumably public), which greatly complicates the scenario.
  • If we wanted to display the sources inside VS Code, we would need to implementation the authentication code ourselves, which is prohibitive given the number of possible hosts
  • No guarantee the sources haven't been changed after publishing

Decision

We will store the sources directly with the module

Link to source code repo

We should provide an optional way of providing a link to the source code's repo URI (e.g. a github URI), something like --sourceRepoLink. This enables the scenarios where the user may want to contribute back. This can be specified with or instead of --sourceRepoLink

TODO: How do we show the link in the UI?

TODO: How is this affected by --allowRepublishSource?

How do we get the link to the repo?

Probable answer: require user to specify it on command line.

  • ISSUE: Is it simply a link the developer gives us with no linking capabilities to individual files? Or do we know how to find the path to individual files? NOTE: SourceLink has some code we should be able to use to figure out the current github info when compiling.

TODO: Needs more investigation to understand.

Sensitive information in sources

The compiled module's JSON is of necessity publicly available to users referencing a module, since it is packaged inside the final full JSON. This includes for instance things like variable names. There are additional pieces of sensitive information available with published sources that are not present in the compiled JSON:

  • Comments
  • Full paths of references to ACR repositories containing nested modules and template specs (unless aliases are used). These paths do not appear in the compiled JSON. e.g.:
module myNestedModule 'br:myrepopath.azurecr.io/mypath/mymodule' = {...}

It's reasonable to assume the user will be responsible for ensuring the sources do not contain sensitive information before choosing to opt in with --publishSource.

TODO: Should we give a warning if --publishSource is used and a full repo path is used rather than an alias in a module reference?

TODO: Should could provide an option to strip comments automatically when publishing (easy to do)?

bicepconfig.json file

bicepconfig.json will not be published along with the sources, as it contains possibly-sensitive URIs and other information. (Note that it is not in the compiler's compilation group anyway, which is the set of sources we package up.)

Link to repo

We will allow the developer to provide a link to the source code repo (e.g. github project) to enable the end developer to easily contribute or file bugs

Possible future direction

We can consider (using SourceLink?) bringing the user directly to individual files at the git version when it was published.

Nested sources scenario

For the rest of this document, consider this scenario:

Developer1 creates module1 and publishes it to acr1 with sources

Developer2 creates module2 that uses module1 and publishes it to acr2 with sources

Developer3 uses module2 in a local bicep file

What is packaged

When a developer publishes a module with --publishSource, a sources.zip (gzip?) containing all files in the compilation group will be published along with the compiled main.json file. This includes the JSON sources of any file that is compiled into main.json, including any local .bicep or .json files, plus the compiled JSON of referenced Bicep modules and template specs. These JSON files are what the Bicep compiler actually combines in to a single compiled main.json file.

Fallback when sources not published

The contents of these nested JSON files are currently available to a developer using the module (via F12), but they are difficult to disentangle and read, and are slightly modified from their original JSON (to package as nested template). Having the original JSON available means that even if the original .bicep sources of nested modules are not published, the compiled JSON for those nested modules will be available.

I.e., if Developer3 presses F12 on module2 to view sources in Bicep, and then presses F12 on the reference to module1 inside of module2.bicep, if bicep1 sources are available, he will be shown module1.bicep. If those sources are not available, he will be shown module1.json.

(This assumes module2 was published with sources. If not, the user will only be shown module2.json and no other source files will be visible to him, not even module1.JSON. It's possible to instead always package source JSON files even if --publishSource is not specified, but probably not desired and requires a way to allow the user to navigate to module1.json from inside module2.json in the UI.)

Example

When developer1 publishes module1 with --publishSource, the published artifact will contain:

module1 artifact in ACR1:

  • compiled JSON (we'll call it module1.json)
  • sources.zip:
    • metadata.json
      • contains information about packaged sources and their original names
    • all files in the compilation group of module1:
      • module1.json
      • module1a.bicep
      • module1b.bicep
      • any other local .bicep files compiled into module1.json
    • allowRepublish option module1 (true or false)

Allow republish?

If developer3 is to be able to see .bicep sources for both module1 and module2, then when module2 is published with --publishSource, it would need to contain the sources for both module2 and module1, e.g.:

module2 artifact in ACR2:

  • compiled JSON (we'll call it module2.json)
    • (contains contents of module1.json compiled into it)
  • sources.zip:
    • metadata.json
      • contains information about packaged sources and their original names
    • all files in the compilation group of module2:
      • module2a.bicep
      • module2b.bicep
      • any other local .bicep files compiled into module2.json
      • module1.json
      • sources.zip for module1
        • allows us to navigate to module1 sources from inside an editor showing module2.bicep
      • .json (and possibly sources.zip) for any other files belonging to referenced modules or template specs
    • allowRepublish option for module2 (true or false)

If we include the sources.zip file for module1 inside of the module2 artifact, then we are exposing the full sources of module1 to developer3. If we were publishing links to source files rather than actual source files, then access to module1 sources would be gated by authentication/authorization to the module1 source file location itself (as for instance with the C# debugger using Source Link). Since we are not doing that, we need a way to allow developer1 to gate access of his/her sources. To do that, we will have an additional command-line option, something like:

--allowRepublish

When specified with --publishSource for module1, indicates permission to include the published sources along with any other published modules that reference this one

This option's value is stored in sources.zip in module1. When publishing module2, if --allowRepublish was specified for module1, then module1's sources.zip will be included in module2's artifact. Otherwise, it will not, and developer3 will be able to view only the following:

  • module2a.bicep
  • module2b.bicep
  • etc.
  • optionally module2.json
  • module1.json (but not the .bicep or nested source files for module1)

TODO: We need a way to allow developer3 to optionally show the JSON for module1 instead of the .bicep source files.

How are sources packaged in the ACR?

Sources will be packaged as a zip (TODO: gzip?) file inside a layer of the module's artifact.

TODO: more details

Mapping file (__metadata.json)

TODO: details TODO: details about handling of ".." and multiple roots TODO: Can we re-use Source Link's mapping file and code for this?

How do we cache the sources on the local machine?

Sources will be downloaded and stored in the module cache on the local machine when a module is restored, and in the same location.

More specifically, all layers in a manifest file will be downloaded and stored as binary data in the module cache along with the manifest, main.json and other module files. This allows bicep version x to restore a module successfully in a way to bicep version x+1 will be able to read and use it, even if version x+1 adds a new layer that bicep version x does not know about.

The layers must also not be written into the module cache in any other form. E.g., if a module layer contains a sources.zip file, it must not be written as "sources.zip" into the module cache, but must remain as layer1.bin (or whatever we end up using) and read and interpreted directly from that file for each version of bicep that is being used. This is because module cache contents are not allowed to be in-place updated, due to the way the contents are protected from simultaneous reads/writes via a semaphore file.

(Note: we could theoretically allow it as long as the semaphore is taken first, but it just complicates the scenario of having multiple versions share the same cache. On the other hand, we have also decided to version the cache whenever its format changes. So perhaps this is being overly careful.)

Terminology

"sources" or "source"?

I will use "source" everywhere

e.g.

  • -- publishSource
  • source.gzp

CLI options (TODO: draft)

bicep.exe

Publish source

New argument:

bicep publish ... --publishSource

or

bicep publish ... --withSource

Default: Do not publish sources

If we add an option to navigate through multiple levels of nested, published modules, we will consider adding an additional option:

bicep publish ... --publishSource [allowRepublish]

Alternative: bicep publish ... --publish disallowRepublish But that would mean the default would be a change in behavior.

Also considered: using separate flags like --publishSource and --allowRepublishSource, but was rejected as being unpopular.

Code repository link

Suggestions:

--codeUri <uri>

or

--codeRepositoryUri <uri>

or

--contributionUri <uri>

or

use --documentationUri

or ...?

TODO: can it be in the config file?

Examples

bicep publish "my entrypoint.bicep" --target br:<repo>.azurecr.io/path:v1 --documentationUri https://github.com/Azure/bicep-registry-modules/blob/main/modules/ai/bing-resource/README.md --withSource allowRepublish --contributionUri https://github.com/Azure/bicep-registry-modules/blob/main/modules/ai/bing-resource/main.bicep

az and PowerShell

I expect these to be the same as for bicep

UI

TODO

Top-level navigation to external module source

This will be the first level of implementation.

TODO: There's an inconsistency in how F12 and hover are currently handled for modules:

module m2 'br/public:app/dapr-containerapps-environment:1.2.2' = {
  name: 'm2'
  params: {
    location: 'l'
    daprComponentType: 'pubsub.azure.servicebus'
    nameseed: 'ns'
  }
}

Hovering over m2 gives this: image

Pressing F12 on 'br/public:app/dapr-containerapps-environment:1.2.2' takes you to the JSON (soon to be the .bicep if available)

Should these be consistent?

TODO: How do we indicate to users that no .bicep source is available for an external module? TODO: How do we allow users to view the JSON instead of the .bicep for an external module?

Deep navigation

When sources are not available

When sources are published, user should be able to navigate to all top-level source files referenced. For nested modules without bicep sources available, and for all template spec referenences, the user will be taken to the compiled JSON for that file.

That means the user will not be able to navigate from the compiled JSON of a nested module nested1 to the JSON of a deeply nested module nested2 that's referenced from nested1 if nested is published without sources. They will however be able to view deeply nested modules such as nested2 in the form or embedded deployments in nested1's compiled JSON.

TODO: should we publish JSON sources for all nested modules, even if --publishSource is false (or --noRepublishSource is specified)? The JSON is available for viewing anyway.

OUT OF SCOPE: Displaying source code using a different version of Bicep

DECISION: Considered out of scope for now. We will use the installed version of Bicep, but may disable errors and warnings.

Because the source code of a published module could have been compiled using a different version of bicep (either newer or older) than the version installed with the language server, and also because the bicepconfig.json is not included in the published sources, the viewing of the source may be different than expected:

  1. Unexpected analyzer warnings may occur TODO: Turn these off when displaying external sources?
  2. Unexpected compiler warnings and errors may occur (e.g. syntax that's not recognized by this version of Bicep) TODO: Turn these off when displaying external sources?
  3. Symbols may be different than expected TODO: Embed symbol information generated by the compiler during module publish?

TODO: We should place bicep compiler version into published module's artifact for possible future use

How allow user to follow deeply-nested links?

TODO: My expectation is that we will show links in the module's source code to take them to deeply-linked sources, just like we do right now inside local modules. TODO: Explore/design UI more

TODO: Will user find this in the UI?

TODO: To make this work, I will need to modify the language server to handle external .bicep files differently. Needs investigation/experimentation.

TODO: Do we have a better way to display the tree structure of external sources?

TODO: possible UI:

  • outline view
  • file explorer
  • links inside editor
  • breadcrumbs (driven by same mechanism as outline view) image
  • links at top of the source file?

TODO: Show OCI structure

TODO: Where place "canRepublishSource"?

TODO: "disableSourcePublishing" in bicepconfig.json?

TODO: Possible future directions (probably out of scope for now)

  • show source code for loadjsoncontent() etc
  • Integration with Bicep visualizer
  • another option to open the folder of the module and add to my workspace.
    • need access to code repo
  • navigate to the source code repo at the specific version of the published module

Notes and investigation results

Investigtion summary: SourceLink

What SourceLink does

  • Provides a standard way of mapping source files from the build machine to source files at a location where the debugger can access them.
  • Provides nuget packages with implementation details for accessing sources at various locations (e.g. github, Bit Bucket)ץ

What SourceLink does not do

  • Used by individual debuggers (like C# in VS Code), not implemented as service into VS or VS Code themselves
  • Provides no UI
  • Provides no authentication capabilities, debugger must handle that

What we can use from SourceLink

  • Possibly re-use their sourcelink.json file format, esp if they have support for reading/writing
  • Possibly use their ability to find the location of sources in github etc
  • Needs more investigation

Related technologies

TODO: To Investigate

  • What exactly does the source mapping experimental feature in bicep do? Is it relevant?
  • Could we provide links from Azure errors to take the user to the original Bicep location of the resource in the module source?
  • Can we use these?
	// AnnotationVersion is the annotation key for the version of the packaged software.
	// The version MAY match a label or tag in the source code repository.
	// The version MAY be Semantic versioning-compatible.
	AnnotationVersion = "org.opencontainers.image.version"
	// AnnotationRevision is the annotation key for the source control revision identifier for the packaged software.
	AnnotationRevision = "org.opencontainers.image.revision"
	- **org.opencontainers.image.url** URL to find more information on the image (string)
- **org.opencontainers.image.documentation** URL to get documentation on the image (string)
- **org.opencontainers.image.source** URL to get source code for building the image (string)
- **org.opencontainers.image.version** version of the packaged software
  - The version MAY match a label or tag in the source code repository
  - version MAY be [Semantic versioning-compatible](https://semver.org/)
- **org.opencontainers.image.revision** Source control revision identifier for the packaged software.

TODO: does contribution URI get published? TODO: bicepconfig.json - ability to disable "--withSource" or "--AllowRepublishSource"?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment