Dummy `rst2myst` example for issue

!Dummy rst2myst example for issue.md

dummy-rst2myst-example-for-issue

This repository demonstrates an issue with the rst2myst conversion tool.

Issue

The way rst2myst translates rst substitutions to Markdown does not seem to work properly. So far 2 inconsistencies have been identified:

1. rst2myst relies on MyST ability to use Jinja2 for substitutions. Jinja2 requires a valid identifier (i.e., valid Python variable names), which is a much more restrictive condition, and therefore not compatible with rst (e.g. rst accepts strings with spaces)

2. rst2myst fails on translating links with substitutions (e.g. |ref|_), which are a very common use case when writing rst (e.g. for creating links with bold, italic or monospaced text elements)

This issue was first identified in pyscaffold/pyscaffoldext-markdown#25. An equivalent but minimal/simplified example can be found in original.rst.

The following command was used to convert the file to MyST-flavoured markdown, producing converted.md.

$ rst2myst stream original.rst > converted.md

As we can see, converted.md contains invalid Jinja2 syntax, e.g. {{ the repository service }}, and the original link with substitution was converted to a simple link, without a substitution.

Expected output

Two different behaviours could be expected from rst2myst:

1. The "rst substitution key" could be pre-processed to produce a valid Python identifier and in turn, that identifier could be used in Jinja2. This include replacing or removing invalid characters, e.g. |the repository service| could be translated to {{ the_repository_service }}. This approach in shown in expected_option1.md
2. The "rst substitution key" could be nested inside a dict value with an arbitrary name (e.g. __), e.g. |the repository service| could be translated to {{ __["the repository service"] }}. This approach in shown in expected_option2.md

Regarding links with substitutions, CommonMark's full reference links in combination with Jinja2 seem to be a good alternative, e.g. |ref|_ could be translated to [{{ ref }}][ref]. This approach is shown in both expected_option1.md and expected_option2.md.

Running the example

To run this example, you will need a Python virtual environment configured with the right dependencies. The following commands can be used to create one:

$ virtualenv -p py38 .venv
$ source .venv/bin/activate
$ pip install sphinx myst-parser 'rst-to-myst[sphinx]'

(A POSIX operating system an shell is also assumed)

The following commands can be used to compile the examples and serve HTML locally:

$ rst2myst stream original.rst > converted.md
$ make html
$ python3 -m http.server --directory _build/html

.gitignore

*~
*.py[cod]
*.orig
*.log
*.pot
__pycache__/*
.cache/*
.*.swp
tags
.tox
api/*
_rst/*
_build/*
.venv*/
.conda*/

conf.py

# Configuration file for the Sphinx documentation builder.
#
# This file only contains a selection of the most common options. For a full
# list see the documentation:
# http://www.sphinx-doc.org/en/master/config

# -- Path setup --------------------------------------------------------------

# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
#
# import os
# import sys
# sys.path.insert(0, os.path.abspath('.'))


# -- Project information -----------------------------------------------------

project = 'Dummy rst2myst example for issue'
copyright = '2021, Anderson Bravalheri'
author = 'Anderson Bravalheri'


# -- General configuration ---------------------------------------------------

# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = [
    "myst_parser"
]

# Configure MyST-Parser
myst_enable_extensions = [
    "amsmath",
    "colon_fence",
    "deflist",
    "dollarmath",
    "html_image",
    "replacements",
    "smartquotes",
    "substitution",
]

# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']

# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
# This pattern also affects html_static_path and html_extra_path.
exclude_patterns = ['.venv', '_build', 'README.md', '!README.md', 'Thumbs.db', '.DS_Store']


# -- Options for HTML output -------------------------------------------------

# The theme to use for HTML and HTML Help pages.  See the documentation for
# a list of builtin themes.
#
html_theme = 'alabaster'

# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ['_static']

converted.md

substitutions
contribute button the repository service virtual environments "Create pull request" GitHub `virtualenv`

Automatic Conversion (current state)

1. To avoid any problems with your installed Python packages, consider using virtual environments

2. If everything works fine, push your local branch to {{ the repository service }} with:

   git push -u origin my-feature
   

3. Go to the web page of your fork and click {{ contribute button }} to send your changes for review.

expected_option1.md

substitutions
contribute_button the_repository_service virtual_environments "Create pull request" GitHub `virtualenv`

Expected Conversion (option 1)

1. To avoid any problems with your installed Python packages, consider using {{ virtual_environments }}

2. If everything works fine, push your local branch to {{ the_repository_service }} with:

   git push -u origin my-feature
   

3. Go to the web page of your fork and click {{ contribute_button }} to send your changes for review.

expected_option2.md

substitutions
__ contribute button the repository service virtual environments "Create pull request" GitHub `virtualenv`

Expected Conversion (option 2)

1. To avoid any problems with your installed Python packages, consider using {{ __["virtual environments"] }}

2. If everything works fine, push your local branch to {{ __["the repository service"] }} with:

   git push -u origin my-feature
   

3. Go to the web page of your fork and click {{ __["contribute button"] }} to send your changes for review.

index.rst

Welcome to Dummy rst2myst example for issue's documentation!

The purpose of this file is to setup the project in a way Sphinx process all example .rst and .md files.

Original reST file <original> Automatic Conversion (current state) <converted> Expected Conversion (option 1) <expected_option1> Expected Conversion (option 2) <expected_option2>

Indices and tables

• genindex
• modindex
• search

make.bat

@ECHO OFF

pushd %~dp0

REM Command file for Sphinx documentation

if "%SPHINXBUILD%" == "" (
	set SPHINXBUILD=sphinx-build
)
set SOURCEDIR=.
set BUILDDIR=_build

if "%1" == "" goto help

%SPHINXBUILD% >NUL 2>NUL
if errorlevel 9009 (
	echo.
	echo.The 'sphinx-build' command was not found. Make sure you have Sphinx
	echo.installed, then set the SPHINXBUILD environment variable to point
	echo.to the full path of the 'sphinx-build' executable. Alternatively you
	echo.may add the Sphinx directory to PATH.
	echo.
	echo.If you don't have Sphinx installed, grab it from
	echo.http://sphinx-doc.org/
	exit /b 1
)

%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
goto end

:help
%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%

:end
popd

Makefile

# Minimal makefile for Sphinx documentation
#

# You can set these variables from the command line, and also
# from the environment for the first two.
SPHINXOPTS    ?=
SPHINXBUILD   ?= sphinx-build
SOURCEDIR     = .
BUILDDIR      = _build

# Put it first so that "make" without argument is like "make help".
help:
	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

.PHONY: help Makefile

# Catch-all target: route all unknown targets to Sphinx using the new
# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
%: Makefile
	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

original.rst

Original File

1. To avoid any problems with your installed Python packages, consider using virtualenv_

2. If everything works fine, push your local branch to GitHub with:

   git push -u origin my-feature

3. Go to the web page of your fork and click "Create pull request" to send your changes for review.