digitalmoksha/fix-links.lua

## readme.md

      
    Raw
  

              readme.md
            
          
    Table of Contents


How to use pandoc with gitlab markdown

Installing the Dependencies
Setting Up the Environment
Generate PDF/HTML


How to use pandoc with gitlab markdown

If you're like me, then you use gitlab-flavored markdown quite a bit. The combination of mermaid and plantuml graphs and katex math really makes for a productive experience. However, due to issues like gitlab-org/gitlab#215084 and gitlab-org/gitlab#243533, you probably also want to have some other way to render your notes, rather than relying on gitlab to do it for you.
This guide will show you how to do just that with a tool called pandoc, as long as you're OK with not having all the features of gitlab markdown. At the moment, this setup supports the following features:

Mermaid Graphs
PlantUML Graphs
Katex Math
Relative and Absolute References to Project Files
References to Directories with READMEs

This setup does run in CI/CD, which is how I primarily use it, but this guide will focus on how to set this up on your own machine. If you're interested in a CI/CD project template though, I have one on my gitlab. I will tend to update that faster than this document, and I track known issues with it there too.
Installing the Dependencies

To produce this setup, I installed:


Dependency
Rationale


mermaid-cli
Required by timofurrer's pandoc-mermaid-filter


plantuml
Required by timofurrer's pandoc-plantuml-filter


texlive-full^[1]
Required by the pandoc LaTeX template


My fork of @timofurrer's pandoc-plantuml-filter
Allows pandoc to render plantuml code blocks as graphs


My fork of @timofurrer's pandoc-mermaid-filter
Allow pandoc to make sense of mermaid code blocks as graphs


This GitHub pandoc html5 template
Used to make the html all pretty :D


This pandoc LaTeX Template
Used to make the pdf all pretty :D


@lierdakil's gitlab-math.lua script
Supports gitlab's special math syntax.


My fix-links.lua script
Fixes links and urls to resources within the project.


... and of course, pandoc itself!
I'm on Manjaro Linux, so I ended up using the following commands to install these:
# install plantuml & mermaid
yarn global add @mermaid-js/mermaid-cli
pamac install plantuml
# Install pandoc plantuml & mermaid filter
pip install git+https://github.com/MyriaCore/pandoc-plantuml-filter.git
pip install git+https://github.com/MyriaCore/pandoc-mermaid-filter.git
# Install lua filters
mkdir -p ~/.pandoc/filters
wget https://gist.githubusercontent.com/lierdakil/00d8143465a488e0b854a3b4bf355cf6/raw/gitlab-math.lua
mv gitlab-math.lua ~/.pandoc/filters/gitlab-math.lua
wget https://gist.githubusercontent.com/MyriaCore/75729707404cba1c0de89cc03b7a6adf/raw/fix-links.lua
mv fix-links.lua ~/.pandoc/filters/fix-links.lua
Just a quick note, you will need to install yarn to install mermaid-cli, unfortunately. You could try npm, but that didn't work as well for me.

1: On manjaro, I had to get it from the aur, but if you're on an ubuntu-based distro, you should be able to get it more easily. ↩
Setting Up the Environment

All you'll need to do to setup the environment is expose the mermaid binary (assuming you installed via yarn):
# Exposes mmdc (mermaid binary) for us
export PATH=$(yarn global bin):$PATH 
Generate PDF/HTML

I wrote a small script to generate a pdf or html file (at the user's request). If you don't care about the commands and just want to get up and running, I'd recommend putting it somewhere on your $PATH:
wget https://gist.githubusercontent.com/MyriaCore/75729707404cba1c0de89cc03b7a6adf/raw/570ec66358b95f634a28be69ba5de59854e41b24/gfm.sh
mv gfm.sh /usr/local/bin

# create readme.html
gfm.sh readme.md
gfm.sh -html readme.md

# create readme.pdf
gfm.sh -pdf readme.md
This script basically just provides a nice wrapper for the commands below. If you're interested in understanding those, keep reading. Otherwise, gfm.sh should complete your setup.
HTML Conversion

Using the Github html5 template:
MERMAID_THEME='neutral' # use gitlab's mermaid theme
pandoc --from markdown --to html5   \
   --self-contained --standalone    \
   --filter pandoc-plantuml         \
   --filter pandoc-mermaid          \
   --lua-filter gitlab-math.lua     \
   --lua-filter fix-links.lua       \
   --katex --template=GitHub.html5  \
   -o <name of output>.html <name of input>.md
The --self-contained option allows links to images, pictures, and mermaid/plantuml graphs to be embedded with data:... URIs instead of relative links. This allows the HTML file to be truly portable, if not a bit large.
NOTE: Using --self-contained did work locally on my laptop, but didn't work with my CI/CD setup. I had to remove this flag in the makefile as a result. See this discussion for more info.
PDF Conversion

The best template I could find out there is Eisvogel, which is a latex template that will help us create a pdf.
MERMAID_THEME='neutral' # use gitlab's mermaid theme
pandoc --from markdown --to latex  \
  --self-contained --standalone    \
  --filter pandoc-plantuml         \
  --filter pandoc-mermaid          \
  --lua-filter gitlab-math.lua     \
  --lua-filter fix-links.lua       \
  --katex --template=eisvogel      \
  -o <name of output>.pdf <name of input>.md
In my experience, this template is super buggy in combination with pandoc-mermaid-filter and pandoc-plantuml-filter. The images pretty regularly spill over the margins of the document, even when using svg images. I'll look into this further to see if I can tweak it, or find a better template.
NOTE: Using --self-contained did work locally on my laptop, but didn't work with my CI/CD setup. I had to remove this flag in the makefile as a result. See this discussion for more info.

  
## fix-links.lua
-- see if the file exists
function file_exists(file)
  local f = io.open(file, "rb")
  if f then f:close() end
  return f ~= nil
end

-- 1. Replaces .md with .html
-- 2. Replaces absolute paths with relative ones
-- 3. Replaces folder links with links to their index / readme
function fix_link (url)
  -- Replace md with html
  fixed_url = url:gsub("%.md", ".html")
  -- Replace project-root-relative (i.e. /path/to/thing.md) paths with relative ones
  fixed_url = fixed_url:gsub("^/", (
    function(s)
      basedir = io.popen("realpath --relative-to=\"$(dirname \"" .. s:sub(2) .. "\")\" .", 'r'):read('*a')
      return basedir:gsub("%s", "") .. "/" .. s:sub(2)
    end
  ))
  -- Allowed readme / index names
  dir_name = {} -- values are functions that return the path to the new link
  dir_name["index.md"] = (function(s) return s:gsub("/?$", "/index.html") end)
  dir_name["INDEX.md"] = (function(s) return s:gsub("/?$", "/INDEX.html") end)
  dir_name["readme.md"] = (function(s) return s:gsub("/?$", "/readme.html") end)
  dir_name["README.md"] = (function(s) return s:gsub("/?$", "/README.html") end)
  -- Is the url pointing to a folder?
  file_or_folder = io.popen("[ -d '" .. fixed_url .. "' ] && echo 'folder'", 'r'):read('*a')
  -- If it's a folder, find the readme / index and replace the link with that.
  if file_or_folder:gsub("%s", "") == "folder" then
    for index,mk_path in pairs(dir_name) do
      if file_exists(fixed_url .. "/" .. index) then
        fixed_url = mk_path(fixed_url)
        break
      end
    end
  end
  return fixed_url
end


function Link(link) link.target = fix_link(link.target); return link end
function Image(img) img.src = fix_link(img.src); return src end
-- return {{Meta = Meta}, {Link = Link, Image = Image}}

## gfm.sh
#!/bin/bash

if [[ $# == 0 ]] || [[ $1 == '-h' ]] || [[ $1 == '--help' ]]; then
  cat <<-EOF
	USAGE: gfm.sh [ OPTIONS ] <markdown file>

	Converts Gitlab Flavored Markdown into PDF or HTML format

	DEFAULT OPTIONS: gfm.sh -html --file <markdown file>

	FORMAT OPTIONS:
	  -pdf           Converts the GFM to pdf
	  -html          Converts the GFM to html

	OUTPUT OPTIONS:
	  -p --print     Prints the generated output
	  -f --file      Creates a file of the same name
	EOF
  exit 0
fi

# Set defaults
filetype='html'
output='file'

# Process Arguments
let "n = $#"
while [[ $n > 1 ]]; do
  echo $n
  case $1 in
    -pdf|-html) filetype="${1/-/}" ;;
    -p|--print|-f|--file)
      output="${1/-/}" ;;
    --)
      shift 1
      break ;;
  esac
  let "n--"
  shift 1
done

# Run script
if [[ $output == 'file' ]]; then
  case $filetype in
    html)
      MERMAID_THEME='neutral'
      pandoc --from markdown --to html5   \
         --self-contained --standalone    \
         --filter pandoc-plantuml         \
         --filter pandoc-mermaid          \
         --lua-filter gitlab-math.lua     \
         --lua-filter fix-links.lua       \
         --katex --template=GitHub.html5  \
         -o "${1/.md/.html}" "$1"
      ;;
    pdf)
      MERMAID_THEME='neutral'
      pandoc --from markdown --to latex  \
        --self-contained --standalone    \
        --filter pandoc-plantuml         \
        --filter pandoc-mermaid          \
        --lua-filter gitlab-math.lua     \
        --lua-filter fix-links.lua       \
        --katex --template=eisvogel      \
        -o "${1/.md/.pdf}" "$1"
      ;;
  esac
else
  case $filetype in
    html)
      MERMAID_THEME='neutral'
      pandoc --from markdown --to html5   \
         --self-contained --standalone    \
         --filter pandoc-plantuml         \
         --filter pandoc-mermaid          \
         --lua-filter gitlab-math.lua     \
         --lua-filter fix-links.lua       \
         --katex --template=GitHub.html5  \
         "$1"
         ;;
    pdf)
      MERMAID_THEME='neutral'
      pandoc --from markdown --to latex  \
        --self-contained --standalone    \
        --filter pandoc-plantuml         \
        --filter pandoc-mermaid          \
        --lua-filter gitlab-math.lua     \
        --lua-filter fix-links.lua       \
        --katex --template=eisvogel      \
        "$1"
        ;;
  esac
fi
Dependency	Rationale
`mermaid-cli`	Required by timofurrer's `pandoc-mermaid-filter`
`plantuml`	Required by timofurrer's `pandoc-plantuml-filter`
`texlive-full`^[1]	Required by the pandoc LaTeX template
My fork of @timofurrer's `pandoc-plantuml-filter`	Allows pandoc to render plantuml code blocks as graphs
My fork of @timofurrer's `pandoc-mermaid-filter`	Allow pandoc to make sense of mermaid code blocks as graphs
This GitHub pandoc html5 template	Used to make the html all pretty :D
This pandoc LaTeX Template	Used to make the pdf all pretty :D
@lierdakil's `gitlab-math.lua` script	Supports gitlab's special math syntax.
My `fix-links.lua` script	Fixes links and urls to resources within the project.
	-- see if the file exists
	function file_exists(file)
	local f = io.open(file, "rb")
	if f then f:close() end
	return f ~= nil
	end

	-- 1. Replaces .md with .html
	-- 2. Replaces absolute paths with relative ones
	-- 3. Replaces folder links with links to their index / readme
	function fix_link (url)
	-- Replace md with html
	fixed_url = url:gsub("%.md", ".html")
	-- Replace project-root-relative (i.e. /path/to/thing.md) paths with relative ones
	fixed_url = fixed_url:gsub("^/", (
	function(s)
	basedir = io.popen("realpath --relative-to=\"$(dirname \"" .. s:sub(2) .. "\")\" .", 'r'):read('*a')
	return basedir:gsub("%s", "") .. "/" .. s:sub(2)
	end
	))
	-- Allowed readme / index names
	dir_name = {} -- values are functions that return the path to the new link
	dir_name["index.md"] = (function(s) return s:gsub("/?$", "/index.html") end)
	dir_name["INDEX.md"] = (function(s) return s:gsub("/?$", "/INDEX.html") end)
	dir_name["readme.md"] = (function(s) return s:gsub("/?$", "/readme.html") end)
	dir_name["README.md"] = (function(s) return s:gsub("/?$", "/README.html") end)
	-- Is the url pointing to a folder?
	file_or_folder = io.popen("[ -d '" .. fixed_url .. "' ] && echo 'folder'", 'r'):read('*a')
	-- If it's a folder, find the readme / index and replace the link with that.
	if file_or_folder:gsub("%s", "") == "folder" then
	for index,mk_path in pairs(dir_name) do
	if file_exists(fixed_url .. "/" .. index) then
	fixed_url = mk_path(fixed_url)
	break
	end
	end
	end
	return fixed_url
	end


	function Link(link) link.target = fix_link(link.target); return link end
	function Image(img) img.src = fix_link(img.src); return src end
	-- return {{Meta = Meta}, {Link = Link, Image = Image}}
	#!/bin/bash

	if [[ $# == 0 ]] \|\| [[ $1 == '-h' ]] \|\| [[ $1 == '--help' ]]; then
	cat <<-EOF
	USAGE: gfm.sh [ OPTIONS ] <markdown file>

	Converts Gitlab Flavored Markdown into PDF or HTML format

	DEFAULT OPTIONS: gfm.sh -html --file <markdown file>

	FORMAT OPTIONS:
	-pdf Converts the GFM to pdf
	-html Converts the GFM to html

	OUTPUT OPTIONS:
	-p --print Prints the generated output
	-f --file Creates a file of the same name
	EOF
	exit 0
	fi

	# Set defaults
	filetype='html'
	output='file'

	# Process Arguments
	let "n = $#"
	while [[ $n > 1 ]]; do
	echo $n
	case $1 in
	-pdf\|-html) filetype="${1/-/}" ;;
	-p\|--print\|-f\|--file)
	output="${1/-/}" ;;
	--)
	shift 1
	break ;;
	esac
	let "n--"
	shift 1
	done

	# Run script
	if [[ $output == 'file' ]]; then
	case $filetype in
	html)
	MERMAID_THEME='neutral'
	pandoc --from markdown --to html5 \
	--self-contained --standalone \
	--filter pandoc-plantuml \
	--filter pandoc-mermaid \
	--lua-filter gitlab-math.lua \
	--lua-filter fix-links.lua \
	--katex --template=GitHub.html5 \
	-o "${1/.md/.html}" "$1"
	;;
	pdf)
	MERMAID_THEME='neutral'
	pandoc --from markdown --to latex \
	--self-contained --standalone \
	--filter pandoc-plantuml \
	--filter pandoc-mermaid \
	--lua-filter gitlab-math.lua \
	--lua-filter fix-links.lua \
	--katex --template=eisvogel \
	-o "${1/.md/.pdf}" "$1"
	;;
	esac
	else
	case $filetype in
	html)
	MERMAID_THEME='neutral'
	pandoc --from markdown --to html5 \
	--self-contained --standalone \
	--filter pandoc-plantuml \
	--filter pandoc-mermaid \
	--lua-filter gitlab-math.lua \
	--lua-filter fix-links.lua \
	--katex --template=GitHub.html5 \
	"$1"
	;;
	pdf)
	MERMAID_THEME='neutral'
	pandoc --from markdown --to latex \
	--self-contained --standalone \
	--filter pandoc-plantuml \
	--filter pandoc-mermaid \
	--lua-filter gitlab-math.lua \
	--lua-filter fix-links.lua \
	--katex --template=eisvogel \
	"$1"
	;;
	esac
	fi