on_latex_document_compilation.md

On LaTeX Document Compilation

Reddit user Paimun asks:

why does it feel like I need to compile six times while holding the computer at the right angle above my head to get documents to compile right sometimes? It's arcane black magic...

Something that almost everyone that gets into LaTeX notices once they start writing non-trivial documents is that they have to run their LaTeX compiler multiple times in order to get a correct document without ? marks scattered everywhere. So why does this happen?

I think it's easiest if I walk through compiling such a document. Here's one project I wrote in grad school, a research proposal:

$ ls
build.sh                lolg_scale.eps   proposal.bib    simple.png  snow_5.png
CGC2010_Grad_cover.doc  lolg_scale.png   proposal.ps     slices.png  snow_6.png
compilation.ps          m_ellipsoid.eps  proposalR1.tex  snow_1.png  snow_face.png
compilation.tex         m_ellipsoid.png  proposal.tex    snow_2.png
isotherms.png           nature.bst       resume.tex      snow_3.png
letterofsupport.pdf     proposal.bbl     Science.bst     snow_4.png

I wrote myself a little build script, build.sh, which looks like this:

#!/usr/bin/env bash

pdflatex resume
xelatex proposal
bibtex proposal
xelatex proposal
xelatex proposal
pdflatex compilation

This project has a few interesting properties. One is that it generates two documents, resume.pdf and proposal.pdf, and then combines them together in a third document, compilation.pdf, using the pdfpages package.

Because this step isn't really relevant to the question, we shall instead focus on the steps to build proposal.pdf:

xelatex proposal
bibtex proposal
xelatex proposal
xelatex proposal

The second interesting thing here is that this project uses xelatex instead of pdflatex. This is because there was a requirement that the proposal be in Times New Roman font. If you find you need to use ttf fonts in LateX, xelatex is pretty sweet. Aside from that though, this example is generally the same.

So, what happens when I run xelatex proposal for the first time? Well, it chugs along, dumps a bunch of spam to the screen and exits successfully. However, there are a bunch of reference and citation warnings:

$ xelatex proposal | grep Warning
LaTeX Warning: Reference `fig:snow' on page 1 undefined on input line 39.
LaTeX Warning: Citation `sturm2' on page 1 undefined on input line 39.
LaTeX Warning: Reference `fig:snow' on page 1 undefined on input line 39.
LaTeX Warning: `h' float specifier changed to `ht'.
LaTeX Warning: Reference `fig:snow' on page 1 undefined on input line 53.
LaTeX Warning: Reference `fig:snow' on page 1 undefined on input line 53.
LaTeX Warning: Reference `fig:snow' on page 1 undefined on input line 53.
LaTeX Warning: Citation `sturm3' on page 2 undefined on input line 55.
LaTeX Warning: Citation `conductivitymodels' on page 2 undefined on input line 
LaTeX Warning: Citation `mustructure2' on page 2 undefined on input line 55.
LaTeX Warning: Citation `sturm1' on page 2 undefined on input line 55.
LaTeX Warning: Citation `probetheory' on page 3 undefined on input line 68.
LaTeX Warning: Citation `werc' on page 3 undefined on input line 68.
LaTeX Warning: Citation `probetheory' on page 3 undefined on input line 74.
LaTeX Warning: Citation `ktb' on page 4 undefined on input line 86.
LaTeX Warning: There were undefined references.
LaTeX Warning: Label(s) may have changed. Rerun to get cross-references right.

Most of these warnings mean that, somewhere, something got question-marked. In fact, it appears that every figure reference and citation reference was undefined. So where should they be defined?

I happen to have this project under git, so I can get a list of files generated by running a git command:

$ git clean -xdn
Would remove proposal.aux
Would remove proposal.log
Would remove proposal.pdf

We expected proposal.pdf and proposal.log, but what is proposal.aux?

As it turns out, proposal.aux contains our references!

$ cat proposal.aux 
\relax 
\citation{sturm2}
\@writefile{toc}{\contentsline {section}{\numberline {1}Introduction and Background}{1}}
\citation{sturm3}
\citation{conductivitymodels}
\citation{mustructure2}
\citation{sturm1}
\providecommand*\caption@xref[2]{\@setref\relax\@undefined{#1}}
\newlabel{fig:snow}{{\caption@xref {fig:snow}{ on input line 43}}{2}}
\@writefile{lof}{\contentsline {figure}{\numberline {1}{\ignorespaces Mechanisms for the Development of Anisotropy in Snow\relax }}{2}}
\citation{probetheory}
\citation{werc}
\citation{probetheory}
\newlabel{fig:needleprobe}{{\caption@xref {fig:needleprobe}{ on input line 59}}{3}}
\@writefile{lof}{\contentsline {figure}{\numberline {2}{\ignorespaces A COMSOL model of a needle probe in snow. The cube represents a block of snow, while the needle probe itself appears as a thin line running part-way down the middle.\relax }}{3}}
\citation{ktb}
\newlabel{fig:loggraph}{{\caption@xref {fig:loggraph}{ on input line 79}}{4}}
\@writefile{lof}{\contentsline {figure}{\numberline {3}{\ignorespaces When plotting \(T\) vs. \(\qopname  \relax o{ln}(t)\) of a needle probe, the slope of the linear portion may be used to calculate the heat conductivity of an isotropic medium.\relax }}{4}}
\@writefile{toc}{\contentsline {section}{\numberline {2}Objectives and Methodology}{4}}
\@writefile{toc}{\contentsline {subsection}{\numberline {2.1}Numerical Modeling}{4}}
\newlabel{fig:isotherms}{{\caption@xref {fig:isotherms}{ on input line 112}}{5}}
\@writefile{lof}{\contentsline {figure}{\numberline {4}{\ignorespaces The same COMSOL model as in figure 2 has been solved. The surfaces plotted in this visualization represent isotherms in the snow after the needle probe has operated for a "long time." The needle probe itself appears as a thin line running down the middle.\relax }}{5}}
\@writefile{toc}{\contentsline {subsection}{\numberline {2.2}Experimental Corroboration}{6}}
\newlabel{fig:m_ellipsoid}{{\caption@xref {fig:m_ellipsoid}{ on input line 124}}{6}}
\@writefile{lof}{\contentsline {figure}{\numberline {5}{\ignorespaces One way to visualize a symmetric, positive-definite matrix such as \(K\) is to plot the ellipsoid \(\mathaccentV {vec}87E{x}^TK\mathaccentV {vec}87E{x}=1\). The length of a vector \(\mathaccentV {vec}87E{x}\) which satisfies this equation is equal to \(1/\sqrt  {k}\) for the effective thermal conductivity \(k\) in that direction.\relax }}{6}}
\@writefile{toc}{\contentsline {subsection}{\numberline {2.3}Field Application Testing}{6}}
\citation{*}
\bibstyle{Science}
\bibdata{proposal}
\bibcite{sturm2}{1}
\bibcite{sturm3}{2}
\bibcite{conductivitymodels}{3}
\bibcite{mustructure2}{4}
\bibcite{sturm1}{5}
\bibcite{probetheory}{6}
\bibcite{werc}{7}
\bibcite{ktb}{8}
\bibcite{temprange}{9}
\bibcite{mustructure1}{10}

Every time LaTeX runs, it's able to extract these references and write them to a .aux file, and it's able to read references from an existing .aux file. However, the LaTeX compiler only does one "pass" on every invocation, meaning the user has to manage the "dependency resolution" by-hand or using tools like make and bash.

The next step in the build process for this document is running BibTeX against it. Why run xelatex first? BibTeX depends on the aux file in order to work:

$ bibtex proposal
I couldn't open file name `proposal.aux'

Running BibTeX generates your bibliography in a .bbl file:

$ cat proposal.bbl
\begin{thebibliography}{10}

\bibitem{sturm2}
M.~Sturm, J.~B. Johnson, {\it Journal of Geophysical Research\/} {\bf 97}, 2129
  (1992).

\bibitem{sturm3}
M.~Sturm, J.~Johnson, {\it Journal of Geophysical Research\/} {\bf 96}, 11675
  (1991).

\bibitem{conductivitymodels}
T.~Yun, J.~Santamarina, {\it Granular Matter\/} {\bf 10}, 197 (2008).

\bibitem{mustructure2}
T.~Kaempfer, M.~Schneebeli, S.~Sokratov, {\it Geophysical Research Letters\/}
  {\bf 32} (2005).

\bibitem{sturm1}
M.~Sturm, D.~K. Perovich, J.~Holmgren, {\it Journal of Geophysical Research\/}
  {\bf 107} (2002).

\bibitem{probetheory}
D.~{De Vries}, A.~Peck, {\it Australian Journal of Physics\/} {\bf 11}, 255
  (1957).

\bibitem{werc}
P.~Overduin, D.~Kane, W.~{van Loon}, {\it Cold Regions Science and
  Technology\/} {\bf 45}, 8 (2006).

\bibitem{ktb}
E.~Huenges, H.~Burkhardt, K.~Erbas, {\it Scientific Drilling\/} {\bf 1}, 224
  (1990).

\bibitem{temprange}
J.~Putkonen, {\it Permafrost and Periglacial Processes\/} {\bf 14}, 343 (2003).

\bibitem{mustructure1}
P.~Satyawalio, A.~Singh, {\it Journal of Earth Systems Science\/} {\bf 117},
  465 (2008).

\end{thebibliography}

Now running xelatex again generates a slightly different warnings:

josh@onix:/tmp/anisotropy/papers/proposal$ xelatex proposal | grep Warning
LaTeX Warning: Citation `sturm2' on page 1 undefined on input line 39.
LaTeX Warning: `h' float specifier changed to `ht'.
LaTeX Warning: Citation `sturm3' on page 2 undefined on input line 55.
LaTeX Warning: Citation `conductivitymodels' on page 2 undefined on input line 
LaTeX Warning: Citation `mustructure2' on page 2 undefined on input line 55.
LaTeX Warning: Citation `sturm1' on page 2 undefined on input line 55.
LaTeX Warning: Citation `probetheory' on page 3 undefined on input line 68.
LaTeX Warning: Citation `werc' on page 3 undefined on input line 68.
LaTeX Warning: Citation `probetheory' on page 3 undefined on input line 74.
LaTeX Warning: Citation `ktb' on page 4 undefined on input line 86.
LaTeX Warning: There were undefined references.
LaTeX Warning: Label(s) may have changed. Rerun to get cross-references right.

In this case though we at least have the bibtex stuff sorted out, so running it one more time should clear those right up:

$ xelatex proposal | grep Warning
LaTeX Warning: `h' float specifier changed to `ht'.

Finally, a warning we don't care about!

...and that's how it's made.