How do we use babel: Empty Lines

emptylines.md

The Firefox debugger has undergone a massive rewrite in the last two years, moving away from old Mozilla-specific technologies like XUL etc, to more modern technologies like React, webpack and babel.

Babel is a tool for compiling Javascript into Javascript. It generates an Abstract Syntax Tree (AST) which can be transformed, transversed or manipulated in various ways for use. Babel and AST's have played a major part in growth of the modern web tooling ecosystem.

Over the past year, we have used babel extensively in building the debugger, from disabling non-executable lines so breakpoints cannot be set to highlighting out of scope code and much more.

I felt it would be nice to write a couple of blog posts, documenting some of the use cases and looking into some debugger internals as we go.

And by the way the Firefox DevTools is really cool now, you should try it! 😉

In this blog post, we will look at one of our simpler use cases.

Use Case One: Empty Lines

Problem

We want to disable lines in the editor that do not have executable code so breakpoints can't be set where not useful (as shown in the figure below).

The grayed out lines (as shown in the pic above) indicate which lines are disabled.

Lets take a look at the technical details of how this is achieved.

Solution

At a high level, we parse the source and get the AST, which is then used to get the data for the disabled lines that the UI renders.

Firstly, when any source file (in this case todo.js ) is selected and displayed, a call is made to the getEmptyLines function passing the source object which contains all the data about the source selected.

export default function getEmptyLines(sourceToJS) {
  if (!sourceToJS) {
    return null;
  }

  const ast = getAst(sourceToJS);
  if (!ast || !ast.comments) {
    return [];
  }
  ...
  // more code to get to later
}

This takes the source and tries to get the AST for it by calling thegetAst function.

getAst parses the source using the Babylon parser, adds the generated AST to the cache, and returns the AST, as shown below

...
import * as babylon from "babylon";
...

let ASTs = new Map();

function parse(code, opts) {
  return babylon.parse(
    code, Object.assign({}, opts, {
      sourceType: "module",
      ...
    })
  );
}

...

// Parse the source and return the AST
export function getAst(source: Source) {
  if (!source || !source.text) {
    return {};
  }

  if (ASTs.has(source.id)) {
    return ASTs.get(source.id);
  }

  let ast = {};
  if (source.contentType == "text/html") {
    ...
  } else if (source.contentType == "text/javascript") {
    ast = parse(source.text);
  }
  ASTs.set(source.id, ast);
  return ast;
}

Once we have the AST, we can do a lot of poweful things. The screenshot below shows a part of what the AST for todo.js looks like.

For a complete view of the AST see here.

Yay, we have our AST!

Back to the getEmptyLines function, next we want to get an array of lines that have executable code by calling getExecutableLines , then get an array of all the lines for the source by calling getLines . The difference of both gives our array of empty lines.

export default function getEmptyLines(sourceToJS) {
  // code to get the ast as shown above
  ...
  const executableLines = getExecutableLines(ast);
  const lines = getLines(ast);
  return difference(lines, executableLines);
}

getExecutable takes the tokens from the AST, and simply filters out all comment or EOF tokens, and since only the lines are needed, it maps over results returning just the start lines. There will be lines with multiple tokens, this will cause a line to show up more than once in the array, so it makes sure the lines are unique before returning the array.

const commentTokens = ["CommentBlock", "CommentLine"];
...

// The following sequence stores lines which have executable code
// (contents other than comments or EOF, regardless of line position)
function getExecutableLines(ast) {
  const lines = ast.tokens
    .filter(
      token =>
        !commentTokens.includes(token.type) &&
        (!token.type || (token.type.label && token.type.label != "eof"))
    )
    .map(token => token.loc.start.line - 1);

  return uniq(lines);
}

Now we have our executable lines, getLines returns an array of all the lines in source, from 0 to the end line on the last token in the AST.

...

function fillRange(start, end) {
  return Array(end - start + 1)
    .fill()
    .map((item, index) => start + index);
}

// Populates a pre-filled array of every line number,
function getLines(ast) {
  return fillRange(0, ast.tokens[ast.tokens.length - 1].loc.end.line);
}

Once all the neccesarry line data is gotten, the difference in the two line arrays is determined using the lodash difference utility function.

export default function getEmptyLines(sourceToJS) {
  // code as shown above
  ...
  return difference(lines, executableLines);
}

As you might noticed, the getEmptyLines function handles the bulk of the work, you might also be thinking, this looks perfomance intensive! You would be right! To deal with the perfomance hit that might be encountered while processing really large files, getEmptylines is run in a web worker.

So finally here is what the array of empty lines for the todo.js file

[0, 2, 5, 6, 7, 8, 9, 11, 12, 17, 18, 26, 27]

Does this match our disabled lines in the figure above?

Yay it does! Noting that codemirror lines are one based (starts at 1 rather than 0).

Conclusion

Babel powers a lot in the Firefox debugger these days. If you are interested in working on these and other cool stuff. Check us out here.