arXMLiv 08.2018, MathML element report

apply_cutoffs.pl

#!/usr/bin/env perl
# Applies cutoffs to the very noisy 250 MB mathml_statistics.txt 
#  which was generated by llamapun over arXMLiv 08.2018.
#
# It rewrites to a CSV file, throwing out all known erroneous markup, including:
# - discard all SVG-associated markup (wrongly in MathML)
# - discard all (non-math) HTML-associated markup (wrongly in MathML)
# - discard all XMath-associated markup (wrongly in MathML)
# - less noisy for uninteresting values (numbers with known units, hex colors, open-ended id schemes, etc)
#
# And computes the ratios of elements when compared to the number of all <math> elements reported.

use warnings;
use strict;
use Data::Dumper;

open my $fh, "<", "mathml_statistics.txt";
open my $pres_fh, ">", "pmml_cutoffs.csv";
open my $content_fh, ">", "cmml_cutoffs.csv";
print $pres_fh "name\@attr\[value\], relative %, frequency\n";
print $content_fh "name\@attr\[value\], relative %, frequency\n";

my %discarded = ();
my @kept = ();
my $math_count = 0;

for my $line (<$fh>) {
  if ($line =~ /^(?:(:?sv)?g|rect|path|circle|foreignobject|marker|ellipse|object|use|fe(flood|merge(node)?)|pattern|text|radialgradient|stop|filter|clippath|lineargradient|switch|defs)(\@|\s)/) {
    $discarded{"svg-variety"} = 1 + ($discarded{"svg-variety"} || 0);
  } elsif ($line =~ /^xm/) {
    $discarded{"xmath-variety"} = 1 + ($discarded{"xmath-variety"} || 0);
  } elsif ($line =~ /^(div|span|img|a|th|td|tr|table|tbody|thead|tfoot|section|code|br|su[pb]|p|h\d|abbr|cite|figure|[uo]l|del|li|d[ldt]|blockquote|figcaption|pre)(\@|\s)/) {
    $discarded{"html-variety"} = 1 + ($discarded{"html-variety"} || 0);
  } elsif ($line =~ /math\@style\[background\-color:#/ || $line=~ /(style|color)\[\#/) {
    my $key = "hex-color";
    $discarded{$key} = 1 + ($discarded{$key} || 0);
  } elsif ($line =~ /\@(fragid|idref|math(?:color|background))\[/) {
    my $key = "\@$1\[";
    $discarded{$key} = 1 + ($discarded{$key} || 0);
  } elsif ($line =~ /((?:p[tx]|ex|%)\])\s/) { # discard attr values with known units
    $discarded{$1} = 1 + ($discarded{$1} || 0);
  } elsif ($line =~ /^document_count\s(\d+)/) {
    print STDERR "Document count: $1\n";
  } elsif ($line =~ /^math\s(\d+)/) {
    $math_count = int($1) * 1.0;
  } else { # keep the rest
    push(@kept, $line);
  }
}

for my $line(@kept) {
  my ($k,$v) = split(/\s+/,$line);
  my $ratio = sprintf('%.2f',(int($v)* 100.0) / $math_count);
  my $csv_row = "$k, $ratio, $v\n";
    if ($k =~ /^m/ && $k ne 'minus') {
      print $pres_fh $csv_row;
    } else {
      print $content_fh $csv_row;
    }
}

close $fh;
close $pres_fh;
close $content_fh;

$Data::Dumper::Sortkeys = sub {
    # Using <=> to sort numeric values
    [ sort { $_[0]->{$b} <=> $_[0]->{$a} } keys %{ $_[0] } ]
};

print Dumper(\%discarded);

cmml_cutoffs.csv

pmml_cutoffs.csv

The discarded data via the cutoffs script, starting from the 250 MB report collection was:

Document count: 1232184
\%discarded = {
          'xmath-variety' => 4690212,
          'svg-variety' => 344924,
          'html-variety' => 153042,
          'pt]' => 23825,
          'hex-color' => 5570,
          '@mathbackground[' => 1762,
          '%]' => 1241,
          'px]' => 79,
          'ex]' => 16,
          '@mathcolor[' => 10
        };

The kept data can be seen in the cmml and pmml reports in this Gist.

Some inaccuracies I've spotted - since I split the presentation/content elements after the report was generated, my heuristic can let some elements slip in the presentation report if they're content elements starting with m, embarrassingly enough. So matrix and matrixrow are listed in the wrong report, potentially others? Probably quite minor, since this isn't the focus of the current study.

Another note that the cmml report is not really pure Content MathML, but rather all elements that are meaningful MathML (or at least I tried to denoise to only those!) which are not within presentation MathML. So constructs such as semantics and annotation-xml can be found in the cmml report, as well as any remaining non-mathml noise...