Skip to content

Instantly share code, notes, and snippets.

@davepagurek
Created June 23, 2015 23:38
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save davepagurek/94967956d42fe724c94a to your computer and use it in GitHub Desktop.
Save davepagurek/94967956d42fe724c94a to your computer and use it in GitHub Desktop.
Get NATA entry info
#!/usr/bin/perl
use v5.10;
use strict;
use warnings;
use URI;
use Web::Scraper;
use Encode;
use Data::Dumper;
# USAGE
#
# ./scraper.pl
# or
# perl scraper.pl
#
# Change $round to scrape http://www.ngtournament.com/$round for the entrant list.
# It then looks at the $moviesToCheck recent movies from each user's Newgrounds
# page to find movies tagged with $roundTag.
my $round = "novice";
my $roundTag = "nata2015open";
my $moviesToCheck = 2;
my $participants = scraper {
process "h3 + ul li", "entrants[]" => "TEXT";
result "entrants";
};
my $movies = scraper {
process ".movies a", "movies[]" => {
url => '@href',
title => scraper {
process "span strong", "title" => "TEXT";
result "title";
}
};
result "movies";
};
my $entry = scraper {
process ".tags ~ dd a", "tags[]" => "TEXT";
result "tags";
};
print "Entrants:\n\n";
my $entrants = $participants->scrape( URI->new("http://www.ngtournament.com/$round") );
for my $entrant (@{ $entrants }) {
print "$entrant\n";
my @matchedMovies = grep {
$roundTag ~~ @{ $entry->scrape( URI->new($_->{url}) || [] ) }
} splice @{ $movies->scrape( URI->new("http://$entrant.newgrounds.com/movies/") ) }, 0, $moviesToCheck;
if (my $movie = $matchedMovies[0]) {
print "\t$movie->{title}\n";
print "\t$movie->{url}\n";
} else {
print "\tNo entry yet!\n";
}
print "\n";
}
__END__
Output looks like:
Entrants:
AmazingSpin
Lizardman Xander (NATA2015OPEN)
http://www.newgrounds.com/portal/view/659337
BooneBum
Strauss
http://www.newgrounds.com/portal/view/659456
Britbau
Third Law
http://www.newgrounds.com/portal/view/659436
brynimation
Ten amazing facts!
http://www.newgrounds.com/portal/view/659430
CarelessShenanigans
Ten Fabulous Facts!!!
http://www.newgrounds.com/portal/view/659299
DocJoshimitsu
NATA Open Round 2015
http://www.newgrounds.com/portal/view/659365
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment