Create a gist now

Instantly share code, notes, and snippets.

What would you like to do?
Using an indexed list of anime links from MAL and scraping them one by one using Jikan - my unofficial MAL API
<?php
// prerequisites
require 'mal-uapi.php';
$mal = new \MAL\GET;
$database = array();
// let's load the database of links
// I've formated it as the canonical links from the sitemap separated with commas, we just need the ID so we'll use ReGeX.
$animeIDs = array();
$animeArr = explode(",", file_get_contents("links.db"));
foreach ($animeArr as $key => $value) {
$matches = array();
preg_match("~https://myanimelist.net/anime/(.*)/(.*)~", $value, $matches);
$animeIDs[] = array($matches[1], $matches[2]);
}
echo "Prepared ".count($animeIDs)." Anime for scraping\n";
// Now lets iterate through those IDS and fetch list!
$lastID = $animeIDs[count($animeIDs)-1][0];
foreach ($animeIDs as $key => $value) {
echo "Fetching Anime ".$value[0]."/".$lastID." | ".round($key/count($animeIDs)*100, 3)."% complete \r";
$database[$value[0]] = $mal->anime((int)$value[0])->data;
// save to my database
file_put_contents("db.json", json_encode($database));
}
?>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment