Skip to content

Instantly share code, notes, and snippets.

@andreaselia
Last active October 21, 2021 16:43
Show Gist options
  • Save andreaselia/962dc9060b3b77308ab61bfd8bfa7449 to your computer and use it in GitHub Desktop.
Save andreaselia/962dc9060b3b77308ab61bfd8bfa7449 to your computer and use it in GitHub Desktop.
<?php
namespace App\Console\Commands;
use Illuminate\Console\Command;
class ScrapeFunko extends Command
{
/**
* The name and signature of the console command.
*
* @var string
*/
protected $signature = 'scrape:funko';
/**
* The console command description.
*
* @var string
*/
protected $description = 'Funko POP! Vinyl Scraper';
/**
* The list of funko collection slugs.
*
* @var array
*/
protected $collections = [
'animation',
'disney',
'games',
'heroes',
'marvel',
'monster-high',
'movies',
'pets',
'rocks',
'sports',
'star-wars',
'television',
'the-vault',
'the-vote',
'ufc',
];
/**
* Create a new command instance.
*
* @return void
*/
public function __construct()
{
parent::__construct();
}
/**
* Execute the console command.
*
* @return mixed
*/
public function handle()
{
foreach ($collections as $collection) {
$this->scrape($collection);
}
}
/**
* For scraping data for the specified collection.
*
* @param string $collection
* @return boolean
*/
public static function scrape($collection)
{
$crawler = Goutte::request('GET', env('FUNKO_POP_URL').'/'.$collection);
$pages = ($crawler->filter('footer .pagination li')->count() > 0)
? $crawler->filter('footer .pagination li:nth-last-child(2)')->text()
: 0
;
for ($i = 0; $i < $pages + 1; $i++) {
if ($i != 0) {
$crawler = Goutte::request('GET', env('FUNKO_POP_URL').'/'.$collection.'?page='.$i);
}
$crawler->filter('.product-item')->each(function ($node) {
$sku = explode('#', $node->filter('.product-sku')->text())[1];
$title = trim($node->filter('.title a')->text());
print_r($sku.', '.$title);
});
}
return true;
}
}
@13ens
Copy link

13ens commented May 8, 2017

[Symfony\Component\Console\Exception\CommandNotFoundException] There are no commands defined in the "scrape" namespace.

@DevinNorgarb
Copy link

@13ens remember to register the command in the app/Console/Kernel.php file like so:
protected $commands = [ \App\Console\Commands\ScrapeFunko::class, ];
and import the command at the top of the Kernel class:
use App\Console\Commands\ScrapeFunko;
then run php artisan in the terminal and it should be listed as a command there.

Hope this helps

@13en91
Copy link

13en91 commented May 24, 2017

Thank you Devin!

@khizershahid
Copy link

Help me solve this.
[ErrorException]
Undefined variable: collections

@khizershahid
Copy link

i wrote $this->collections in foreach loop and problem resolved
now the error is

 [Symfony\Component\Debug\Exception\FatalThrowableError]
  Class 'App\Console\Commands\Goutte' not found

i have written goutte in config/app.php
but still getting error

@khizershahid
Copy link

any body can help me with this
[GuzzleHttp\Exception\RequestException] cURL error 60: SSL certificate problem: unable to get local issuer certificate (see http://curl.haxx.se/libcurl/c/libcurl-errors.html)

@anandiamy
Copy link

[GuzzleHttp\Exception\RequestException]
cURL error 56: Recv failure: Connection reset by peer (see http://curl.haxx.se/libcurl/c/libcurl-errors.html)

how to solve this error ?

@mugiayomi
Copy link

@khizershahid

Try this :
use Goutte\Client;

$goutte = new Client();
$crawler = $goutte->request('GET', $url.'/'.$collection);

@bw70316
Copy link

bw70316 commented Jul 13, 2018

I was able to work through the errors to get the scrape command to return nothing in my terminal. How do I see the scrape itself? Moreover, how do I see it on my site? I think this is really neat.

@ludo1960
Copy link

ErrorException : Undefined variable: collections

at /var/www/test.local.com/app/Console/Commands/ScrapeFunko.php:57

Using above code:??

@MrKriKri
Copy link

i want result with html tag ??? how to??

@Ahmed-Aboud
Copy link

i want result with html tag ??? how to??

->html()

@Ahmed-Aboud
Copy link

use \Goutte::request instead of Goutte::request if you get not found exception

@Hamza160
Copy link

Couldn't connect to server for "http://localhost/animation".

at vendor/symfony/http-client/Chunk/ErrorChunk.php:65

@andreaselia
Copy link
Author

Couldn't connect to server for "http://localhost/animation".

at vendor/symfony/http-client/Chunk/ErrorChunk.php:65

I have no idea if this still even still works, but make sure you have the FUNKO_POP_URL in your env file set at the funko website URL, e.g. https://www.funko.com

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment