Skip to content

Instantly share code, notes, and snippets.

@alex-moreno
Created November 27, 2020 09:19
Show Gist options
  • Save alex-moreno/39c273a6be0fe4396d04746582fd6e16 to your computer and use it in GitHub Desktop.
Save alex-moreno/39c273a6be0fe4396d04746582fd6e16 to your computer and use it in GitHub Desktop.
Using promise and concurrency in php to trigger multiple requests
foreach ($listOfSites as $site) {
file_put_contents('/tmp/lockprocess.lock','locked');
$url = $site[0];
// Add loop here to iterate if concurrency is bigger than a threshold.
echo "starting ASYNC request for $url" . PHP_EOL;
$promises[] = $client->requestAsync('GET', $url, [
'headers' => [
'User-Agent' => 'AcquiaScrapperBot/0.1',
'Accept' => 'application/json',
'X-WSF-Sesame' => 'k8nBKu7bK6rNZL3t'
],
'http_errors' => false
])
->then(function($response) use ($url){
echo PHP_EOL . 'querying:::' . $url . 'timestamp: ' . date("H:i:s");
return array($url, $response->getStatusCode(), $response->getBody()->getContents()); //
});
}
$results = Promise\unwrap($promises);
foreach ($results as $result) {
$siteCrawled = Array();
$siteCrawled['url'] = $result[0];
$siteCrawled['statusCode'] = $result[1];
$siteCrawled['size'] = strlen($result[2]);
$siteCrawled['footprint'] = md5($result[2]);
$csvManager->writeCsvLine($siteCrawled,$fileToWrite);
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment