Skip to content

Instantly share code, notes, and snippets.

@shiondev
shiondev / get_crawl_results_sample
Last active August 29, 2015 13:57
Get download link for results of crawl
[
"http://datafiniti-voltron-results.s3.amazonaws.com/12/34.txt?AWSAccessKeyId=xxx&Signature=yyy&Expires=1395859223"
]
@shiondev
shiondev / view_crawl_sample
Created March 18, 2014 20:13
View a crawl
{
"id": 25,
"name": "name_of_crawl",
"user": "your_user_token",
"user_agent": "008",
"app": "full_page_content.js",
"urllist": "name_of_url_list",
"data": "undefined",
"max_depth": 1,
"max_urls": 10,
@shiondev
shiondev / urllists_sample
Created March 18, 2014 20:10
Sample response from urllists call
[
{
"location": "urllists/your_user_token/name_of_url_list",
"name": "name_of_url_list",
"user": "your_user_token",
"date_created": "2014-3-18"
}
]
@shiondev
shiondev / get_crawl_results_sample
Last active August 29, 2015 13:57
Get download link to results of crawl
> curl -X GET https://your_user_token:@api.80legs.com/v2/results/name_of_crawl
@shiondev
shiondev / crawl-status_sample
Last active August 29, 2015 13:57
Get the status of a crawl
> curl -X GET https://your_user_token:@api.80legs.com/v2/crawls/name_of_crawl
@shiondev
shiondev / create_crawl_sample
Last active August 29, 2015 13:57
Create a crawl
> curl -X PUT https://your_user_token:@api.80legs.com/v2/crawls/name_of_crawl -H "Content-Type: application/json" -d "{\"app\": \"full_page_content.js\", \"urllist\": \"name_of_url_list\", \"data\": \"name_of_data_file\", \"max_depth\": 1, \"max_urls\": 10 }" -i
@shiondev
shiondev / url_list_status_sample
Last active August 29, 2015 13:57
Get status of URL list
> curl -X GET https://your_user_token:@api.80legs.com/v2/urllists/ -H "Content-Type: application/octet-stream"
@shiondev
shiondev / upload_url_list_sample
Last active August 29, 2015 13:57
Upload a URL list to 80legs
> curl -X PUT https://your_user_token:@api.80legs.com/v2/urllists/name_of_url_list -H "Content-Type: application/octet-stream" --data-binary "[\"http://www.example.com/\", \"http://www.sample.com/\", \"http://www.test.com/\"]" -i
@shiondev
shiondev / sample_datafiniti_product_json
Created March 17, 2014 16:45
Sample Datafiniti product record in JSON
{
"total": 1,
"records": [
{
"key": "857603002760",
"name": "Sharkoon - X-tatic S7 Gaming Headset For Playstation 3 Xbox 360 and Windows",
"type": "product",
"dateUpdated": "2013-11-07T07:53:02Z",
"category": [
"Computers & Tablets",
@shiondev
shiondev / php_sample.php
Created March 15, 2014 16:08
Sample PHP for parsing Datafiniti JSON record
<?php
$json_str = file_get_contents("https://gist.githubusercontent.com/shiondev/9569051/raw/69b7d357190c3c5dde7f23bd220a37752b290a2c/sample_datafiniti_json_with_reviews");
$json_obj = json_decode($json_str,true);
echo "Name: " . $json_obj['records'][0]['name'];
echo "Text of 1st Review: " . $json_obj['records'][0]['review'][0]['text'];
?>