- $ crontab -e
- Add
@reboot sh /path/to/find_or_start_scraper.sh
- Be sure to replace
/path/to/
with the correct path.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
<!doctype html> | |
<html> | |
<head> | |
<meta charset="UTF-8"> | |
<title>Document</title> | |
</head> | |
<body> | |
<div> | |
<form> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
<!doctype html> | |
<html> | |
<head> | |
<meta charset="UTF-8"> | |
<title>Document</title> | |
</head> | |
<body> | |
<div> | |
<form> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#! /bin/bash | |
### IMPORTANT ### | |
# Remove all 3 instances of /path/to/ and replaces them with proper paths | |
################# | |
logfile='/path/to/logs.txt' | |
start_text='Starting MGS_scraper.py' | |
current_time=$(date) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
from bs4 import BeautifulSoup | |
import csv | |
from datetime import datetime | |
import html | |
import requests | |
import sqlite3 | |
from time import time, sleep | |
# README | |
# pip install beautifulsoup4 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
describe("flatten an array of arbitrarily nested arrays of integers into a flat array of integers", function () { | |
it("Should return empty array when given empty array provided", function () { | |
expect(flatten([])).toEqual([]); | |
}); | |
it("Should return a flat array when given a flat array provided", function () { | |
expect(flatten([1])).toEqual([1]); | |
}); |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
function convertDecimalsToDegrees(num) { | |
var integer = num.toString().split('.')[0] | |
var decimal = '.' + num.toString().split('.')[1] | |
var convert = decimal * 60; | |
var min = convert.toString() | |
return integer + ' ' + min; | |
} |
I hereby claim:
- I am andygimma on github.
- I am andygimma (https://keybase.io/andygimma) on keybase.
- I have a public key ASAlRW9l5JgmaW0lzmssQza-kiic8E0Qfm8VP4uAjTaC9wo
To claim this, I am signing this object:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#!/usr/bin/env python | |
import urllib2 | |
import thread | |
import sys | |
def fetch_api(url): | |
res = urllib2.urlopen(url) | |
def fetch_api_hits(url, number_of_hits): | |
for i in range(number_of_hits): |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
var sites = { | |
A1: { | |
name: "Wendy Williams", | |
city: "Houston", | |
work_type: "flood", | |
urgency: 5 | |
}, | |
A2: { | |
name: "Jello Biafra", | |
city: "Houston", |
NewerOlder