Skip to content

Instantly share code, notes, and snippets.

@andykais
Last active September 18, 2018 16:28
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save andykais/1636bc8a1ed20a8c9c09af0c09318388 to your computer and use it in GitHub Desktop.
Save andykais/1636bc8a1ed20a8c9c09af0c09318388 to your computer and use it in GitHub Desktop.
Possible new spec for passing vars like login to scraper. Essentially passing one scraper as input variables to another scraper. New syntax includes limiting the number of parsed values ("max"), parsing headers ("expect": "header") and "asInput" clause,
{
"input": ["username","password"],
"scrape": {
"download": {
"urlTemplate": "http://example-site.com/login",
"headers": {
"username": "{username}",
"password": "{password"
},
"parse": {
"expect": "header",
"selector": "SESSIONID",
"max": 1
},
"asInput": {
"alias": "token",
"inputFor": {
"input": ["token", "searchQuery"],
"scrape": {
"download": {
"urlTemplate": "https://example-site.com/search?{searchQuery}",
"headers": {
"SESSIONID": "{token}"
}
},
"parse": {
"selector": "li.item span"
}
}
}
}
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment