Skip to content

Instantly share code, notes, and snippets.

@byuniqueman
Created September 15, 2023 19:20
Show Gist options
  • Save byuniqueman/2c007a4880d9ce19bbe5174253f905d4 to your computer and use it in GitHub Desktop.
Save byuniqueman/2c007a4880d9ce19bbe5174253f905d4 to your computer and use it in GitHub Desktop.
skeleton.json
{
"Name": "awstest-via-cli",
"Type": "WEBCRAWLER",
"Description": "awstest-via-cli",
"IndexId": "yourindexid",
"Configuration": {
"WebCrawlerConfiguration": {
"Urls": {
"SeedUrlConfiguration": {
"SeedUrls": [
"https://www.someurl.com/"
],
"WebCrawlerMode": "SUBDOMAINS"
}
},
"CrawlDepth": 10,
"MaxContentSizePerPageInMegaBytes": 50,
"MaxLinksPerPage": 100,
"MaxUrlsPerMinuteCrawlRate": 300
}
},
"RoleArn": "somearn",
"Schedule": ""
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment