Skip to content

Instantly share code, notes, and snippets.

Show Gist options
  • Save GitterDoneScott/812f93f158ad10cabe1d8035eab380c8 to your computer and use it in GitHub Desktop.
Save GitterDoneScott/812f93f158ad10cabe1d8035eab380c8 to your computer and use it in GitHub Desktop.
wget scrape of a Confluence Wiki - This isn't perfect becuase it gets some duplicate pages
wget \
--header="User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.72 Safari/537.36" \
--header="Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9" \
--header="Accept-Language: en-us,en;q=0.5" \
--header="Accept-Encoding: gzip, deflate" \
--header="Cookie: JSESSIONID=<Paste your session ID here>" \
--cipher 'DEFAULT:!DH' \
--no-check-certificate \
--page-requisites \
--html-extension \
--convert-links \
--domains <allowed domain> \
--mirror \
-e robots=off \
--reject-regex "version=\
|diffpages\.action\
|display\/~\
|viewpreviousversions\.action\
|viewinfo\.action\
|managepagenotifications\.action\
|listpages-dirview\.action\
|viewpagesrc\.action\
|pdfpageexport\.action\
|exportword\
|uploadimport\.action\
|diffpagesbyversion.action\
|dashboard\.action\
|listpages\.action\
|blogposts\.action\
|createblogpost\.action\
|copypage\.action\
|addfavourite\.action\
|listpagetemplates\.action\
|listlabels-heatmap\.action\
|listattachmentsforspace\.action\
|viewmailarchive\.action\
|viewspacesummary\.action\
|editspace\.action\
|peopledirectory\.action\
|configurerssfeed\.action\
|viewuserhistory\action\
|viewmyprofile\.action\
|display\/status\/.*\/list\
|viewmyfavourites\.action\
|viewnotifications\.action\
|viewmydrafts\.action\
|viewmysettings\.action\
|logout\.action\
|login\.action\
|revertpagebacktoversion\
|confirmattachmentversionremoval\
|editattachment\.action\
|editpage\.action\
|createpage\.action\
|listpages\.action\
|recentlyupdated\.action\
|removepage\.action\
|benryanconversion\
|confirmattachmentremoval\.action\
|editinword\
|editattachment\.action\
|downloadallattachments\.action\
|pagenotification\.action\
|template\.action\
|worddav\
|rssfeed\.action\
|\/feeds\/\
|changes\.action\
|sortBy=\
|showComment\
|spacedirectory\
|homepage\.action\
|viewuserhistory\.action\
|viewfollow\.action\
|opensearch\
|\/label\/\
|\/labels\/\
|showChildren=false\
|dosearchsite\.action\
|browsespace\.action\
|comment\.action" \
"https://<wiki domain>/spacedirectory/view.action"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment