Skip to content

Instantly share code, notes, and snippets.

@pojda
Last active August 20, 2020 14:41
Show Gist options
  • Save pojda/4524db3b9a504b0bb4fef8111122a946 to your computer and use it in GitHub Desktop.
Save pojda/4524db3b9a504b0bb4fef8111122a946 to your computer and use it in GitHub Desktop.
Based on https://www.domwoodman.com/posts/search-console-crawl-data-bookmarklet/, it just adds the GSC property name to the downloaded file name.
javascript:(function(){const crawlGraphs %3D document.querySelectorAll%28".hostload-activity script"%29%3B%0A%0Aconst isSafari %3D %28%29 %3D>%0A%09%2F%5E%28%28%3F%21chrome%7Candroid%29.%29%2Asafari%2Fi.test%28navigator.userAgent%29%3B%0A%0Aconst toCSV %3D %28data%2C headers%2C separator%2C enclosingCharacter%29 %3D> %7B%0A%09return jsons2csv%28data%2C headers%2C separator%2C enclosingCharacter%29%3B%0A%7D%3B%0A%0Aconst isJsons %3D array %3D>%0A%09Array.isArray%28array%29 %26%26%0A%09array.every%28row %3D> typeof row %3D%3D%3D "object" %26%26 %21%28row instanceof Array%29%29%3B%0A%0Aconst jsons2arrays %3D %28jsons%2C headers%29 %3D> %7B%0A%09headers %3D headers %7C%7C jsonsHeaders%28jsons%29%3B%0A%0A%09%2F%2F allow headers to have custom labels%2C defaulting to having the header data key be the label%0A%09let headerLabels %3D headers%3B%0A%09let headerKeys %3D headers%3B%0A%09if %28isJsons%28headers%29%29 %7B%0A%09%09headerLabels %3D headers.map%28header %3D> header.label%29%3B%0A%09%09headerKeys %3D headers.map%28header %3D> header.key%29%3B%0A%09%7D%0A%0A%09console.log%28headerLabels%29%3B%0A%09console.log%28headerKeys%29%3B%0A%0A%09const data %3D jsons.map%28object %3D>%0A%09%09headerKeys.map%28header %3D> getHeaderValue%28header%2C object%29%29%0A%09%29%3B%0A%0A%09return %5BheaderLabels%2C ...data%5D%3B%0A%7D%3B%0A%0Aconst jsonsHeaders %3D array %3D>%0A%09Array.from%28%0A%09%09array%0A%09%09%09.map%28json %3D> Object.keys%28json%29%29%0A%09%09%09.reduce%28%28a%2C b%29 %3D> new Set%28%5B...a%2C ...b%5D%29%2C %5B%5D%29%0A%09%29%3B%0A%0Aconst getHeaderValue %3D %28property%2C obj%29 %3D> %7B%0A%09const foundValue %3D property%0A%09%09.replace%28%2F%5C%5B%28%5B%5E%5C%5D%5D%2B%29%5D%2Fg%2C ".%241"%29%0A%09%09.split%28"."%29%0A%09%09.reduce%28function%28o%2C p%2C i%2C arr%29 %7B%0A%09%09%09%2F%2F if at any point the nested keys passed do not exist%2C splice the array so it doesnt keep reducing%0A%09%09%09if %28o%5Bp%5D %3D%3D%3D undefined%29 %7B%0A%09%09%09%09arr.splice%281%29%3B%0A%09%09%09%7D else %7B%0A%09%09%09%09return o%5Bp%5D%3B%0A%09%09%09%7D%0A%09%09%7D%2C obj%29%3B%0A%0A%09return foundValue %3D%3D%3D undefined %3F "" %3A foundValue%3B%0A%7D%3B%0A%0Aconst jsons2csv %3D %28data%2C headers%2C separator%2C enclosingCharacter%29 %3D>%0A%09joiner%28jsons2arrays%28data%2C headers%29%2C separator%2C enclosingCharacter%29%3B%0A%0Aconst joiner %3D %28data%2C separator %3D "%2C"%2C enclosingCharacter %3D %27"%27%29 %3D> %7B%0A%09const output %3D data%0A%09%09.filter%28e %3D> e%29%0A%09%09.map%28row %3D>%0A%09%09%09row%0A%09%09%09%09.map%28element %3D> elementOrEmpty%28element%29%29%0A%09%09%09%09.map%28%0A%09%09%09%09%09column %3D>%0A%09%09%09%09%09%09%60%24%7BenclosingCharacter%7D%24%7Bcolumn%7D%24%7BenclosingCharacter%7D%60%0A%09%09%09%09%29%0A%09%09%09%09.join%28separator%29%0A%09%09%29%0A%09%09.join%28%60%5Cn%60%29%3B%0A%0A%09return output%3B%0A%7D%3B%0A%0Aconst elementOrEmpty %3D element %3D> %28element %7C%7C element %3D%3D%3D 0 %3F element %3A ""%29%3B%0A%0Aconst buildURI %3D %28data%2C uFEFF%2C headers%2C separator%2C enclosingCharacter%29 %3D> %7B%0A%09const csv %3D toCSV%28data%2C headers%2C separator%2C enclosingCharacter%29%3B%0A%09const type %3D isSafari%28%29 %3F "application%2Fcsv" %3A "text%2Fcsv"%3B%0A%09const blob %3D new Blob%28%5BuFEFF %3F "%5CuFEFF" %3A ""%2C csv%5D%2C %7B type %7D%29%3B%0A%09const dataURI %3D %60data%3A%24%7Btype%7D%3Bcharset%3Dutf-8%2C%24%7BuFEFF %3F "%5CuFEFF" %3A ""%7D%24%7Bcsv%7D%60%3B%0A%0A%09return dataURI%3B%0A%7D%3B%0A%0A%2F%2F This texts the script tag from search console%0Afunction extractJSONDataFromGraphs%28chartData%29 %7B%0A%09const extractJSON %3D new RegExp%28%2FsetData%5C%28%28%5B%5E%3B%5D%2B%29%5C%29%3B%2F%2C "i"%29%3B%0A%0A%09%2F%2F Extract the JSON from the string%0A%09const jsonStr %3D chartData.match%28extractJSON%29%5B1%5D%3B%0A%09%2F%2F const jsonStr %3D regexObj%5B1%5D%3B%0A%0A%09%2F%2F There are a bunch of raw dates in the output. Remove them.%0A%09const jsonStrSanitize %3D jsonStr.replace%28%0A%09%09%2Fnew Date%5C%28%5Cd%2B%2C%5Cd%2B%2C%5Cd%2B%5C%29%2Fgi%2C%0A%09%09%27"dateObj"%27%0A%09%29%3B%0A%0A%09%2F%2F Sanitize the JSON%0A%09const graphJSON %3D JSON.parse%28jsonStrSanitize%29%3B%0A%0A%09const headers %3D graphJSON.cols.map%28colData %3D> %7B%0A%09%09let colKey %3D ""%3B%0A%0A%09%09%2F%2F Map for CSV output%0A%09%09if %28colData.label %3D%3D%3D "Date"%29 %7B%0A%09%09%09colKey %3D "date"%3B%0A%09%09%7D else %7B%0A%09%09%09colKey %3D "data"%3B%0A%09%09%7D%0A%0A%09%09return %7B key%3A colKey%2C label%3A colData.label %7D%3B%0A%09%7D%29%3B%0A%0A%09const data %3D graphJSON.rows.map%28rowData %3D> %7B%0A%09%09const date %3D rowData.c%5B0%5D.f%3B%0A%09%09const number %3D rowData.c%5B1%5D.v%3B%0A%0A%09%09return %7B date%3A date%2C data%3A number %7D%3B%0A%09%7D%29%3B%0A%0A%09return %7B%0A%09%09headers%2C%0A%09%09data%0A%09%7D%3B%0A%7D%0A%0Aconst gscProperty%3D%28%29%3D>%7B%0A%09gsc_property%3Ddocument.title.replace%28%2F%5B%3A.%2F%5D%2Fgi%2C"_"%29.split%28" "%29%3B%0A%09return gsc_property%5Bgsc_property.length-1%5D%3B%0A%7D%0A%0Agraphs%3D%5BgscProperty%28%29%2B"pages_crawled_per_day.csv"%2CgscProperty%28%29%2B"kb_downloaded_per_day.csv"%2CgscProperty%28%29%2B"time_spent_downloading_page.csv"%5D%3B%0A%0A%0A%2F%2F Loop over the graphs%0AcrawlGraphs.forEach%28%28chart%2Ci%29 %3D> %7B%0A%09output %3D extractJSONDataFromGraphs%28chart.text%29%3B%0A%09dataURI %3D buildURI%28output.data%2C "uFEFF"%2C output.headers%2C "%2C"%2C %27"%27%29%3B%0A%0A%09var link %3D document.createElement%28"a"%29%3B%0A%09link.setAttribute%28"href"%2C dataURI%29%3B%0A%09link.setAttribute%28"download"%2C graphs%5Bi%5D%29%3B%0A%09document.body.appendChild%28link%29%3B %2F%2F Required for FF%0A%0A%09link.click%28%29%3B%0A%7D%29%3B})();
@pojda
Copy link
Author

pojda commented Oct 29, 2019

For some encoding problem the downloaded csv file as coming empty. I just fixed it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment