Skip to content

Instantly share code, notes, and snippets.

@kamaljoshi
Last active May 27, 2017 21:57
Show Gist options
  • Save kamaljoshi/2cce5f6d35cd28de8f6dbb27d586f064 to your computer and use it in GitHub Desktop.
Save kamaljoshi/2cce5f6d35cd28de8f6dbb27d586f064 to your computer and use it in GitHub Desktop.
This script collects domains from your Chrome history for the default profile and checks to see if they are being proxied through Cloudflare by checking presence of a header.
# Except sqlite3 all the libraries are standard that should be present with a Ruby installation.
# If you don't have sqlite3 installed. Use `gem install sqlite3`
require 'fileutils'
require 'sqlite3'
require 'uri'
require 'net/http'
require 'set'
require 'thread'
chrome_history_location = "#{ENV['HOME']}/Library/Application\ Support/Google/Chrome/Default/History"
temp_location = "/tmp/Chrome_history"
FileUtils.cp(chrome_history_location, temp_location)
sqlite_db = SQLite3::Database.new temp_location
chrome_history = sqlite_db.execute('SELECT DISTINCT(url) FROM urls;').flatten; nil
FileUtils.rm([temp_location])
domain_set = Set.new
cloudflare_set = Set.new
query_uris = Array.new
chrome_history.each do |url|
host = URI.parse(url).host rescue nil
query_uris += [URI::HTTPS.build({host: host}), URI::HTTP.build({host: host})] if !domain_set.include?(host) && !host.nil?
domain_set.add(host)
end; nil
uri_mutex, set_mutex, read_mutex = Mutex.new, Mutex.new, Mutex.new
(1..16).map do
Thread.new(query_uris, cloudflare_set) do |query_uris, cloudflare_set|
while !(uri = uri_mutex.synchronize { query_uris.pop }).nil?
cf_header_present = !Net::HTTP.get_response(uri)['cf-ray'].nil? rescue nil
read_mutex.synchronize{ print("#{query_uris.length} remaining\r") }
set_mutex.synchronize { cloudflare_set.add(uri.host) } if cf_header_present
end
end
end.each(&:join); nil
p cloudflare_set.to_a.sort
@skylarmb
Copy link

skylarmb commented Feb 24, 2017

@lbeltrame Happened to me as well with a different URL. I think its the _ in the url. I just searched for the problem domain in my chrome history, deleted it, and then the script ran fine.

@darrentu
Copy link

For those who simply want to check their history or whatever tables are in your chrome history database, I created a small shell script to generate those files.

https://github.com/darrentu/convert-db-to-csv

@bmurithi
Copy link

On Linux, I had to switch the path to this:

chrome_history_location = "#{ENV['HOME']}/.config/google-chrome/Default/History"

See Chrome/Chromium User Data Directories

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment