Instantly share code, notes, and snippets.

Embed
What would you like to do?
This script collects domains from your Chrome history for the default profile and checks to see if they are being proxied through Cloudflare by checking presence of a header.
# Except sqlite3 all the libraries are standard that should be present with a Ruby installation.
# If you don't have sqlite3 installed. Use `gem install sqlite3`
require 'fileutils'
require 'sqlite3'
require 'uri'
require 'net/http'
require 'set'
require 'thread'
chrome_history_location = "#{ENV['HOME']}/Library/Application\ Support/Google/Chrome/Default/History"
temp_location = "/tmp/Chrome_history"
FileUtils.cp(chrome_history_location, temp_location)
sqlite_db = SQLite3::Database.new temp_location
chrome_history = sqlite_db.execute('SELECT DISTINCT(url) FROM urls;').flatten; nil
FileUtils.rm([temp_location])
domain_set = Set.new
cloudflare_set = Set.new
query_uris = Array.new
chrome_history.each do |url|
host = URI.parse(url).host rescue nil
query_uris += [URI::HTTPS.build({host: host}), URI::HTTP.build({host: host})] if !domain_set.include?(host) && !host.nil?
domain_set.add(host)
end; nil
uri_mutex, set_mutex, read_mutex = Mutex.new, Mutex.new, Mutex.new
(1..16).map do
Thread.new(query_uris, cloudflare_set) do |query_uris, cloudflare_set|
while !(uri = uri_mutex.synchronize { query_uris.pop }).nil?
cf_header_present = !Net::HTTP.get_response(uri)['cf-ray'].nil? rescue nil
read_mutex.synchronize{ print("#{query_uris.length} remaining\r") }
set_mutex.synchronize { cloudflare_set.add(uri.host) } if cf_header_present
end
end
end.each(&:join); nil
p cloudflare_set.to_a.sort
@vrischmann

This comment has been minimized.

vrischmann commented Feb 24, 2017

Not sure if it's due to my Ruby version (2.4.0p0) but I had to replace the last line with

cloudflare_set.each do |el|
    puts el
end

otherwise it would print something like #<Set:0x007fd3c482ec50>

@kamaljoshi

This comment has been minimized.

Owner

kamaljoshi commented Feb 24, 2017

@vrischmann I changed p to puts when I copied the script from local to gist. Will change it. Thanks!

@johnlinvc

This comment has been minimized.

johnlinvc commented Feb 24, 2017

if you got cannot load such file -- sqlite3 (LoadError)
you'll need to gem install sqlite3

@lpetre

This comment has been minimized.

lpetre commented Feb 24, 2017

I've added multiple-profile support here: https://gist.github.com/lpetre/77a33d295671409382f8b0608420eca0

@jcfrei

This comment has been minimized.

jcfrei commented Feb 24, 2017

chrome_history_location on ubuntu:

chrome_history_location = "#{ENV['HOME']}/.config/google-chrome/Default/History"
@llimllib

This comment has been minimized.

llimllib commented Feb 24, 2017

ruby seems to have a bug with hostnames with an underscore in them; I had to add url.gsub! /_/, '%5F' before line 19 to make this run

@bxkx

This comment has been minimized.

bxkx commented Feb 24, 2017

Can you please make it so that it writes all the domains in to a text file? The windows cmd doesn't let me scroll all the way up so I can't see all the domains.

@Quinny

This comment has been minimized.

Quinny commented Feb 24, 2017

@bxkx Try using output redirection

ruby script > file

@bxkx

This comment has been minimized.

bxkx commented Feb 24, 2017

That did it, thanks!

@carchrae

This comment has been minimized.

carchrae commented Feb 24, 2017

nice work. i wonder if there is any way to look at the cached pages and see if the page had been re-written by cloudflare - i somehow doubt it, but would be nice to know, as i believe only those sites are at risk.

@bxkx - you can increase the cmd buffer or pipe to a file, eg, ruby script.rb > output.txt

@lbeltrame

This comment has been minimized.

lbeltrame commented Feb 24, 2017

It doesn't seem to work if there's an underscore in the host:

/usr/lib64/ruby/2.2.0/uri/generic.rb:593:in `check_host': bad component(expected host component): fate_extella.wicurio.com (URI::InvalidComponentError)
        from /usr/lib64/ruby/2.2.0/uri/generic.rb:634:in `host='
        from /usr/lib64/ruby/2.2.0/uri/generic.rb:668:in `hostname='
        from /usr/lib64/ruby/2.2.0/uri/generic.rb:187:in `initialize'
        from /usr/lib64/ruby/2.2.0/uri/generic.rb:134:in `new'
        from /usr/lib64/ruby/2.2.0/uri/generic.rb:134:in `build'
        from /usr/lib64/ruby/2.2.0/uri/http.rb:62:in `build'
        from tmp.rb:26:in `block in <main>'
        from tmp.rb:24:in `each'
        from tmp.rb:24:in `<main>'

(Ruby 2.2)

@kamaljoshi

This comment has been minimized.

@skylarmb

This comment has been minimized.

skylarmb commented Feb 24, 2017

@lbeltrame Happened to me as well with a different URL. I think its the _ in the url. I just searched for the problem domain in my chrome history, deleted it, and then the script ran fine.

@darrentu

This comment has been minimized.

darrentu commented Feb 25, 2017

For those who simply want to check their history or whatever tables are in your chrome history database, I created a small shell script to generate those files.

https://github.com/darrentu/convert-db-to-csv

@bmurithi

This comment has been minimized.

bmurithi commented Feb 25, 2017

On Linux, I had to switch the path to this:

chrome_history_location = "#{ENV['HOME']}/.config/google-chrome/Default/History"

See Chrome/Chromium User Data Directories

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment