Create a gist now

Instantly share code, notes, and snippets.

What would you like to do?
A Ruby script for collecting phone record statistics from a Facebook user data dump
#! /usr/bin/env ruby
# NOTE: Requires Ruby 2.1 or greater.
# This script can be used to parse and dump the information from
# the 'html/contact_info.htm' file in a Facebook user data ZIP download.
#
# It prints all cell phone call + SMS message + MMS records, plus a summary of each.
#
# It also dumps all of the records into CSV files inside a 'CSV' folder, that is created
# in whatever the working directory of the program is when executed.
#
# Place this script inside the extracted Facebook data download folder
# alongside the 'html' folder.
#
# This script requires Ruby and the Nokogiri library to be installed.
#
# Open source licensing
# ---------------------
#
# Dual-licensed under the MIT and Apache 2.0 open source licenses. Either license can be chosen
# by any user of the program.
#
# The MIT license is duplicated here, the Apache 2.0 license can be found here
# https://opensource.org/licenses/Apache-2.0
#
# The MIT License (MIT)
# Copyright (c) 2018 Dylan McKay
#
# Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated
# documentation files (the "Software"), to deal in the Software without restriction, including without limitation
# the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software,
# and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE
# WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS
# OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
# OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
require 'nokogiri'
require 'time'
require 'fileutils'
def hr
$stdout.puts "-" * 24
end
def indent(level = 1)
$stdout.print " " * (level - 1)
$stdout.flush
end
def section(title, level: 1)
indent(level) and hr
indent(level) and $stdout.puts title
indent(level) and $stdout.puts
yield
indent(level) and hr
indent(level) and $stdout.puts
end
# Extracts metadata from a call/text/sms/mms table
# Returns nil if there is no metadata in this table.
# Returns a 2d list of row/colums
def extract_table_metadata(metadata_table)
headings = metadata_table.css('tr').first.css('th').map(&:text).map(&:chomp)
records = metadata_table.css('tr')[1..-1]
return nil if records.size <= 1 # many tables are empty (excluding headings).
[headings] + records.map do |call_record|
call_record.css('td').map(&:text).map(&:chomp).map do |field|
if field.include? ' at ' # some fields are dates/times
# Time example: "Wednesday, 14 June 2017 at 19:02 UTC+12"
Time.strptime(field, "%A, %e %B %Y at %R UTC%z") rescue field
else
field # no special processing
end
end
end
end
def dig_out_metadata(container:)
# If a specific type of metadata is missing (calls, texts, ..), the
# container div will simply not be present.
return [] if container.nil?
contact_tables = container.children.select { |c| c.name == "table" }
contact_tables.map do |contact_table|
metadata_table = contact_table.css('table')[0]
extract_table_metadata(metadata_table)
end.compact.select { |t| t.size > 1 } # must include non-header rows
end
def print_metadata(metadata, metadata_title:)
section(metadata_title) do
metadata.each do |phone_records|
puts
indent(2) and puts "Another phone number"
puts
phone_records.each do |record|
indent(2) and puts record.join(", ")
end
end
end
end
def print_timestamps(metadata, metadata_name:)
timestamps = metadata.map { |r| r[1].to_s.chomp }.select { |s| s.size > 0 }.map do |t|
begin
Time.parse(t)
rescue ArgumentError # do not parse timestamp if unparseable
t
end
end
if timestamps.size > 0
puts "The oldest #{metadata_name} is from #{timestamps.min.to_date}, the most recent at #{timestamps.max.to_date}"
end
end
def print_status_breakdown(metadata, metadata_name:)
grouped_statuses = metadata.flatten(1).group_by(&:first)
if grouped_statuses.size > 0
puts "This includes " + grouped_statuses.map { |status,records| "#{records.size} #{status.downcase} #{metadata_name}"}.join(", ")
end
end
def metadata_to_csv(metadata)
metadata.flatten(1).map { |record| record.join(',') }.join("\n")
end
def dump_metadata_csv(html_doc)
call_history_container = html_doc.xpath("//h2[text()='Call History']/following-sibling::div")[0]
sms_history_container = html_doc.xpath("//h2[text()='SMS History']/following-sibling::div")[0]
mms_history_container = html_doc.xpath("//h2[text()='MMS History']/following-sibling::div")[0]
FileUtils.mkdir_p("csv")
call_metadata = dig_out_metadata(:container => call_history_container)
sms_metadata = dig_out_metadata(:container => sms_history_container)
mms_metadata = dig_out_metadata(:container => mms_history_container)
File.write(File.join("csv", "call.csv"), metadata_to_csv(call_metadata))
File.write(File.join("csv", "sms.csv"), metadata_to_csv(sms_metadata))
File.write(File.join("csv", "mms.csv"), metadata_to_csv(mms_metadata))
end
def print_metadata_human(html_doc)
call_history_container = html_doc.xpath("//h2[text()='Call History']/following-sibling::div")[0]
sms_history_container = html_doc.xpath("//h2[text()='SMS History']/following-sibling::div")[0]
mms_history_container = html_doc.xpath("//h2[text()='MMS History']/following-sibling::div")[0]
call_metadata = dig_out_metadata(:container => call_history_container)
sms_metadata = dig_out_metadata(:container => sms_history_container)
mms_metadata = dig_out_metadata(:container => mms_history_container)
if call_history_container
phone_numbers = call_history_container.xpath("//b[text()='Number:']/following-sibling::text()")
.map(&:text).sort.uniq
else
phone_numbers = []
end
print_metadata(call_metadata, :metadata_title => "Call History")
print_metadata(sms_metadata, :metadata_title => "SMS History")
print_metadata(mms_metadata, :metadata_title => "MMS History")
section("The full list of phone numbers that have stored data") do
phone_numbers.each_slice(8).to_a.map { |g| g.join(", ") }.each do |line|
indent(2) and $stdout.puts line
end
end
$stdout.puts "A brief summary of phone records"
hr
$stdout.puts "There are phone records for #{phone_numbers.size} distinct phone numbers"
$stdout.puts "There are records of #{call_metadata.flatten(1).size} distinct cell phone calls"
indent(2) and print_timestamps(call_metadata, :metadata_name => "cell phone call")
indent(2) and print_status_breakdown(call_metadata, :metadata_name => "cell phone calls")
$stdout.puts "There are records of #{sms_metadata.flatten(1).size} distinct SMS messages"
indent(2) and print_timestamps(sms_metadata, :metadata_name => "SMS message")
indent(2) and print_status_breakdown(sms_metadata, :metadata_name => "SMS messages")
$stdout.puts "There are records of #{mms_metadata.flatten(1).size} distinct MMS messages"
indent(2) and print_timestamps(mms_metadata, :metadata_name => "MMS message")
indent(2) and print_status_breakdown(mms_metadata, :metadata_name => "MMS messages")
hr
end
html_text = File.read('html/contact_info.htm')
html_doc = Nokogiri::HTML(html_text)
print_metadata_human(html_doc)
$stdout.puts
hr
$stdout.puts "dumped metadata to CSV files at #{Dir.pwd}/csv"
dump_metadata_csv(html_doc)
@dylanmckay

This comment has been minimized.

Show comment Hide comment
@dylanmckay

dylanmckay Mar 22, 2018

screen shot 2018-03-23 at 5 29 48 am

Owner

dylanmckay commented Mar 22, 2018

screen shot 2018-03-23 at 5 29 48 am

@arudyk

This comment has been minimized.

Show comment Hide comment
@arudyk

arudyk Mar 22, 2018

./facebook-contact-info-summary.rb:48:in dig_out_metadata': undefined methodchildren' for nil:NilClass (NoMethodError)
from ./facebook-contact-info-summary.rb:84:in print_call_history' from ./facebook-contact-info-summary.rb:120:in

'

Did they remove those elements from the dump?

arudyk commented Mar 22, 2018

./facebook-contact-info-summary.rb:48:in dig_out_metadata': undefined methodchildren' for nil:NilClass (NoMethodError)
from ./facebook-contact-info-summary.rb:84:in print_call_history' from ./facebook-contact-info-summary.rb:120:in

'

Did they remove those elements from the dump?

@dylanmckay

This comment has been minimized.

Show comment Hide comment
@dylanmckay

dylanmckay Mar 22, 2018

It looks that way, somebody on Twitter had the same issue when they had no call history

I've fixed the script up now so it should work fine even if there are no calls, texts, or MMSes.

Owner

dylanmckay commented Mar 22, 2018

It looks that way, somebody on Twitter had the same issue when they had no call history

I've fixed the script up now so it should work fine even if there are no calls, texts, or MMSes.

@dylanmckay

This comment has been minimized.

Show comment Hide comment
@dylanmckay

dylanmckay Mar 23, 2018

I've adjusted the script so that it also dumps CSV files of each metadata to disk as well

Owner

dylanmckay commented Mar 23, 2018

I've adjusted the script so that it also dumps CSV files of each metadata to disk as well

@sjanahan

This comment has been minimized.

Show comment Hide comment
@sjanahan

sjanahan Mar 23, 2018

I guess I didn't have FB or messenger on my phone in the offending time frame

OR

These elements have been removed from the dump

A brief summary of phone records
------------------------
There are phone records for 0 distinct phone numbers
There are records of 0 distinct cell phone calls
      There are records of 0 distinct SMS messages
      There are records of 0 distinct MMS messages

I ran it in the base directory of the download as detailed

sjanahan commented Mar 23, 2018

I guess I didn't have FB or messenger on my phone in the offending time frame

OR

These elements have been removed from the dump

A brief summary of phone records
------------------------
There are phone records for 0 distinct phone numbers
There are records of 0 distinct cell phone calls
      There are records of 0 distinct SMS messages
      There are records of 0 distinct MMS messages

I ran it in the base directory of the download as detailed

@adiack

This comment has been minimized.

Show comment Hide comment
@adiack

adiack Mar 23, 2018

Same here and I have been a facebook user for a longtime.

------------------------
Call History

------------------------

------------------------
SMS History

------------------------

------------------------
MMS History

------------------------

------------------------
The full list of phone numbers that have stored data


------------------------

A brief summary of phone records
------------------------
There are phone records for 0 distinct phone numbers
There are records of 0 distinct cell phone calls
      There are records of 0 distinct SMS messages
      There are records of 0 distinct MMS messages
      -----------------------


adiack commented Mar 23, 2018

Same here and I have been a facebook user for a longtime.

------------------------
Call History

------------------------

------------------------
SMS History

------------------------

------------------------
MMS History

------------------------

------------------------
The full list of phone numbers that have stored data


------------------------

A brief summary of phone records
------------------------
There are phone records for 0 distinct phone numbers
There are records of 0 distinct cell phone calls
      There are records of 0 distinct SMS messages
      There are records of 0 distinct MMS messages
      -----------------------


@diejmon

This comment has been minimized.

Show comment Hide comment
@diejmon

diejmon Mar 23, 2018

Did you use Facebook on Android?

diejmon commented Mar 23, 2018

Did you use Facebook on Android?

@KramKroc

This comment has been minimized.

Show comment Hide comment
@KramKroc

KramKroc Mar 23, 2018

Using ruby 2.5.0 and getting an error with the Time module:

Traceback (most recent call last):
	13: from facebook-contact-info-summary.rb:161:in `<main>'
	12: from facebook-contact-info-summary.rb:121:in `print_metadata_human'
	11: from facebook-contact-info-summary.rb:64:in `dig_out_metadata'
	10: from facebook-contact-info-summary.rb:64:in `map'
	 9: from facebook-contact-info-summary.rb:66:in `block in dig_out_metadata'
	 8: from facebook-contact-info-summary.rb:45:in `extract_table_metadata'
	 7: from facebook-contact-info-summary.rb:45:in `map'
	 6: from /usr/local/lib/ruby/gems/2.5.0/gems/nokogiri-1.8.2/lib/nokogiri/xml/node_set.rb:189:in `each'
	 5: from /usr/local/lib/ruby/gems/2.5.0/gems/nokogiri-1.8.2/lib/nokogiri/xml/node_set.rb:189:in `upto'
	 4: from /usr/local/lib/ruby/gems/2.5.0/gems/nokogiri-1.8.2/lib/nokogiri/xml/node_set.rb:190:in `block in each'
	 3: from facebook-contact-info-summary.rb:46:in `block in extract_table_metadata'
	 2: from facebook-contact-info-summary.rb:46:in `map'
	 1: from facebook-contact-info-summary.rb:49:in `block (2 levels) in extract_table_metadata'
/usr/local/Cellar/ruby/2.5.0_2/lib/ruby/2.5.0/time.rb:433:in `strptime': invalid strptime format - `%A, %e %B %Y at %R UTC%z' (ArgumentError)

Using ruby 2.5.0 and getting an error with the Time module:

Traceback (most recent call last):
	13: from facebook-contact-info-summary.rb:161:in `<main>'
	12: from facebook-contact-info-summary.rb:121:in `print_metadata_human'
	11: from facebook-contact-info-summary.rb:64:in `dig_out_metadata'
	10: from facebook-contact-info-summary.rb:64:in `map'
	 9: from facebook-contact-info-summary.rb:66:in `block in dig_out_metadata'
	 8: from facebook-contact-info-summary.rb:45:in `extract_table_metadata'
	 7: from facebook-contact-info-summary.rb:45:in `map'
	 6: from /usr/local/lib/ruby/gems/2.5.0/gems/nokogiri-1.8.2/lib/nokogiri/xml/node_set.rb:189:in `each'
	 5: from /usr/local/lib/ruby/gems/2.5.0/gems/nokogiri-1.8.2/lib/nokogiri/xml/node_set.rb:189:in `upto'
	 4: from /usr/local/lib/ruby/gems/2.5.0/gems/nokogiri-1.8.2/lib/nokogiri/xml/node_set.rb:190:in `block in each'
	 3: from facebook-contact-info-summary.rb:46:in `block in extract_table_metadata'
	 2: from facebook-contact-info-summary.rb:46:in `map'
	 1: from facebook-contact-info-summary.rb:49:in `block (2 levels) in extract_table_metadata'
/usr/local/Cellar/ruby/2.5.0_2/lib/ruby/2.5.0/time.rb:433:in `strptime': invalid strptime format - `%A, %e %B %Y at %R UTC%z' (ArgumentError)
@zeagord

This comment has been minimized.

Show comment Hide comment
@zeagord

zeagord Mar 23, 2018

You have modify the Time.strptime function. Time.strptime(field, "%A, %B %e, %Y at %R%p UTC%z")

Follow the reference here if you need more details.

zeagord commented Mar 23, 2018

You have modify the Time.strptime function. Time.strptime(field, "%A, %B %e, %Y at %R%p UTC%z")

Follow the reference here if you need more details.

@dylanmckay

This comment has been minimized.

Show comment Hide comment
@dylanmckay

dylanmckay Mar 23, 2018

I've pushed a new version of the script that will ignore time parse errors, and use the original timestamp instead.

@KramKroc If you can copy-paste a timestamp or two from html/contact_info.htm, I can check it on my machine and get it parsing.

Owner

dylanmckay commented Mar 23, 2018

I've pushed a new version of the script that will ignore time parse errors, and use the original timestamp instead.

@KramKroc If you can copy-paste a timestamp or two from html/contact_info.htm, I can check it on my machine and get it parsing.

@moimikey

This comment has been minimized.

Show comment Hide comment
@moimikey

moimikey Mar 23, 2018

how bout posting a dataset that works :)?

how bout posting a dataset that works :)?

@sirlebert

This comment has been minimized.

Show comment Hide comment
@sirlebert

sirlebert Mar 23, 2018

thanks for all the job done

when I tried to run the script I've got this outcome: "facebook-contact-info-summary.rb:29: syntax error, unexpected tLABEL
def section(title, level: 1)" I might be doing something wrong?

thanks for all the job done

when I tried to run the script I've got this outcome: "facebook-contact-info-summary.rb:29: syntax error, unexpected tLABEL
def section(title, level: 1)" I might be doing something wrong?

@bendoane

This comment has been minimized.

Show comment Hide comment
@bendoane

bendoane Mar 23, 2018

@dylanmckay,

  1. THANK YOU for noticing/calling this out.
  2. Because of the political ramifications & recent revelations of data use, I have to ask - are you in any way tied to work in US?

The script ran fine for me using ruby 2.3.3 (happened to have that version loaded at the time), and, luckily, I am showing none of that information was stored. Though, I am in Indiana - a very red state.

And - friendly reminder to anyone not-so-tech or new-to-tech, ASK QUESTIONS if you have trouble running this and you would like to do so. We were all new once, and you should never be afraid to ask/learn.

*I've edited this post as I've found answers to my own questions. sorry

bendoane commented Mar 23, 2018

@dylanmckay,

  1. THANK YOU for noticing/calling this out.
  2. Because of the political ramifications & recent revelations of data use, I have to ask - are you in any way tied to work in US?

The script ran fine for me using ruby 2.3.3 (happened to have that version loaded at the time), and, luckily, I am showing none of that information was stored. Though, I am in Indiana - a very red state.

And - friendly reminder to anyone not-so-tech or new-to-tech, ASK QUESTIONS if you have trouble running this and you would like to do so. We were all new once, and you should never be afraid to ask/learn.

*I've edited this post as I've found answers to my own questions. sorry

@dylanmckay

This comment has been minimized.

Show comment Hide comment
@dylanmckay

dylanmckay Mar 23, 2018

@sirlebert

when I tried to run the script I've got this outcome: "facebook-contact-info-summary.rb:29: syntax error, unexpected tLABEL
def section(title, level: 1)" I might be doing something wrong?

Sounds like there's a missing end somewhere. I just redownloaded the gist and it ran fine - did you miss a line in the copy paste? The "Raw" button next to the file is the easiest way to download it.

https://stackoverflow.com/questions/32279232/ruby-syntax-error-unexpected-tlabel

Owner

dylanmckay commented Mar 23, 2018

@sirlebert

when I tried to run the script I've got this outcome: "facebook-contact-info-summary.rb:29: syntax error, unexpected tLABEL
def section(title, level: 1)" I might be doing something wrong?

Sounds like there's a missing end somewhere. I just redownloaded the gist and it ran fine - did you miss a line in the copy paste? The "Raw" button next to the file is the easiest way to download it.

https://stackoverflow.com/questions/32279232/ruby-syntax-error-unexpected-tlabel

@derekslenk

This comment has been minimized.

Show comment Hide comment
@derekslenk

derekslenk Mar 23, 2018

I get this, but manually looking in contact_info it does have a bunch of phone numbers attached.


Call History



SMS History



MMS History



The full list of phone numbers that have stored data


A brief summary of phone records

There are phone records for 0 distinct phone numbers
There are records of 0 distinct cell phone calls
There are records of 0 distinct SMS messages
There are records of 0 distinct MMS messages
------------------------


I get this, but manually looking in contact_info it does have a bunch of phone numbers attached.


Call History



SMS History



MMS History



The full list of phone numbers that have stored data


A brief summary of phone records

There are phone records for 0 distinct phone numbers
There are records of 0 distinct cell phone calls
There are records of 0 distinct SMS messages
There are records of 0 distinct MMS messages
------------------------


@SunDevil71

This comment has been minimized.

Show comment Hide comment
@SunDevil71

SunDevil71 Mar 23, 2018

Hi - I'm a ruby newb, but technically capable - I'm running on a Mac, I've loaded Nokogiri via gems, but this error persists. THis is my last one where I tried elevated creds. Any suggestion
s - thanks in advance - this is not only a great but a NECESSARY project. THanks much for the script Dylan-

When I try to run this i get:

sudo ruby Facebook_Info.rb
Password:
/Users/XXXXXXXXXX/.rvm/rubies/ruby-2.3.0-heard/lib/ruby/site_ruby/2.3.0/rubygems/core_ext/kernel_require.rb:59:in require': cannot load such file -- nokogiri (LoadError) from /Users/XXXXXXXXX/.rvm/rubies/ruby-2.3.0-heard/lib/ruby/site_ruby/2.3.0/rubygems/core_ext/kernel_require.rb:59:inrequire'
from Facebook_Info.rb:16:in `

'

Hi - I'm a ruby newb, but technically capable - I'm running on a Mac, I've loaded Nokogiri via gems, but this error persists. THis is my last one where I tried elevated creds. Any suggestion
s - thanks in advance - this is not only a great but a NECESSARY project. THanks much for the script Dylan-

When I try to run this i get:

sudo ruby Facebook_Info.rb
Password:
/Users/XXXXXXXXXX/.rvm/rubies/ruby-2.3.0-heard/lib/ruby/site_ruby/2.3.0/rubygems/core_ext/kernel_require.rb:59:in require': cannot load such file -- nokogiri (LoadError) from /Users/XXXXXXXXX/.rvm/rubies/ruby-2.3.0-heard/lib/ruby/site_ruby/2.3.0/rubygems/core_ext/kernel_require.rb:59:inrequire'
from Facebook_Info.rb:16:in `

'

@dill-does

This comment has been minimized.

Show comment Hide comment
@dill-does

dill-does Mar 23, 2018

Hi Dylan,

I saw your thread on twitter and wound up here. I've managed to download my Facebook data as zip, but I'm not familiar whatsoever with coding or Ruby or Nokogiri... I want to check call history as you've done for yourself... can you (or someone else) please provide instructions for how to go about accessing the contact_info.htm file? I've downloaded ruby, but that's as far as I've gotten.

Thanks !

Grateful and clueless,
Dillon

dill-does commented Mar 23, 2018

Hi Dylan,

I saw your thread on twitter and wound up here. I've managed to download my Facebook data as zip, but I'm not familiar whatsoever with coding or Ruby or Nokogiri... I want to check call history as you've done for yourself... can you (or someone else) please provide instructions for how to go about accessing the contact_info.htm file? I've downloaded ruby, but that's as far as I've gotten.

Thanks !

Grateful and clueless,
Dillon

@skdcbrown

This comment has been minimized.

Show comment Hide comment
@skdcbrown

skdcbrown Mar 23, 2018

Having the same issue on Windows after installing the recommended version of Ruby.
C:/Ruby24-x64/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in require': cannot load such file -- nokogiri (LoadError) from C:/Ruby24-x64/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:inrequire'
from facebook-contact-info-summary.rb:16:in `

'

Having the same issue on Windows after installing the recommended version of Ruby.
C:/Ruby24-x64/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in require': cannot load such file -- nokogiri (LoadError) from C:/Ruby24-x64/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:inrequire'
from facebook-contact-info-summary.rb:16:in `

'

@afturner

This comment has been minimized.

Show comment Hide comment
@afturner

afturner Mar 23, 2018

Hey there @dill-does! In order to access the contact_info.htm file, you shouldn't need Ruby or this code. Dylan merely provided a script to quickly parse out the info. All you need to do is unzip what you downloaded and go to the html folder.

Hey there @dill-does! In order to access the contact_info.htm file, you shouldn't need Ruby or this code. Dylan merely provided a script to quickly parse out the info. All you need to do is unzip what you downloaded and go to the html folder.

@drygoatair

This comment has been minimized.

Show comment Hide comment
@drygoatair

drygoatair Mar 23, 2018

No call/sms/mms history for me either...

Edit: come to think of it, I deleted/disabled the facebook app al long time ago. And even deleted my fb account (got reactivated now because I wanted to download the zip file -_-).

drygoatair commented Mar 23, 2018

No call/sms/mms history for me either...

Edit: come to think of it, I deleted/disabled the facebook app al long time ago. And even deleted my fb account (got reactivated now because I wanted to download the zip file -_-).

@darelover

This comment has been minimized.

Show comment Hide comment
@darelover

darelover Mar 23, 2018

@dill-does paste the script in the unzipped folder and then open the terminal and run the command ruby facebook-contact-info-summary.rb

@dill-does paste the script in the unzipped folder and then open the terminal and run the command ruby facebook-contact-info-summary.rb

@jrmgx

This comment has been minimized.

Show comment Hide comment
@jrmgx

jrmgx Mar 23, 2018

Nothing for me, I use the android app + messenger app, I'm almost confused.... Did they do something on their end?

jrmgx commented Mar 23, 2018

Nothing for me, I use the android app + messenger app, I'm almost confused.... Did they do something on their end?

@Otisnfur

This comment has been minimized.

Show comment Hide comment
@Otisnfur

Otisnfur Mar 23, 2018

@darelover Also a Ruby newbie, when I attempt to do ruby facebook-contact-info-summary.rb I get "ruby: No such file or directory -- facebook-contact-info-summary.rb (LoadError)"

I have downloaded my Facebook Backup and placed the .rb file in the same folder as the index.html
I have also downloaded and installed ruby, and I believe I've downloaded nokogiri

@darelover Also a Ruby newbie, when I attempt to do ruby facebook-contact-info-summary.rb I get "ruby: No such file or directory -- facebook-contact-info-summary.rb (LoadError)"

I have downloaded my Facebook Backup and placed the .rb file in the same folder as the index.html
I have also downloaded and installed ruby, and I believe I've downloaded nokogiri

@zipibz

This comment has been minimized.

Show comment Hide comment
@zipibz

zipibz Mar 23, 2018

Probably shouldn't be shocked cause I gave them these permissions. None the less, this is crazy!
image

zipibz commented Mar 23, 2018

Probably shouldn't be shocked cause I gave them these permissions. None the less, this is crazy!
image

@abhidesikan

This comment has been minimized.

Show comment Hide comment
@abhidesikan

abhidesikan Mar 23, 2018

I also get empty records when I run this script. I placed it in the same folder as the html.

I also get empty records when I run this script. I placed it in the same folder as the html.

@mvilain

This comment has been minimized.

Show comment Hide comment
@mvilain

mvilain Mar 24, 2018

I ran this on MacOS 10.10.5 and it didn't even pass syntax checking. I suspect it requires a more current version of ruby than what's installed natively on my system.

# ./facebook-contact-info-summary.rb 
./facebook-contact-info-summary.rb:59: syntax error, unexpected ')'
./facebook-contact-info-summary.rb:70: syntax error, unexpected keyword_end, expecting end-of-input
# ruby --version
ruby 2.0.0p481 (2014-05-08 revision 45883) [universal.x86_64-darwin14]

I wandered a bunch of sites on updating my system to ruby 2.5. I don't use homebrew and prefer MacPort. I used rbenv to upgrade to ruby (rvm had certificate issues which rbenv fixed). After the dust settled, I ran the script and it didn't find any phone numbers, which is what I expected.

Thanks for writing this. And making me research stuff into ruby.

mvilain commented Mar 24, 2018

I ran this on MacOS 10.10.5 and it didn't even pass syntax checking. I suspect it requires a more current version of ruby than what's installed natively on my system.

# ./facebook-contact-info-summary.rb 
./facebook-contact-info-summary.rb:59: syntax error, unexpected ')'
./facebook-contact-info-summary.rb:70: syntax error, unexpected keyword_end, expecting end-of-input
# ruby --version
ruby 2.0.0p481 (2014-05-08 revision 45883) [universal.x86_64-darwin14]

I wandered a bunch of sites on updating my system to ruby 2.5. I don't use homebrew and prefer MacPort. I used rbenv to upgrade to ruby (rvm had certificate issues which rbenv fixed). After the dust settled, I ran the script and it didn't find any phone numbers, which is what I expected.

Thanks for writing this. And making me research stuff into ruby.

@dylanmckay

This comment has been minimized.

Show comment Hide comment
@dylanmckay

dylanmckay Mar 24, 2018

Interesting, I'm running ruby 2.3.5p376 (2017-09-14 revision 59905) [x86_64-darwin16]

Owner

dylanmckay commented Mar 24, 2018

Interesting, I'm running ruby 2.3.5p376 (2017-09-14 revision 59905) [x86_64-darwin16]

@dylanmckay

This comment has been minimized.

Show comment Hide comment
@dylanmckay

dylanmckay Mar 24, 2018

I see, the script used keyword arguments, which were introduced in Ruby 2.1.

Owner

dylanmckay commented Mar 24, 2018

I see, the script used keyword arguments, which were introduced in Ruby 2.1.

@dylanmckay

This comment has been minimized.

Show comment Hide comment
@dylanmckay

dylanmckay Mar 24, 2018

@SunDevil71

You don't need to run this script as sudo - and I definitely don't recommend it!

It probably also mucks with the permissions of the files it creates, plus it'll probably not have access to any locally-installed gems in your local rvm install.

All you should have to do is

gem install nokogiri
ruby ./facebook-contact-info-summary
Owner

dylanmckay commented Mar 24, 2018

@SunDevil71

You don't need to run this script as sudo - and I definitely don't recommend it!

It probably also mucks with the permissions of the files it creates, plus it'll probably not have access to any locally-installed gems in your local rvm install.

All you should have to do is

gem install nokogiri
ruby ./facebook-contact-info-summary
@dmatej

This comment has been minimized.

Show comment Hide comment
@dmatej

dmatej Mar 24, 2018

Negtive ... but this works:
1.

git clone git@gist.github.com:2b191a10068bd87d0fffba242db44b52.git;
cd 2b191a10068bd87d0fffba242db44b52;
sudo apt-get install ruby-dev zlib1g-dev;
sudo gem install nokogiri;
  1. Visit Facebook and download zip from https://www.facebook.com/settings?tab=account
  2. Save the zip file to the 2b191a10068bd87d0fffba242db44b52 folder
unzip facebook-username.zip;
ruby ./facebook-contact-info-summary

dmatej commented Mar 24, 2018

Negtive ... but this works:
1.

git clone git@gist.github.com:2b191a10068bd87d0fffba242db44b52.git;
cd 2b191a10068bd87d0fffba242db44b52;
sudo apt-get install ruby-dev zlib1g-dev;
sudo gem install nokogiri;
  1. Visit Facebook and download zip from https://www.facebook.com/settings?tab=account
  2. Save the zip file to the 2b191a10068bd87d0fffba242db44b52 folder
unzip facebook-username.zip;
ruby ./facebook-contact-info-summary
@markushausammann

This comment has been minimized.

Show comment Hide comment
@markushausammann

markushausammann Mar 24, 2018

Works fine on my default mac setup and I don't find any call history. I think it's regional. It would be very interesting to do a survey to see what regions/countries have that problem.

Works fine on my default mac setup and I don't find any call history. I think it's regional. It would be very interesting to do a survey to see what regions/countries have that problem.

@joefiorini

This comment has been minimized.

Show comment Hide comment
@joefiorini

joefiorini Mar 24, 2018

I ran it on mine, and manually inspected contact_info.html (after seeing it in the browser). Fortunately, there is no data in that file. Either Facebook got wise and removed it or they don't have call info on me. Been running Messenger & Facebook app for a couple years on iOS.

I ran it on mine, and manually inspected contact_info.html (after seeing it in the browser). Fortunately, there is no data in that file. Either Facebook got wise and removed it or they don't have call info on me. Been running Messenger & Facebook app for a couple years on iOS.

@jpluimers

This comment has been minimized.

Show comment Hide comment
@jpluimers

jpluimers Mar 24, 2018

Hmm:

$ git clone https://gist.github.com/2b191a10068bd87d0fffba242db44b52.git
Cloning into '2b191a10068bd87d0fffba242db44b52'...
remote: Counting objects: 30, done.
remote: Compressing objects: 100% (30/30), done.
remote: Total 30 (delta 9), reused 0 (delta 0), pack-reused 0
Unpacking objects: 100% (30/30), done.
$ ruby 2b191a10068bd87d0fffba242db44b52/facebook-contact-info-summary.rb 
2b191a10068bd87d0fffba242db44b52/facebook-contact-info-summary.rb:60: syntax error, unexpected ')'
2b191a10068bd87d0fffba242db44b52/facebook-contact-info-summary.rb:71: syntax error, unexpected keyword_end, expecting end-of-input
$ ruby --version
ruby 2.0.0p648 (2015-12-16 revision 53162) [universal.x86_64-darwin16]

Too bad I cannot upgrade to MacOS high sierra yet, as MacOS sierra comes with this very old ruby version.

Hmm:

$ git clone https://gist.github.com/2b191a10068bd87d0fffba242db44b52.git
Cloning into '2b191a10068bd87d0fffba242db44b52'...
remote: Counting objects: 30, done.
remote: Compressing objects: 100% (30/30), done.
remote: Total 30 (delta 9), reused 0 (delta 0), pack-reused 0
Unpacking objects: 100% (30/30), done.
$ ruby 2b191a10068bd87d0fffba242db44b52/facebook-contact-info-summary.rb 
2b191a10068bd87d0fffba242db44b52/facebook-contact-info-summary.rb:60: syntax error, unexpected ')'
2b191a10068bd87d0fffba242db44b52/facebook-contact-info-summary.rb:71: syntax error, unexpected keyword_end, expecting end-of-input
$ ruby --version
ruby 2.0.0p648 (2015-12-16 revision 53162) [universal.x86_64-darwin16]

Too bad I cannot upgrade to MacOS high sierra yet, as MacOS sierra comes with this very old ruby version.

@dylanmckay

This comment has been minimized.

Show comment Hide comment
@dylanmckay

dylanmckay Mar 24, 2018

@markushausammann

It would be very interesting to do a survey to see what regions/countries have that problem.

The survey: form
The results: responses

Owner

dylanmckay commented Mar 24, 2018

@markushausammann

It would be very interesting to do a survey to see what regions/countries have that problem.

The survey: form
The results: responses

@MatthewRalston

This comment has been minimized.

Show comment Hide comment
@MatthewRalston

MatthewRalston Mar 24, 2018

Jesus people... use RVM.

Jesus people... use RVM.

@invdj

This comment has been minimized.

Show comment Hide comment
@invdj

invdj Mar 24, 2018

Script works.
No any history.
My fb page from 2008 used in Netherlands and Serbia.
capture

invdj commented Mar 24, 2018

Script works.
No any history.
My fb page from 2008 used in Netherlands and Serbia.
capture

@handmilkingsoftware

This comment has been minimized.

Show comment Hide comment
@handmilkingsoftware

handmilkingsoftware Mar 24, 2018

I tried to run the script but kept getting 0 results

user@machine /home/user/Downloads/fb/facebook-myfbuserdirectory $ ruby -v
ruby 2.3.5p376 (2017-09-14 revision 59905) [x86_64-linux]
user@machine /home/user/Downloads/fb/facebook-myfbuserdirectory $ ll
total 224
drwxrwxr-x  7 user user   4096 Mar 24 12:26 ./
drwxrwxr-x  3 user user   4096 Mar 24 12:25 ../
drwxrwxr-x  2 user user   4096 Mar 24 12:26 csv/
-rwxrwxr-x  1 user user   6314 Mar 24 12:34 facebook-contact-info-summary.rb*
drwxrwxrwx  2 user user   4096 Mar 24 11:10 html/
-rw-r--r--  1 user user 174426 Mar 24 11:10 index.htm
drwxrwxrwx  8 user user  20480 Mar 24 11:10 messages/
drwxrwxrwx 18 user user   4096 Mar 24 11:10 photos/
drwxrwxrwx  3 user user   4096 Mar 24 11:10 videos/
user@machine /home/user/Downloads/fb/facebook-myfbuserdirectory $ ./facebook-contact-info-summary.rb 
------------------------
Call History

------------------------

------------------------
SMS History

------------------------

------------------------
MMS History

------------------------

------------------------
The full list of phone numbers that have stored data

------------------------

A brief summary of phone records
------------------------
There are phone records for 0 distinct phone numbers
There are records of 0 distinct cell phone calls
      There are records of 0 distinct SMS messages
      There are records of 0 distinct MMS messages
      ------------------------

------------------------
dumped metadata to CSV files at /home/user/Downloads/fb/facebook-myfbuserdirectory/csv
user@machine /home/user/Downloads/fb/facebook-myfbuserdirectory $ 

I tried to run the script but kept getting 0 results

user@machine /home/user/Downloads/fb/facebook-myfbuserdirectory $ ruby -v
ruby 2.3.5p376 (2017-09-14 revision 59905) [x86_64-linux]
user@machine /home/user/Downloads/fb/facebook-myfbuserdirectory $ ll
total 224
drwxrwxr-x  7 user user   4096 Mar 24 12:26 ./
drwxrwxr-x  3 user user   4096 Mar 24 12:25 ../
drwxrwxr-x  2 user user   4096 Mar 24 12:26 csv/
-rwxrwxr-x  1 user user   6314 Mar 24 12:34 facebook-contact-info-summary.rb*
drwxrwxrwx  2 user user   4096 Mar 24 11:10 html/
-rw-r--r--  1 user user 174426 Mar 24 11:10 index.htm
drwxrwxrwx  8 user user  20480 Mar 24 11:10 messages/
drwxrwxrwx 18 user user   4096 Mar 24 11:10 photos/
drwxrwxrwx  3 user user   4096 Mar 24 11:10 videos/
user@machine /home/user/Downloads/fb/facebook-myfbuserdirectory $ ./facebook-contact-info-summary.rb 
------------------------
Call History

------------------------

------------------------
SMS History

------------------------

------------------------
MMS History

------------------------

------------------------
The full list of phone numbers that have stored data

------------------------

A brief summary of phone records
------------------------
There are phone records for 0 distinct phone numbers
There are records of 0 distinct cell phone calls
      There are records of 0 distinct SMS messages
      There are records of 0 distinct MMS messages
      ------------------------

------------------------
dumped metadata to CSV files at /home/user/Downloads/fb/facebook-myfbuserdirectory/csv
user@machine /home/user/Downloads/fb/facebook-myfbuserdirectory $ 
@daniel361

This comment has been minimized.

Show comment Hide comment
@daniel361

daniel361 Mar 25, 2018

Downloaded the dump, ran the script on CentOS with 2.4.2 (installed with rvm btw, works like a charm).
No records on phone, sms or mms, I use FB and Messenger App on Android, but I didn't give the apps any permissions to access this data.
Interesting fact is the contacts dump in the zip, I found numbers that I deleted long time ago from my google account, shocking...

Downloaded the dump, ran the script on CentOS with 2.4.2 (installed with rvm btw, works like a charm).
No records on phone, sms or mms, I use FB and Messenger App on Android, but I didn't give the apps any permissions to access this data.
Interesting fact is the contacts dump in the zip, I found numbers that I deleted long time ago from my google account, shocking...

@Lala5th

This comment has been minimized.

Show comment Hide comment
@Lala5th

Lala5th Mar 25, 2018

@handmilkingsoftware I had the same results so I dug around in the code and realized that some terms are in your language. Strangely I could not get it work by rewriting the appropriate parts of the code (most likely because I can barely understand Ruby) so I changed all of the translations in the .htm to their original counterparts and it worked. Hope this helps

Lala5th commented Mar 25, 2018

@handmilkingsoftware I had the same results so I dug around in the code and realized that some terms are in your language. Strangely I could not get it work by rewriting the appropriate parts of the code (most likely because I can barely understand Ruby) so I changed all of the translations in the .htm to their original counterparts and it worked. Hope this helps

@kimduckhwan

This comment has been minimized.

Show comment Hide comment
@kimduckhwan

kimduckhwan Mar 25, 2018

Hi. I just ran the code and some of my contacts names written in Korean were broken. So hard to read the csv result. I can see correct name in html file though.

Hi. I just ran the code and some of my contacts names written in Korean were broken. So hard to read the csv result. I can see correct name in html file though.

@dylanmckay

This comment has been minimized.

Show comment Hide comment
@dylanmckay

dylanmckay Mar 26, 2018

If anybody wants to copy-paste the exact text of the metadata headings in their language, I can add support for it to the script.

I specifically need to know the inner HTML of the <h2> element sitting above the different metadata types

in English, these look like

<h2>Call History</h2>
...
<h2>SMS History</h2>
..
<h2>MMS History</h2>
...
Owner

dylanmckay commented Mar 26, 2018

If anybody wants to copy-paste the exact text of the metadata headings in their language, I can add support for it to the script.

I specifically need to know the inner HTML of the <h2> element sitting above the different metadata types

in English, these look like

<h2>Call History</h2>
...
<h2>SMS History</h2>
..
<h2>MMS History</h2>
...
@Brunocerous

This comment has been minimized.

Show comment Hide comment
@Brunocerous

Brunocerous Mar 26, 2018

Thank you for the script, and for all the help in getting to run my first Ruby code in OS X terminal.

Glad I never downloaded the Facebook app, or used the built-in Android versions some phones had preinstalled.


Call History



SMS History



MMS History



The full list of phone numbers that have stored data


A brief summary of phone records

There are phone records for 0 distinct phone numbers
There are records of 0 distinct cell phone calls
There are records of 0 distinct SMS messages
There are records of 0 distinct MMS messages
------------------------

Thank you for the script, and for all the help in getting to run my first Ruby code in OS X terminal.

Glad I never downloaded the Facebook app, or used the built-in Android versions some phones had preinstalled.


Call History



SMS History



MMS History



The full list of phone numbers that have stored data


A brief summary of phone records

There are phone records for 0 distinct phone numbers
There are records of 0 distinct cell phone calls
There are records of 0 distinct SMS messages
There are records of 0 distinct MMS messages
------------------------

@nedwards1980

This comment has been minimized.

Show comment Hide comment
@nedwards1980

nedwards1980 Mar 26, 2018

Hello, could someone possibly talk/walk me through how to run this please? I have downloaded Ruby but I have no idea how to work or run this! Thank you for your pity!!!

Hello, could someone possibly talk/walk me through how to run this please? I have downloaded Ruby but I have no idea how to work or run this! Thank you for your pity!!!

@Nazalas

This comment has been minimized.

Show comment Hide comment
@Nazalas

Nazalas Mar 26, 2018

@nedwards1980,

You need to first install ruby. If you are on windows, you can install it here https://rubyinstaller.org/downloads/.
Then you need to install the nokogirl lib. you can do that after you install ruby by opening a command prompt and pasting "https://rubyinstaller.org/downloads/"
Download the facebook zip file and extract it.
Save this script as "facebook-contact-info-summary.rb" in the same folder as your facebook file
navigate to that folder with command prompt and run "ruby facebook-contact-info-summary.rb"

Nazalas commented Mar 26, 2018

@nedwards1980,

You need to first install ruby. If you are on windows, you can install it here https://rubyinstaller.org/downloads/.
Then you need to install the nokogirl lib. you can do that after you install ruby by opening a command prompt and pasting "https://rubyinstaller.org/downloads/"
Download the facebook zip file and extract it.
Save this script as "facebook-contact-info-summary.rb" in the same folder as your facebook file
navigate to that folder with command prompt and run "ruby facebook-contact-info-summary.rb"

@suziqblu84

This comment has been minimized.

Show comment Hide comment
@suziqblu84

suziqblu84 Mar 26, 2018

I am less tech savy... Looking to see if someone can provide step-by-step how to apply this? I am on a MAC. Very curious..... Thanks in advance!

I am less tech savy... Looking to see if someone can provide step-by-step how to apply this? I am on a MAC. Very curious..... Thanks in advance!

@PhoenixAF24

This comment has been minimized.

Show comment Hide comment
@PhoenixAF24

PhoenixAF24 Mar 26, 2018

@Nazalas and @dylanmckay,

I've installed both Ruby and extracted the nokogirl gem and when I go to run the script, I get the same output as a number of others (i.e. 0 records).

OS: Windows 10
Ruby: Installed from https://rubyinstaller.org/downloads
Ran from the "DOS" command prompt: gem install nokogirl

Question for the group... I did change the default prompts during the install from C: to D: as I have more space on D. I don't seem to think it would be the problem.

Any help would be appreciated.

@Nazalas and @dylanmckay,

I've installed both Ruby and extracted the nokogirl gem and when I go to run the script, I get the same output as a number of others (i.e. 0 records).

OS: Windows 10
Ruby: Installed from https://rubyinstaller.org/downloads
Ran from the "DOS" command prompt: gem install nokogirl

Question for the group... I did change the default prompts during the install from C: to D: as I have more space on D. I don't seem to think it would be the problem.

Any help would be appreciated.

@davisx23

This comment has been minimized.

Show comment Hide comment
@davisx23

davisx23 Mar 27, 2018

Anyone running Windows 10 have a step-by-step on what worked? I've managed to get to the folder path in Command Prompt w/Ruby, but every time I run the facebook-contact-info-summary.rb command and get read errors. I have the .rb file saved in the HTML folder from the Facebook download, but nothing seems to work.

I'd appreciate any help or successful tips anyone running Windows 10 has. Thanks!

Anyone running Windows 10 have a step-by-step on what worked? I've managed to get to the folder path in Command Prompt w/Ruby, but every time I run the facebook-contact-info-summary.rb command and get read errors. I have the .rb file saved in the HTML folder from the Facebook download, but nothing seems to work.

I'd appreciate any help or successful tips anyone running Windows 10 has. Thanks!

@JohnAmbra

This comment has been minimized.

Show comment Hide comment
@JohnAmbra

JohnAmbra Mar 27, 2018

Thank you for the script.

The script does not generate the csv files and I think the problem are my timestamps in Brazilian Portuguese. Any way to solve?

There are phone records for 0 distinct phone numbers
There are records of 511 distinct cell phone calls
   C:/Ruby24-x64/lib/ruby/2.4.0/time.rb:306:in `local': argument out of range (ArgumentError)
        from C:/Ruby24-x64/lib/ruby/2.4.0/time.rb:306:in `make_time'
        from C:/Ruby24-x64/lib/ruby/2.4.0/time.rb:370:in `parse'
        from facebook-contact-info-summary.rb:88:in `block in print_timestamps'
        from facebook-contact-info-summary.rb:88:in `map'
        from facebook-contact-info-summary.rb:88:in `print_timestamps'
        from facebook-contact-info-summary.rb:153:in `print_metadata_human'
        from facebook-contact-info-summary.rb:167:in `<main>'

Timestamp in contact_info.html:
<td>Domingo, 15 de janeiro de 2017 às 18:28 UTC-02</td>

Anyway, this is a realy good work, thank you again.

Thank you for the script.

The script does not generate the csv files and I think the problem are my timestamps in Brazilian Portuguese. Any way to solve?

There are phone records for 0 distinct phone numbers
There are records of 511 distinct cell phone calls
   C:/Ruby24-x64/lib/ruby/2.4.0/time.rb:306:in `local': argument out of range (ArgumentError)
        from C:/Ruby24-x64/lib/ruby/2.4.0/time.rb:306:in `make_time'
        from C:/Ruby24-x64/lib/ruby/2.4.0/time.rb:370:in `parse'
        from facebook-contact-info-summary.rb:88:in `block in print_timestamps'
        from facebook-contact-info-summary.rb:88:in `map'
        from facebook-contact-info-summary.rb:88:in `print_timestamps'
        from facebook-contact-info-summary.rb:153:in `print_metadata_human'
        from facebook-contact-info-summary.rb:167:in `<main>'

Timestamp in contact_info.html:
<td>Domingo, 15 de janeiro de 2017 às 18:28 UTC-02</td>

Anyway, this is a realy good work, thank you again.

@scifigeek64

This comment has been minimized.

Show comment Hide comment
@scifigeek64

scifigeek64 Mar 27, 2018

As is the case with many others, when running the script I get this:
Call History

SMS History

MMS History

The full list of phone numbers that have stored data

A brief summary of phone records
There are phone records for 0 distinct phone numbers
There are records of 0 distinct cell phone calls
There are records of 0 distinct SMS messages
There are records of 0 distinct MMS messages

Does this mean they are no longer including that data in the dump, or that they never collected it for me? Anyone have any ideas?

As is the case with many others, when running the script I get this:
Call History

SMS History

MMS History

The full list of phone numbers that have stored data

A brief summary of phone records
There are phone records for 0 distinct phone numbers
There are records of 0 distinct cell phone calls
There are records of 0 distinct SMS messages
There are records of 0 distinct MMS messages

Does this mean they are no longer including that data in the dump, or that they never collected it for me? Anyone have any ideas?

@AyabongaQwabi

This comment has been minimized.

Show comment Hide comment
@AyabongaQwabi

AyabongaQwabi Mar 27, 2018

Is there a way we could just pull this data from an API without downloading the zip ?

Is there a way we could just pull this data from an API without downloading the zip ?

@nachopants

This comment has been minimized.

Show comment Hide comment
@nachopants

nachopants Mar 27, 2018

Yeah I'm looking at the code, my contact_info.html file looks nothing like what the script is trying to parse.

Australia.

image

Yeah I'm looking at the code, my contact_info.html file looks nothing like what the script is trying to parse.

Australia.

image

@dylanmckay

This comment has been minimized.

Show comment Hide comment
@dylanmckay

dylanmckay Mar 27, 2018

@PhoenixAF24

Question for the group... I did change the default prompts during the install from C: to D: as I have more space on D. I don't seem to think it would be the problem.

Sounds like you did everything right - there aren't any metadata records in the data dump, which is good.

You can confirm this by looking at html/contact_info.htm. Scroll through the page - you shouldn't see any "Call History" or "SMS History" headings

Owner

dylanmckay commented Mar 27, 2018

@PhoenixAF24

Question for the group... I did change the default prompts during the install from C: to D: as I have more space on D. I don't seem to think it would be the problem.

Sounds like you did everything right - there aren't any metadata records in the data dump, which is good.

You can confirm this by looking at html/contact_info.htm. Scroll through the page - you shouldn't see any "Call History" or "SMS History" headings

@dylanmckay

This comment has been minimized.

Show comment Hide comment
@dylanmckay

dylanmckay Mar 27, 2018

@nachopants that looks like the "contacts" table.

That is separate to the call+sms metadata sections.

If you have no contacts, the table you are inspecting via Chrome Dev tools would not exist.

If there are call records, there should be a <h2>Call History</h2>, or a <h2>SMS History</h2>.

If you cannot see the latter, it sounds like you probably don't have any data stored.

Owner

dylanmckay commented Mar 27, 2018

@nachopants that looks like the "contacts" table.

That is separate to the call+sms metadata sections.

If you have no contacts, the table you are inspecting via Chrome Dev tools would not exist.

If there are call records, there should be a <h2>Call History</h2>, or a <h2>SMS History</h2>.

If you cannot see the latter, it sounds like you probably don't have any data stored.

@dylanmckay

This comment has been minimized.

Show comment Hide comment
@dylanmckay

dylanmckay Mar 27, 2018

@AyabongaQwabi

Nope - the metadata does not get exposed to third parties, therefore it is not in the API. You may be able to use the API to request a list just of contacts though.

Owner

dylanmckay commented Mar 27, 2018

@AyabongaQwabi

Nope - the metadata does not get exposed to third parties, therefore it is not in the API. You may be able to use the API to request a list just of contacts though.

@dylanmckay

This comment has been minimized.

Show comment Hide comment
@dylanmckay

dylanmckay Mar 27, 2018

@JohnAmbra I've pushed a new version of the script that should ignore any time parse errors and use the original time format outputted by FB.

Thanks for your kind comments!

Owner

dylanmckay commented Mar 27, 2018

@JohnAmbra I've pushed a new version of the script that should ignore any time parse errors and use the original time format outputted by FB.

Thanks for your kind comments!

@PhoenixAF24

This comment has been minimized.

Show comment Hide comment
@PhoenixAF24

PhoenixAF24 Mar 27, 2018

@dylanmckay,

I concur with @JohnAmbra, thank you for doing this. In light of the situation over the last few days, thank you for putting this together.

I did confirm that I have no Call history or SMS history so thank you for the confirmation.

As a related enhancement, have you thought of possibly extending the script so that it converts each of the HTML pages into respective CSV pages to facilitate our own review?

@dylanmckay,

I concur with @JohnAmbra, thank you for doing this. In light of the situation over the last few days, thank you for putting this together.

I did confirm that I have no Call history or SMS history so thank you for the confirmation.

As a related enhancement, have you thought of possibly extending the script so that it converts each of the HTML pages into respective CSV pages to facilitate our own review?

@mforce

This comment has been minimized.

Show comment Hide comment
@mforce

mforce Mar 27, 2018

@dylanmckay
Thanks for creating this script man, I ran it against the zip file I downloaded today and it did not find any records of txt or calls.
Could it be possible that facebook updated their side to not include that data in the archived zip anymore?

Maybe people that already verified they had txt/calls tracked can re-download their archive to verify they are still there. This will confirm if facebook is doing damage control?

mforce commented Mar 27, 2018

@dylanmckay
Thanks for creating this script man, I ran it against the zip file I downloaded today and it did not find any records of txt or calls.
Could it be possible that facebook updated their side to not include that data in the archived zip anymore?

Maybe people that already verified they had txt/calls tracked can re-download their archive to verify they are still there. This will confirm if facebook is doing damage control?

@davisx23

This comment has been minimized.

Show comment Hide comment
@davisx23

davisx23 Mar 27, 2018

@mforce
I had the same concern, but I'm wondering if it's related to the type of device used? I've been using iOS for quite some time, and with the difference in security prompts and easy access to those controls, maybe Android OS devices (for example) have been more susceptible than iOS devices?

I did the same process that it sounds like you did and found no call records and no sms records, which I was expecting I'd encounter. So either I'm doing something wrong, the records don't exist, or (as you stated) Facebook has stopped providing those records.

@mforce
I had the same concern, but I'm wondering if it's related to the type of device used? I've been using iOS for quite some time, and with the difference in security prompts and easy access to those controls, maybe Android OS devices (for example) have been more susceptible than iOS devices?

I did the same process that it sounds like you did and found no call records and no sms records, which I was expecting I'd encounter. So either I'm doing something wrong, the records don't exist, or (as you stated) Facebook has stopped providing those records.

@GrahamHand

This comment has been minimized.

Show comment Hide comment
@GrahamHand

GrahamHand Mar 28, 2018

@davisx23 And yet I crack open the files (since they're html) and there are plenty of phone number there (some I thought I'd long lost but I'm still mad AF!).

Run the script and nothing found.

@davisx23 And yet I crack open the files (since they're html) and there are plenty of phone number there (some I thought I'd long lost but I'm still mad AF!).

Run the script and nothing found.

@dmitriy1980

This comment has been minimized.

Show comment Hide comment
@dmitriy1980

dmitriy1980 Mar 28, 2018

The reason why script returns no data is because it's reading contact_info.htm file and trying to get information from sections titled "Call History", "SMS History", "MMS History". If they are not present in those sections, script will return no data.

The reason why script returns no data is because it's reading contact_info.htm file and trying to get information from sections titled "Call History", "SMS History", "MMS History". If they are not present in those sections, script will return no data.

@dylanmckay

This comment has been minimized.

Show comment Hide comment
@dylanmckay

dylanmckay Mar 28, 2018

@PhoenixAF24

I concur with @JohnAmbra, thank you for doing this. In light of the situation over the last few days, thank you for putting this together.

Thanks!

As a related enhancement, have you thought of possibly extending the script so that it converts each of the HTML pages into respective CSV pages to facilitate our own review?

It already has this! Check out the csv folder in the same folder when you run the script.

It does have duplicated "Status, Timestamp, Duration, Contact name" headings for each mobile number, but those are easy to filter out manually.

Owner

dylanmckay commented Mar 28, 2018

@PhoenixAF24

I concur with @JohnAmbra, thank you for doing this. In light of the situation over the last few days, thank you for putting this together.

Thanks!

As a related enhancement, have you thought of possibly extending the script so that it converts each of the HTML pages into respective CSV pages to facilitate our own review?

It already has this! Check out the csv folder in the same folder when you run the script.

It does have duplicated "Status, Timestamp, Duration, Contact name" headings for each mobile number, but those are easy to filter out manually.

@dylanmckay

This comment has been minimized.

Show comment Hide comment
@dylanmckay

dylanmckay Mar 28, 2018

@mforce

Thanks for creating this script man, I ran it against the zip file I downloaded today and it did not find any records of txt or calls.
Could it be possible that facebook updated their side to not include that data in the archived zip anymore?

Maybe people that already verified they had txt/calls tracked can re-download their archive to verify they are still there. This will confirm if facebook is doing damage control?

To be honest I doubt it. It'd be way too obvious to simply start hiding this data, and the PR nightmare it'd cause makes the decision an obvious bad one for any business IMO.

I've also redownloaded my data (~2 days ago) and confirmed that the metadata was still there. This is obviously just one data point though.

You need to be careful when doing this because FB seems to cache the ZIP for ~5 days. That means in order to get a freshly built ZIP based on current data, you need to wait a while before requesting it.

I think it's fair to say that the metadata was never collected on your account.

Owner

dylanmckay commented Mar 28, 2018

@mforce

Thanks for creating this script man, I ran it against the zip file I downloaded today and it did not find any records of txt or calls.
Could it be possible that facebook updated their side to not include that data in the archived zip anymore?

Maybe people that already verified they had txt/calls tracked can re-download their archive to verify they are still there. This will confirm if facebook is doing damage control?

To be honest I doubt it. It'd be way too obvious to simply start hiding this data, and the PR nightmare it'd cause makes the decision an obvious bad one for any business IMO.

I've also redownloaded my data (~2 days ago) and confirmed that the metadata was still there. This is obviously just one data point though.

You need to be careful when doing this because FB seems to cache the ZIP for ~5 days. That means in order to get a freshly built ZIP based on current data, you need to wait a while before requesting it.

I think it's fair to say that the metadata was never collected on your account.

@dylanmckay

This comment has been minimized.

Show comment Hide comment
@dylanmckay

dylanmckay Mar 28, 2018

@davisx23

I had the same concern, but I'm wondering if it's related to the type of device used? I've been using iOS for quite some time, and with the difference in security prompts and easy access to those controls, maybe Android OS devices (for example) have been more susceptible than iOS devices?

It seems that way. Of all of the people I've seen with metadata downloads, I can't recall a single one who mentioned iOS, but I've seen dozens of tweets specifically mentioning collected metadata and Android.

Owner

dylanmckay commented Mar 28, 2018

@davisx23

I had the same concern, but I'm wondering if it's related to the type of device used? I've been using iOS for quite some time, and with the difference in security prompts and easy access to those controls, maybe Android OS devices (for example) have been more susceptible than iOS devices?

It seems that way. Of all of the people I've seen with metadata downloads, I can't recall a single one who mentioned iOS, but I've seen dozens of tweets specifically mentioning collected metadata and Android.

@AMurray8

This comment has been minimized.

Show comment Hide comment
@AMurray8

AMurray8 Mar 28, 2018

Any help with this error? I have downloaded Ruby, installed nokogiri but still do not get an output. I manually check the files and do not see anything but i want to understand how to make it work. I want to be able to check out files for my family members.

image

i have the script in the folder with the html file.
image

AMurray8 commented Mar 28, 2018

Any help with this error? I have downloaded Ruby, installed nokogiri but still do not get an output. I manually check the files and do not see anything but i want to understand how to make it work. I want to be able to check out files for my family members.

image

i have the script in the folder with the html file.
image

@TLCowboysFan

This comment has been minimized.

Show comment Hide comment
@TLCowboysFan

TLCowboysFan Mar 28, 2018

I am totally unfamiliar with coding, but am computer literate. I've downloaded Ruby and the FB data, but have no idea how to get it going. Please help. I'm using Win 7, SP1, 64 bit OS.

I am totally unfamiliar with coding, but am computer literate. I've downloaded Ruby and the FB data, but have no idea how to get it going. Please help. I'm using Win 7, SP1, 64 bit OS.

@lgnam

This comment has been minimized.

Show comment Hide comment
@lgnam

lgnam Mar 29, 2018

I am receiving the following error when executing the script:

./facebook-contact-info-summary.rb:55: syntax error, unexpected tLABEL
def section(title, level: 1)
                         ^

lgnam commented Mar 29, 2018

I am receiving the following error when executing the script:

./facebook-contact-info-summary.rb:55: syntax error, unexpected tLABEL
def section(title, level: 1)
                         ^
@dylanmckay

This comment has been minimized.

Show comment Hide comment
@dylanmckay

dylanmckay Apr 1, 2018

@AMurray8 place the script next to the html directory - it looks like it's currently inside of it.

Currently, it is looking for html/html/index.html, but if it is moved up a level it should work.

Owner

dylanmckay commented Apr 1, 2018

@AMurray8 place the script next to the html directory - it looks like it's currently inside of it.

Currently, it is looking for html/html/index.html, but if it is moved up a level it should work.

@dylanmckay

This comment has been minimized.

Show comment Hide comment
@dylanmckay

dylanmckay Apr 1, 2018

@lgnam

You're using Ruby <2.0, the script only runs on Ruby 2.1+, which added support for keyword arguments.

Owner

dylanmckay commented Apr 1, 2018

@lgnam

You're using Ruby <2.0, the script only runs on Ruby 2.1+, which added support for keyword arguments.

@captnkirk

This comment has been minimized.

Show comment Hide comment
@captnkirk

captnkirk Apr 4, 2018

It took a lot of trial and error to finally get Ruby 2.4.1 installed on my MBPro running Sierra, so that I could use this script, but it now works. Most of my problems were due to file ownership issues in the /usr/local/lib and /usr/local/sbin folders that prevented HomeBrew from working properly. Once that was resolved, the rest went smoothly.

It seems that they have cleaned up any damning evidence in the Facebook Data download that they are now releasing to their Users.

I don't know if they have deleted any of the Private information from their systems (ie. phone calls, SMS messages, etc), that some people have seen in earlier downloads, but they are no longer including that data in the ZIP file that Users can download from Facebook.

I hope that Apple gives them hell about recording User activity (phone calls and messaging) that was unrelated to the Facebook App. That is a BIG violation of the Apple Developer Agreement.

It took a lot of trial and error to finally get Ruby 2.4.1 installed on my MBPro running Sierra, so that I could use this script, but it now works. Most of my problems were due to file ownership issues in the /usr/local/lib and /usr/local/sbin folders that prevented HomeBrew from working properly. Once that was resolved, the rest went smoothly.

It seems that they have cleaned up any damning evidence in the Facebook Data download that they are now releasing to their Users.

I don't know if they have deleted any of the Private information from their systems (ie. phone calls, SMS messages, etc), that some people have seen in earlier downloads, but they are no longer including that data in the ZIP file that Users can download from Facebook.

I hope that Apple gives them hell about recording User activity (phone calls and messaging) that was unrelated to the Facebook App. That is a BIG violation of the Apple Developer Agreement.

@rymalia

This comment has been minimized.

Show comment Hide comment
@rymalia

rymalia Apr 4, 2018

I've been a Facebook user since 2007, in the US, with the FB and Messenger app on both iOS and Android phones for much of that time. My data contains none of the phone records in the initial @dylanmckay example. I'm guessing it's a phone or messaging service I never opted into?

My total Facebook data download was about 230 MB uncompressed. In comparison, I just requested my Google data dump and it's 321 GB, of compressed data. I can't wait to learn what that one has in it.

Thank's for putting this script together @dylanmckay

rymalia commented Apr 4, 2018

I've been a Facebook user since 2007, in the US, with the FB and Messenger app on both iOS and Android phones for much of that time. My data contains none of the phone records in the initial @dylanmckay example. I'm guessing it's a phone or messaging service I never opted into?

My total Facebook data download was about 230 MB uncompressed. In comparison, I just requested my Google data dump and it's 321 GB, of compressed data. I can't wait to learn what that one has in it.

Thank's for putting this script together @dylanmckay

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment