Foursquare backup notes

README.md

This is a personal log of getting a Foursquare backup/export setup. It's intended to be a bit of a resource for those who are bridging between technical and non-technical

It's not super explanatory either though. If you do have issues, feel free to email me and I'll update it in fuller detail =)

---

We want to export our Foursquare data and make the switch to Yelp in 2019 We have lost faith in Foursquare being maintained on Android and thus discovery can only be done on Yelp As a result, we might as well consolidate the data

However, we don't want to lose it indefinitely so we're going to export it We should verify we have all the data we expect. This includes:

• Timestamp
• Lat, lng
• Venue name
• Bonus points: Stickers used
• Bonus points: Check-in text
• Bonus points: People checked in with

We started with a roll your own solution:

https://www.quora.com/Is-it-possible-to-export-my-location-history-from-Foursquare

Then signed up for a Personal account which has 995k/day requests (and also a confusing UI which makes it look like you'll be charged)

We verified curl works:

export FOURSQUARE_CLIENT_ID=<CLIENT_ID>
export FOURSQUARE_CLIENT_SECRET=<CLIENT_SECRET>
curl -X GET -G   'https://api.foursquare.com/v2/venues/explore'     -d client_id="$FOURSQUARE_CLIENT_ID"     -d client_secret="$FOURSQUARE_CLIENT_SECRET"     -d v="20180323"     -d ll="40.7243,-74.0018"     -d query="coffee"     -
d limit=1 | underscore pretty
# {
#   meta: { code: 200, requestId: "5c1a2c1b9fb6b75152ac5ab4" },
#   response: {
#     suggestedFilters: {
#       header: "Tap to show:",
#       filters: [
# ...

We're now evaluating GitHub solutions:

• https://github.com/jdevelop/foursquare-kml, dead
• https://github.com/crccheck/foursquare-archiver, seems promising
• https://gist.github.com/FloorLamp/8273330, slightly better than Quora
• https://gist.github.com/tmcw/3350235, this is quite clean
  • https://gist.github.com/tmcw/3407807, can be used on top which makes GeoJSON

Okay, now comparing the promising ones...

• foursquare-archiver is a glorified curl script =/
  • It has a mislabeled function for friend checkins which are really recent checkins
  • Its csv.py is a one-off script
• tmcw gist seems promising
  • TIL % was a placeholder for .format
  • Oooh, first time seeing the fancy open a browser to get a token I've seen
  • The script is pretty old but useful still I guess
  • Clever usage of filesystem for resuming
  • It does some overkill with dumping every item as its own JSON but I guess that's fine

Usage of foursquare_archiver.py

# Clone our repo
git clone https://gist.github.com/tmcw/3350235 gist-foursquare-archive
cd gist-foursquare-archive

# Realize we need to add a valid redirect URI for in Foursquare developer dashboard so we go and do that
# https://foursquare.com/developers/app/<CLIENT_ID>
# Redirect URI should be http://localhost:8000

# Verify this page works
xdg-open https://foursquare.com/oauth2/authenticate?client_id=<CLIENT_ID>&response_type=token&redirect_uri=http://localhost:8000

# It does! Fuck yea!

# Manually take the `access_token` and export it to another environment variable
export FOURSQUARE_OAUTH_TOKEN=<OAUTH_TOKEN>

# Create a contained environment for script to run via `virtualenvwrapper`
mkvirtualenv gist-foursquare-archive
# Can return to environment via `workon gist-foursquare-archive`

# Install critical dependencies
pip install requests

# Run our script with our token
python foursquare_archive.py "$FOURSQUARE_OAUTH_TOKEN"

# Oh, the script out of date =/
# {"meta":{"code":410,"errorType":"param_error","errorDetail":"The Foursquare API no longer supports requests that do not pass in a version parameter. For more details see https:\/\/developer.foursquare.com\/overview\/versioning","requestId":"5c1a3557f594df03edc090ae"},"response":{}}

# Also, it never checks error codes??

# We fixed it up and got it working
# It has all 2557 check-ins to date and downloaded them in under a minute ._.

# Patched repo available here: https://gist.github.com/twolfson/3818a48ee29c6b027d2f61e8c991844b

We should verify content now...

• Timestamp
• Lat, lng
• Venue name
• Bonus points: Stickers used
• Bonus points: Check-in text
• Bonus points: People checked in with

New thing to verify:

• Bonus points: Venues I've liked

This might be even easier with this GeoJSON script =o

# Navigate to up a directory from `gist-foursquare-archive`
# cd .. # If necessary

# Clone our repo
git clone https://gist.github.com/tmcw/3407807 gist-foursquare-to-geojson
cd gist-foursquare-to-geojson

# Copy our checkins from our other repo
cp -r ../gist-foursquare-archive/checkins .

# Run our script
python foursquare_to_geojson.py

# Open http://geojson.io/ and input the `.geojson` file
# Realize we want indentation and need a `type: Feature` key/value so we update the repo
# See it's all what we want and disco =D

Okay, this is fantastic. Let's see... data we have:

• Timestamp -- createdAt and timeZoneOffset
• Lat, lng -- venue.location.lat/lng
• Venue name -- venue.name
• Bonus points: Stickers used -- sticker.* (e.g. sticker.name)
• Bonus points: Check-in text -- shout
• Bonus points: People checked in with
• Bonus points: Venues I've liked
  • This prob needs to be another request. I think it's only 1 curl though with limit of 100 as I don't like that many places
  • Yep, we've tweaked the repo to support multiple runs and formats

So we're good now. We've backed up all types that we want. To run a full backup:

• Use steps to run https://gist.github.com/tmcw/3350235 gist-foursquare-archive but on https://gist.github.com/twolfson/3818a48ee29c6b027d2f61e8c991844b
• Update commented sections of code to run against 3 different sectors to backup
• Disco