Aggregating Twitter search results and displaying them on a web page in realtime

README.md

Aggregating Twitter search results and displaying them on a web page in realtime

If you're trying to do this, you came to the right place!

See this code in action here: https://twitter.com/CodingDoug/status/948225623939473413

Setup

These instructions assume that you already have a Firebase project, and you've provisioned Firestore in it.

1. Create a project workspace with the Fireabse CLI with Hosting, Firestore rules, and Functions enabled (with TypeScript).

2. Allow global read no write access to Realtime Database and Storage. Copy firestore.rules from this gist over the one in your project.

3. Copy index.html and follow.js to your public web content folder.

4. Copy index.ts to the functions/src folder.

5. npm install firebase-functions firebase-admin moment

6. Deploy everything with firebase deploy. Note the public URL of the "iffft" function.

7. Create an account with IFTTT.

8. Create a new applet. For "this", select "Twitter", then "New tweet from search". Authorize IFTTT to use a Twitter account to perform searches.

9. Type the search term you want to receive results for (e.g. Firebase).

10. For "that", select "Webhooks", then "Make a web request"

• URL: The public function url from the earlier deploy

• Method: POST

• Content Type: text/plain

• Body: use the following text with no extra whitespace and on a single line:

  text={{Text}}|*|*|firstLinkUrl={{FirstLinkUrl}}|*|*|userName={{UserName}}|*|*|userImageUrl={{UserImageUrl}}|*|*|linkToTweet={{LinkToTweet}}|*|*|tweetEmbedCode={{TweetEmbedCode}}|*|*|createdAt={{CreatedAt}}

11. Save the applet.

12. Keep an eye on your functions logs to see when it gets invoked. It may take a few minutes to get the first one.

How does it work?

1. The IFTTT applet for Twitter search will periodically call the function endpoint with the payload. The payload contains all available "ingredient" values. (Note: IFTTT doesn't know how to escape strings for use in JSON, which is why I'm choosing to use a delimited string to deliver them as key/value pairs.)

2. When the function is invoked, it will parse the body payload, massage the incoming data, and write a document to Firestore in the tweets collection.

3. The web client is always listening to tweets, so when a new document becomes available, it will render a new tweet to the list, inserting it at the top.

4. You can have multiple applets each delivering results for diffrent searches to the same function (e.g. one for Firebase and another for Firestore). It will aggregate the results for all of them in the same place.

Helpful documentation

• https://support.twitter.com/articles/71577-how-to-use-advanced-twitter-search
• https://firebase.google.com/docs/functions/http-events
• https://firebase.google.com/docs/firestore/query-data/listen

firestore.rules

service cloud.firestore {
  match /databases/{database}/documents {
    match /{document=**} {
      allow read;
      allow write: if false;
    }
  }
}

follow.js

// Copyright 2018 Google LLC.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

'use strict'

const allTweetsEl = document.getElementById('all-tweets')
const tweetContainerEl = document.getElementById('tweet-container-template')
tweetContainerEl.remove()
const retweetTextEl = document.getElementById('retweet-text-template')
retweetTextEl.remove()

firebase.firestore().collection('tweets').orderBy('createdAt').startAt(new Date(Date.now()-1000*60*60*8))
.onSnapshot(snapshot => {
    snapshot.docChanges.forEach(change => {
        if (change.type === 'added') {
            const tweet = change.doc.data()
            console.log('New tweet', tweet.id, tweet)
            renderTweet(tweet)
        }
    })
})

function renderTweet(tweet) {
    const tweetEl = tweetContainerEl.cloneNode(true)
    tweetEl.id = `tweet-${tweet.id}`
    allTweetsEl.insertBefore(tweetEl, allTweetsEl.firstChild)
    if (tweet.rtLink) {
        // Not showing retweets for now
        tweetEl.remove()
        // tweetEl.classList.add('rt')
        // const rtEl = retweetTextEl.cloneNode(true)
        // rtEl.textContent = `@${tweet.userName} ${tweet.text}`
        // tweetEl.appendChild(rtEl)
        // setTimeout(() => { tweetEl.classList.add('fade-in') }, 0)
    }
    else {
        // Use the Twitter API to render the tweet
        twttr.widgets.createTweet(tweet.id, tweetEl).then(() => {
            tweetEl.classList.add('fade-in')
        })
    }
}

index.html

<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <title>Follow Firebase</title>

    <script defer src="/__/firebase/4.8.1/firebase-app.js"></script>
    <script defer src="/__/firebase/4.8.1/firebase-firestore.js"></script>
    <script defer src="/__/firebase/init.js"></script>

    <script defer src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
    <script defer src="follow.js"></script>

    <style media="screen">
      body { background: #ECEFF1; color: rgba(0,0,0,0.87); font-family: Roboto, Helvetica, Arial, sans-serif; font-size: 16px; line-height: 1.4; margin: 0; padding: 0; }
      #all-tweets { margin-left: 10px; margin-right: 10px }
      .tweet-container.initially-hidden { opacity: 0; transition: opacity 1s }
      .tweet-container.fade-in { opacity: 1.0 }
      .rt { width: 500px; margin-top: 10px; margin-bottom: 10px }
      .rt .text { padding-left: 20px; padding-right: 20px }
    </style>
  </head>

  <body>
    <div id="all-tweets"></div>
    <div id="tweet-container-template" class="tweet-container initially-hidden"></div>
    <div id="retweet-text-template" class="text"></div>
  </body>

</html>

index.ts

// Copyright 2018 Google LLC.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

import * as functions from 'firebase-functions'
import * as admin from 'firebase-admin'
import * as moment from 'moment'
admin.initializeApp(functions.config().firebase)

type TweetData = {[key: string]: any}
const rtRegex = new RegExp('.*(https://twitter.com/(.+?)/status/(.+?))\\?ref_src=twsrc%5Etfw')

export const ifttt = functions.https.onRequest((req, res) => {
    const body = (req['rawBody'] as Buffer).toString()
    console.log(body)
    const data = parseTweetData(body)

    if (Object.keys(data).length > 0) {
        addToFirestore(data)
        .then(() => {
            res.status(200).end()
        })
        .catch(error => {
            console.error("Couldn't add tweet", error)
            res.status(500).send(error)
        })
    }
    else {
        console.error("Tweet data was invalid")
        res.status(400).end()
    }
})

function parseTweetData(body: string): TweetData {
    const data : TweetData = {}
    body.split('|*|*|').forEach(kvPair => {
        const idx = kvPair.indexOf('=')
        const key = kvPair.slice(0, idx)
        const value = kvPair.slice(idx+1)
        if (key && value) {
            data[key] = value
        }
        else {
            console.error("No key or value?!", 'key', key, 'value', value)
        }
    })

    const search = data.search
    delete data.search
    console.log("Search", search)

    // Parse the tweet id out of linkToTweet
    const pathParts = data.linkToTweet.split('/')
    data.id = pathParts[pathParts.length-1]

    data.linkToTweet = (data.linkToTweet as string).replace(/^http/, 'https')

    // Parse the date text (why won't they just give us a number?)
    data.createdAt = moment(data.createdAt + ' Z', 'MMM DD YYYY at HH:mma Z', 'en').toDate()

    // Parse retweet information
    const matches = rtRegex.exec(data.tweetEmbedCode)
    if (matches) {
        const rtId = matches[3]
        if (data.id !== rtId) {
            data.rtLink = matches[1]
            data.rtUserName = matches[2]
            data.rtId = rtId
        }
    }

    return data
}

function addToFirestore(data: TweetData): Promise<void> {
    // Prevent writing duplicate tweets by checking for prior existence in a transaction
    const doc = admin.firestore().collection('tweets').doc(data.id)
    return admin.firestore().runTransaction(transaction => {
        return transaction.get(doc).then(snapshot => {
            if (! snapshot.exists) {
                console.log("Creating doc for tweet", data.id)
                transaction.set(doc, data)
            }
            else {
                console.log("Tweet already exists", data.id)
            }
        })
    })
}

Hi, I've a small problem, I tried running this and got the following error after giving the "firebase deploy" command:
Error: HTTP Error: 400, This operation is not available for Cloud Firestore projects in Datastore Mode.

Could anyone please help me out? I'm a newbie. I'm trying to use this in firebase project to stream tweets in realtime. Thanks in advance