tbarbugli/stream_twitter_tutorial.md

## stream_twitter_tutorial.md

      
    Raw
  

              stream_twitter_tutorial.md
            
          
    Build a scalable Twitter clone with Django and GetStream.io

In this tutorial we are going to build a Twitter clone using Django and GetStream.io, a hosted API for newsfeed development.
We will show you how easy is to power your newsfeeds with GetStream.io. For brevity we leave out some basic Django-specific code and recommend you refer you to the Github project for the complete runnable source code. At the end of this tutorial we will have a Django app with a profile feed, a timeline feed, support for following users, hashtags and mentions.
I assume that you are familiar with Django. If you're new to Django the [official tutorial] (https://docs.djangoproject.com/en/2.0/intro/) explains it very well.
Bootstrap the Django application

We will use Python 3.6 and Django 2.0, which is the latest major release at the time of writing.
Make sure you have a working Django project before you continue to the next part of the tutorial.
Create the Django app

Let's start by creating a new Django app called stream_twitter
python manage.py startapp stream_twitter
Install stream_django

stream_django provides the GetStream integration for Django, it is built on top of the stream_python client.
pip install stream-django
To enable stream_django you need to add it to your INSTALLED_APPS:
INSTALLED_APPS = (
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'stream_twitter',
    'stream_django'
)
GetStream.io setup

First of all, we need to create an account on GetStream.io. You can signup with Github and it's free for usage below 3 million feed updates per month. Once you've signed up, get your API 'key' and 'secret' from the dashboard and add them to Django's settings:
STREAM_API_KEY = 'my_api_key'
STREAM_API_SECRET = 'my_api_secret'
The models

In this application we will have three different models: users, tweets and follows.
To keep it as simple as possible, we will use Django's contrib.auth User model.
Have a look below at the initial version of the Tweet and Follow models.
from django.db import models


class Tweet(models.Model):
    user = models.ForeignKey('auth.User', on_delete=models.CASCADE)
    text = models.CharField(max_length=160)
    created_at = models.DateTimeField(auto_now_add=True)


class Follow(models.Model):
    user = models.ForeignKey('auth.User', related_name='friends', on_delete=models.CASCADE)
    target = models.ForeignKey('auth.User', related_name='followers', on_delete=models.CASCADE)
    created_at = models.DateTimeField(auto_now_add=True)

    class Meta:
        unique_together = ('user', 'target')
Now, let's create the schema migrations and apply them.
python manage.py makemigrations stream_twitter
python manage.py migrate
Let's also setup the view to add tweets.
from django.views.generic.edit import CreateView
from stream_twitter.models import Tweet


class TweetView(CreateView):
    model = Tweet
    fields = ['text']

    def form_valid(self, form):
        form.instance.user = self.request.user
        return super(Tweet, self).form_valid(form)
And of course add it to urls.py
from django.contrib import admin
from django.contrib.auth.decorators import login_required
from django.urls import path

from stream_twitter import views

urlpatterns = [
    path('admin/', admin.site.urls),
    path('timeline/', login_required(views.TimelineView.as_view()), name='timeline'),
    re_path(r'^user/(?P<username>.+)/', views.UserView.as_view(), name='user_feed')
]


Now that we have the view setup for creating tweets we can move on to setting up the newsfeed.
Model integration

We want the tweets to be stored in the authors feed. This is when we start using the stream_django integration. We can configure the Tweet model so that it will syncronise automatically with feeds.
To do this we need to make Tweet a subclass of stream_django.activity.Activity
So let's do that by modifying the model we defined earlier in stream_twitter.models:
from stream_django import activity


class Tweet(activity.Activity, models.Model):
    ...
From now on, new tweets will be added to the user feed of the author and to the feeds of all his followers. The same applies to deleting a tweet.
So, let's give it a try using Django's shell:
python manage.py shell
from stream_twitter.models import Tweet
from django.contrib.auth.models import User

user, _created = User.objects.get_or_create(username='tester')

Tweet.objects.create(
    user=user,
    text='Go Cows!'
)
We've now created our first Tweet, and in turn added an Activity to a Feed via the Stream API. By default, stream-django creates and adds the Activity to a feed named after the 'actor' property. This can be customized by overriding the _attr functions inherited from the stream_django.activity.Activity mixin on the Django Model.
Now, this is the first time we talk about Activities and Feeds, let's take a moment to define what an activity is.
An activity is an object that contains the information about an action that is performed by someone involving an object. When you write data to GetStream's feeds, you send this data in the form of activities. The simplest activity is made by these three fields: actor, object and verb. For example: Tommaso tweets: 'Go cows!'
GetStream.io API's allow you to store additional fields in your feeds, as you can see from the documentation here.
We can verify that the Activity was added by using the Data Browser in GetStream.io's Dashboard.
In this example you can determine the feed name by inspecting the activity_actor property:
>>> t.activity_actor
'auth.User:1'

User feed

So now that every tweet gets stored in the author's feed, we are going to add a view that reads them.
from django.contrib.auth.models import User
from django.shortcuts import render, get_object_or_404
from django.views.generic import DetailView

from stream_django.enrich import Enrich
from stream_django.feed_manager import feed_manager

enricher = Enrich()


class UserView(DetailView):
    model = User
    template_name = 'stream_twitter/user.html'

    def get_object(self):
        return self.get_queryset().get(username=self.kwargs['username'])

    def get_context_data(self, object):
        user = self.object
        feeds = feed_manager.get_user_feed(user.id)
        activities = feeds.get()['results']
        activities = enricher.enrich_activities(activities)

        return {
            'activities': activities,
            'user': user,
            'login_user': self.request.user
        }
There are two new things that I should explain: the feed manager and the enricher. As the name suggests, the feed manager takes care of managing the different feeds involved in your app. In this case we ask the feed manager to give us the user feed for the current user.
We learned before that data is stored in feeds in form of activities. This is what a tweet looks like when we read it from GetStream.io:
[{
    'actor': 'auth.User:1',
    'object': 'stream_twitter.Tweet:1',
    'verb': 'tweet',
    ... other fields ...
}]

As you can see, 'object' field does not contain the tweet itself but a reference to that (the same applies to the 'actor' field). The enricher replaces these references with model instances.
Templating feeds

django_stream comes with a templatetag that helps you to show the content from feeds in your templates. This can get quite complex as you add different kinds of activities in your feeds.
Here is a very minimal tweets.html template:
{% load activity_tags %}

{% for activity in activities %}
    {% render_activity activity %}
{% endfor %}
The first time you run this, Django will complain that 'activity/tweet.html' is missing. That's because the render_activity templatetag inspects the activity object and loads the template based on the verb.
Because the verb in this case is 'tweet', it will look for tweet.html in activity path. The templatetag accepts extra options to make your templates as re-usable as possible, see here for the templatetag documentation.
Feed Follow

As a next step we're going to add the ability to follow users to the application. To do this we create a view that creates Follow objects.
from django.views.generic.edit import CreateView, DeleteView
from django.urls import reverse_lazy
from stream_twitter.forms import FollowForm
from stream_twitter.models import Follow

class FollowView(CreateView):
    form_class = FollowForm
    model = Follow
    success_url = reverse_lazy('timeline_feed')

    def form_valid(self, form):
        form.instance.user = self.request.user
        return super(FollowView, self).form_valid(form)


class UnfollowView(DeleteView):
    model = Follow
    success_url = reverse_lazy('timeline_feed')

    def get_object(self):
        target_id = self.kwargs['target_id']
        return self.get_queryset().get(target__id=target_id)
Now we can use Django's signals to perform follow/unfollow requests on GetStream APIs.
def unfollow_feed(sender, instance, **kwargs):
    feed_manager.unfollow_user(instance.user_id, instance.target_id)


def follow_feed(sender, instance, created, **kwargs):
    if created:
        feed_manager.follow_user(instance.user_id, instance.target_id)


post_save.connect(follow_feed, sender=Follow)
post_delete.connect(unfollow_feed, sender=Follow)
Timeline view (AKA flat feed)

The hardest part for a scalable Twitter clone is the feed showing the tweets from people you follow. This is commonly called the timeline view or newsfeed. The code below shows the timeline.
from django.views.generic import TemplateView

class TimelineView(TemplateView):
    template_name = 'stream_twitter/timeline.html'

    def get_context_data(self):
        context = super(TimelineView, self).get_context_data()

        feeds = feed_manager.get_news_feeds(self.request.user.id)
        activities = feeds.get('timeline').get()['results']
        enriched_activities = enricher.enrich_activities(activities)

        context['activities'] = enriched_activities

        return context
The code looks very similar to the code of profile_feed. The main difference is that we use feed manager's get_news_feeds. By default, GetStream.io apps and stream_django come with two newsfeeds predefined: flat and aggregated feeds. When you use feed_manager.get_news_feeds, you get a dictionary with flat and aggregated feeds. Since we are not going to use aggregated feeds, we can adjust Django settings:
STREAM_NEWS_FEEDS = dict(flat='flat')
Adding activities

Let's modify the TimelineView to include a form that will accept new tweets:
from django.views.generic.edit import CreateView


class TimelineView(CreateView):
    fields= ['text']
    model = Tweet
    success_url = reverse_lazy('timeline_feed')
    template_name = 'stream_twitter/timeline.html'

    def form_valid(self, form):
        form.instance.user = self.request.user
        return super(TimelineView, self).form_valid(form)

    def get_context_data(self, form=None):
        context = super(TimelineView, self).get_context_data()

        feeds = feed_manager.get_news_feeds(self.request.user.id)
        activities = feeds.get('timeline').get()['results']
        enriched_activities = enricher.enrich_activities(activities)

        context['activities'] = enriched_activities
        context['login_user'] = self.request.user
        context['hashtags'] = Hashtag.objects.order_by('-occurrences')

        return context
Hashtags feeds

We want Twitter style hashtags to work as well. Doing this is surprisingly easy. Let's first open GetStream.io dashboard and create the 'hashtag' feed type. (Note: By default getstream.io will setup user, flat, aggregated and notification feeds. If you more feeds you need ot configure them in the dashboard)

from django.template.defaultfilters import slugify


class Tweet(activity.Activity, models.Model):

    def parse_hashtags(self):
        return [slugify(i) for i in self.text.split() if i.startswith("#")]
Now that we have parsed the hashtags, we could loop over them and publish the same activity to every hashtag feed. Fortunately there's a shortcut though. GetStream allows you to send a copy of an activity to many feeds with a single request.
To do this, we only need to implement the activity_notify method to the Twitter model we created previously:
from stream_django.feed_manager import feed_manager


class Tweet(activity.Activity, models.Model):

    @property
    def activity_notify(self):
        targets = []
        for hashtag in self.parse_hashtags():
            targets.append(feed_manager.get_feed('hashtag', hashtag))
        return targets
From now on, activities will be stored to hashtags feeds as well. For instance, the feed 'hashtag:Django' will contain all tweets with '#Django' in it.
Again the view code looks really similar to the other views.
from django.views.generic import TemplateView

from stream_django.enrich import Enrich
from stream_django.feed_manager import feed_manager


class HashtagView(TemplateView):
    template_name = 'stream_twitter/hashtag.html'

    def get_context_data(self, hashtag):
        context = super(TemplateView, self).get_context_data()

        hashtag = hashtag.lower()
        feed = feed_manager.get_feed('user', f'hash_{hashtag}')
        activities = feed.get(limit=25)['results']

        context['hashtag'] = hashtag
        context['activities'] = enricher.enrich_activities(activities)

        return context
Mentions

Now that we found out about the activity_notify property, it only takes a bunch of extra lines of code to add user mentions.
class Tweet(activity.Activity, models.Model):

    def parse_mentions(self):
        mentions = [slugify(i) for i in self.text.split() if i.startswith("@")]
        return User.objects.filter(username__in=mentions)

    @property
    def activity_notify(self):
        targets = []
        for hashtag in self.parse_hashtags():
            targets.append(feed_manager.get_feed('hashtag', hashtag))
        for user in self.parse_mentions():
            targets.append(feed_manager.get_news_feeds(user.id)['flat'])
        return targets
Conclusion

Congratulations, you've reached the end of this tutorial. This article showed you how easy it is to build scalable newsfeeds with Django and GetStream.io. It took us just 100 LoC and (I hope) less than one hour to get this far.
You can find the code from this tutorial and the fully functional application on GitHub. The application is also running and can be tested here. I hope you found this interesting and useful and I'd be glad to answer all of your questions.
If you're new to Django or GetStream.io, I highly recommend the official django tutorial and the getstream.io getting started.