BlogPart2

BlogPart2.ipynb

BlogPart2.py

#!/usr/bin/env python
# coding: utf-8

# # Part 2

# Hi there again! In the last blog, we were able to get some results. But we noticed that our program spent majority of its time requesting results from the API. As we are spending a lot of time in requests, it is good to have them done in parallel. So, let's leverage **threading** in Python to spawn multiple threads at the same time. Thankfully, as our threads are not interdependant on each other, we can be chill about concurrancy issues. Keep in mind that threading does not mean that our program will run on multiple cores.

# In[1]:


import concurrent.futures, threading


# Another improvement that we can do is instead of sending all of our information data again and again ( ```response = requests.request("GET", url = browseQuotesURL, headers = self.headers)``` ), we create a session. This session will be remembered by our object and the next time, we just have to make a direct request. This will be also helpful in the future if we want to get the url for ticket purchase.

# In[2]:


class finder:
    
    def __init__(self, originCountry = "DE", currency = "EUR", locale = "en-US", rootURL="https://skyscanner-skyscanner-flight-search-v1.p.rapidapi.com"):
        self.currency = currency
        self.locale =  locale
        self.rootURL = rootURL
        self.originCountry = originCountry
        self.airports = {}
        
    def setHeaders(self, headers):
        self.headers =  headers
        self.createSession()

    # Create a session
    def createSession(self):
        self.session = requests.Session() 
        self.session.headers.update(self.headers)
        return self.session
        
    def browseQuotes(self, source, destination, date):
        quoteRequestPath = "/apiservices/browsequotes/v1.0/"
        browseQuotesURL = self.rootURL + quoteRequestPath + self.originCountry + "/" + self.currency + "/" + self.locale + "/" + source + "/" + destination + "/" + date.strftime("%Y-%m-%d")
        # Use the same session to request again and again
        response = self.session.get(browseQuotesURL)
        resultJSON = json.loads(response.text)
        self.printResult(resultJSON,date)
        
    # A bit more elegant print
    def printResult(self, resultJSON,date):
        if("Quotes" in resultJSON):
            for Places in resultJSON["Places"]:
                self.airports[Places["PlaceId"]] = Places["Name"] 
            for Quotes in resultJSON["Quotes"]:
                source = Quotes["OutboundLeg"]["OriginId"]
                dest = Quotes["OutboundLeg"]["DestinationId"]
                print(date.strftime("%d-%b %a") + " | " + "%s  --> %s"%(self.airports[source],self.airports[dest]) + " | " + "%s EUR" %Quotes["MinPrice"])


# As this is a new Jupyter Notebook, let's dump all our variables from previous code here

# In[3]:


import requests, json, timeit, time, datetime, dateutil, osmapi
import calendar
import pandas as pd
import time

rootURL = "https://skyscanner-skyscanner-flight-search-v1.p.rapidapi.com"
originCountry = "DE"
currancy = "EUR"
locale = "en-US"

source_begin_date = "2020-01-18"
source_end_date =  "2020-01-24"  
daterange_source = pd.date_range(source_begin_date, source_end_date)
airports = { }
source_array = {"BERL-sky"} 
destination_array = {"MAD-sky", "BCN-sky", "SVQ-sky", "VLC-sky"}


headers = {
    'x-rapidapi-host': "skyscanner-skyscanner-flight-search-v1.p.rapidapi.com",
    'x-rapidapi-key': "ae922034c6mshbd47a2c270cbe96p127c54jsnfec4819a7799"
    }


# Now here comes the multi-threading part. The concurrent features library hands it for us. Using threadpoolexecutor as a wrapper, we just submit our task and its parameters to a pool of threads. The executor automatically schedules them as they arrive and they get executed in paralllel. Easiest multithreading ever! 

# In[4]:


cheapest_flight_finder2 = finder()
cheapest_flight_finder2.setHeaders(headers)

function_start = time.time()

with concurrent.futures.ThreadPoolExecutor(max_workers=32) as executor:
    for single_date in daterange_source:
        for destination in destination_array:
            for source in source_array:
                request_start = time.time()
                executor.submit(cheapest_flight_finder2.browseQuotes,source, destination,single_date)

print("\nBenchmark Stats :")
print("Time spent in program: %f seconds"%(time.time()-function_start))


# Whoa, thats a lot faster!  But just how fast? Lets compare the same code using single thread.

# In[5]:



function_start = time.time()

for single_date in daterange_source:
    for destination in destination_array:
        for source in source_array:
            request_start = time.time()
            cheapest_flight_finder2.browseQuotes(source, destination,single_date)

print("\nBenchmark Stats :")
print("Time spent in program: %f seconds"%(time.time()-function_start))


# Thats a tremendous imporvement. Our program is almost 7-8 times faster! You have to do a bit of trial and error for  ```max_thread``` value in order to get the optimum number of threads. For me 32 was the best.
# 
# 
# Also, did you notice the text glitches in our multi-thread version? Our threads are independant, so it could happen that two threads want to use print at the exact same time. Thus, somehow, the next line is not inserted properly. However, if our threads needed a synchronisation, we would have to deal with a lot of stuff like semaphores and mutexs. But, we are safe for now! 
# 
# 
# Now that we have finished our thread *thread* , we move back to our application

# ### Databases and Docker

# You can see that we are generating lots of data for a single request. Now we also have to operate on it to find the cheapest trip.We can use python's data structures or even pandas. But why not use what is actually used in real world products? Databases!
# 
# Databases are organized way of holding data. Think of something like an Excel Spreadsheet, but for easy access from other programs. Like Pokemons, there are 100s of differnt datasbases. But there 2 major types: Relational ones (SQL) and Non relational ones (NoSQL). If you want to know more about their differences and how they function, [this](https://clockwise.software/blog/relational-vs-non-relational-databases-advantages-and-disadvantages/) blog has explained it in much detail. But in our case we are working with JSON anyway and MongoDB is just meant for that kind of data! I could go on and on about why we chose a perticular database. For now, let's use MonogoDB. 
# 
# We can install it like a normal program using exe or deb packages. But, let's consider a practical scenario. If this reaches production, it will likeley be running on a server. And if it , god forbids, gets popular! ; we might me getting a tons of different requests per second from different parts of the globe. In this case, if our code fails, everything will just stop.
# 
# The clever way of overcoming this is using microservices, i.e. using Kubernetes and Docker. Now this is definately a topic of another blog. But now, to explain you how easy it is to set up a service, let's use mongodb's official docker container. Follow the installation process for [docker](https://docs.docker.com/install/). Once you have done that, just run following command in the terminal:
# 
# ```docker run --name eurotrip-planner-mongo mongo:latest```
# 
# This will pull the latest mongodb image and all it's required parts, build a contianer and then start a mongodb server! All in one line! You can check if it is working or not using [MongoDB client application](https://www.mongodb.com/products/compass). Our setup is done!

# Now, I am cheating a bit, but I already wrote wrapper drvier module for MongoDB. You can read the docs on GitHub but it is fairly easy to understand. I am just going to use it directly here.
# 
# Our MongoDB paramerters are here. We create 2 different collections or tables for our Incoming and Outgoing flights.

# In[6]:


import wrapymongo
authdb = "admin"
monogdbport = "27017"
host = "localhost"
link = "mongodb://" + host + ":" + monogdbport
database =  "SkyScanner"
outgoingTable = "Outgoing"
incomingTable = "Incoming"
placesTable = "Places"


# I am adding a function that acts like a template maker that gets an object instance of our MongoDB class.

# In[7]:


# Function to make wrapymongo object
def makeObject(link,dbName = "SkyScanner", dbCollection="test"):
    mdbobject = wrapymongo.driver(link)
    mdbobject.defineDB(dbName)
    mdbobject.defineCollection(dbCollection)
    return mdbobject


# We instantiate our objects and clear their contents if they have any.

# In[8]:


mdbOutgoing = makeObject(link,dbName = database,dbCollection = outgoingTable)
mdbPlaces = makeObject(link,dbName = database,dbCollection = placesTable)
mdbIncoming = makeObject(link,dbName = database,dbCollection = incomingTable)

mdbOutgoing.dropCollection()
mdbPlaces.dropCollection()
mdbIncoming.dropCollection()


# And we initalize our arrays as usual. I saved our ```finder``` class in another file called ```flightfinder.py``` to make it easier to work with.

# In[9]:


import flightfinder as ff

airports = { }


outgoing_flight_finder = ff.finder()
outgoing_flight_finder.setHeaders(headers)

incoming_flight_finder = ff.finder()
incoming_flight_finder.setHeaders(headers)

source_array = {"BERL-sky"} 
destination_array = {"MAD-sky", "BCN-sky", "SVQ-sky", "VLC-sky"}


# Let it rip!

# In[10]:


processing_start = time.time()

with concurrent.futures.ThreadPoolExecutor(max_workers=32) as executor:
    for single_date in daterange_source:
        for destination in destination_array:
            for source in source_array:
                request_start = time.time()
                executor.submit(outgoing_flight_finder.browseQuotes,source, destination,single_date)

outgoingQuotes = outgoing_flight_finder.getQuotes()

for quote in outgoingQuotes:
    mdbOutgoing.insertRecords(quote)                

airports.update(outgoing_flight_finder.getAirports())


# At this point, you can go on the previously installed mongodb client application and see how the database has updated with a tablename _outgoing_ and all the entries related to it! We are just adding all the quotes, one by one, in the database. Now, let's do the same for the "coming back" part of the trip. 

# In[11]:


destination_begin_date = "2020-01-24"
destination_end_date =  "2020-01-30"  
daterange_destination = pd.date_range(destination_begin_date, destination_end_date)


# In[12]:


# We reverse the arrays here
with concurrent.futures.ThreadPoolExecutor(max_workers=32) as executor:
    for single_date in daterange_destination:
        for destination in source_array:
            for source in destination_array:
                request_start = time.time()
                executor.submit(incoming_flight_finder.browseQuotes,source, destination,single_date)

incomingQuotes = incoming_flight_finder.getQuotes()

for quote in incomingQuotes:
    mdbIncoming.insertRecords(quote)      

airports.update(incoming_flight_finder.getAirports())


# At this point we have all we need stored in the database. We just have to make sense of all the data. So first, let's get top 20 cheapest entries from each of the colllection. We just make a query for ```sortRecords``` with ```MinPrice``` as key and sorted from lowest to highest (indicated by 1)

# In[13]:


# Sort both dbs by cheapest
cheapestOutgoingFlights = {}
cheapestOutgoingFlights = mdbOutgoing.sortRecords([('MinPrice', 1)], 20)

cheapestIncomingFlights = {}
cheapestIncomingFlights = mdbIncoming.sortRecords([('MinPrice', 1)], 20)


# To get the cheapest trip, we check for all possible combinations between incoming and outgoing quotes. Let's just combine the data first and print it.

# In[14]:


finalListElement = {}
finalList = []
for incomingQuotes in cheapestIncomingFlights:
    for outgoingQuotes in cheapestOutgoingFlights:
        finalListElement = {}
        finalListElement["TotalPrice"] = incomingQuotes["MinPrice"] + outgoingQuotes["MinPrice"]
        finalListElement["TakeOff1"] =  airports[outgoingQuotes["OutboundLeg"]["OriginId"]]              
        finalListElement["Land1"] =  airports[outgoingQuotes["OutboundLeg"]["DestinationId"]]  
        finalListElement["TakeOff2"] =  airports[incomingQuotes["OutboundLeg"]["OriginId"]]  
        finalListElement["Land2"] =  airports[incomingQuotes["OutboundLeg"]["DestinationId"]] 
        finalListElement["Date1"] = outgoingQuotes["OutboundLeg"]["DepartureDate"]
        finalListElement["Date2"] = incomingQuotes["OutboundLeg"]["DepartureDate"]
        finalList.append(finalListElement)


# In[15]:


print(finalList[:10])


# Awesome!! Now we just leverage our mongodb to put all these records in the database and then sort them by their total cost

# In[16]:


mdbFinal = makeObject(link, dbName=database, dbCollection="FinalDatabase")
mdbFinal.dropCollection()
mdbFinal.insertRecords(finalList)

print("The Top ten cheapest flights are:")
topQuotes = mdbFinal.sortRecords([('TotalPrice', 1)], 10)
for quote in topQuotes:
    print("\n*****\nOnwards: " + quote["Date1"] + " " + quote["TakeOff1"] + " --> " + quote["Land1"] + " \nReturn: " +
          quote["Date2"] + " " + quote["TakeOff2"] + " --> " + quote["Land2"] + " \n \t   | " + "%s EUR" % quote["TotalPrice"])



# Wohoo! We have finally received what we were looking for. Now I have some cool options to choose my trip from.
# 
# If you have made it until this point, then congratulations!! You have conqured the mountain and the summit is yours! Now its time for retrospection. Was this really necessary, or you could have just flown on the top of the using helicopter? (Wiz: could you just have used Google Flights over and over to do this?) Yes!! Is our path (solution) the most elegent and the easiest of all? Of course not! Does it even make sense to use Docker and MongoDB for such small tasks? Mostly not! Is it overengineered? You bet! Is it at least useful? Mostly not as we don't even get the timings!
# 
# But then, even in this toy problem, we went through major steps in software designing. We developed a real, scalable system which can give us some results. It may seem useless to just get the chepest flights, but we can easily extend it to any number of parameters that we want. We could sort using dates, airlines, stopovers and create a real product. This was somewhat real problem, and we found a real solution. I think thats a win!

# The code of this project is available on GitHub. If you would like to contribute towards it, just send me a pull request. Also, if you think this project can be developed into a real website which a lot of people would like to use it for, hit me up!!