Skip to content

Instantly share code, notes, and snippets.

@moshez

moshez/Twisted-Tutorial.ipynb Secret

Last active July 8, 2018 11:18
Show Gist options
  • Star 3 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save moshez/24aab9ba211181dc31f1d780f8b132aa to your computer and use it in GitHub Desktop.
Save moshez/24aab9ba211181dc31f1d780f8b132aa to your computer and use it in GitHub Desktop.
Download ZIP
Display the source blob
Display the rendered blob
Raw
Twisted-Tutorial.ipynb
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Creating and Consuming Modern Web Services with Twisted\n",
"\n",
"### Assumptions\n",
"\n",
"* Docker is installed (i.e., `docker images` from command line succeeds)\n",
"* You have internet connection (i.e., `docker pull scratch` from command line succeds)\n",
"\n",
"## Running the container\n",
"\n",
"```\n",
"docker run -it -p 127.0.0.1:8888:8888 -p 127.0.0.1:9090:9090 moshez/greatredspot\n",
"```\n",
"\n",
"The 8888 port is the Jupyter notebook port.\n",
"The 9090 port is the port of the application we will implement\n",
"\n",
"Note that since it is running ephemerally, it is a good idea to File -> Download As -> .ipynb\n",
"occasionaly.\n",
"\n",
"## Alternate Reality\n",
"\n",
"Welcome to a different universe than the one you are used to!\n",
"In this universe, multiplication and negation (but not addition)\n",
"are computationally-heavy operations.\n",
"Most computers cannot perform them.\n",
"Instead, they call into heavy-duty web services,\n",
"which churn and churn,\n",
"and finally return the answer.\n",
"The most popular of these is [calculate.solutions](https://calculate.solutions).\n",
"However, it is expensive to test against calculate.solutions.\n",
"For this tutorial, you will be running a simulation inside the same container.\n",
"It can multiply and negate -- but it does take a bit of time to do so,\n",
"after all,\n",
"it has only your laptop to run on.\n",
"\n",
"## Starting up is hard to do\n",
"\n",
"The first thing we need to do is to set up the klein application. If what follows seems a bit arcane, there is no need to worry: it is only needed to integrate Klein properly with Jupyter. Every way of deploying a Klein application will need, eventually, some sort of \"rubber meets road\" part, where we connect the application to the network. There will always be some complicating consideration: which low-level event loop to use? What kind of sockets to listen on? Here is the specific way we do it for this demo on Jupyter.\n",
"\n",
"Another compromise done here is that the entire initialization is in one cell. From a pedagogical perspective, it would have been better to split it. However, this way it is much easier to hit \"restart kernel\" and then re-run the initialization. \n",
"\n",
"So what does happen here? (It is safe to skip this part) Jupyter is running on top of the Tornado event loop. Twisted can run its \"reactor\", the Twisted event loop, on top of Tornado's. This is not the default -- the default is the native-Twisted event loop. The first thing we do is install the Tornado reactor. It is important this is the first thing to be done: Twisted will automatically install the default reactor as soon as a reactor is requested, if one is not already installed.\n",
"\n",
"Then, we created a server endpoint which listens on port 9090. We get the global Klein resource, wrap it in a Site object and connect it to the endpoint. Note that for production use of Klein, it is best not to use the global resource. However, for a quick prototype, it is extremely useful."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from tornado.platform.twisted import install\n",
"reactor = install()\n",
"\n",
"from twisted.web.server import Site\n",
"from twisted.internet import endpoints\n",
"from klein import route, resource\n",
"\n",
"description = \"tcp:9090\"\n",
"\n",
"ep = endpoints.serverFromString(reactor, description)\n",
"s = Site(resource())\n",
"ep.listen(s)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Static Routes\n",
"\n",
"As is traditional, the first thing to be done in any programming environment is to greet. This allows both showing what is the \"minimum program\" as well as have an easy way to demonstrate the environment is working. Klein uses decorators to indicate routing. We put a resource at the root which greets with a constant string."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"from klein import route\n",
"@route('/')\n",
"def home(request):\n",
" return 'Hello, everybody!'"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Click on [this link](http://localhost:9090) to see the results.\n",
"\n",
"Now, we change this to a different, yet still constant string.\n",
"\n",
"> Note: The key that Klein uses to decide to replace a route is actually the route's *name*, rather than its *path*. So, if you initially define a `/` route called `home`, any replacement for `/` in that same process must also be a function named `home`. In normal use of Klein (like, in a Python program you write and then run), you would not update a route dynamically while a site is running, so while this is probably a bug, it's not one you will likely run into outside the scope of this tutorial.\n",
"> \n",
"> If you see a route refusing to update for this or any other reason, just restart your IPython kernel and re-evaluate your cells from the top again."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"from klein import route\n",
"@route('/')\n",
"def home(request):\n",
" return 'Hello, PyCon!'"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Click on [the link](http://localhost:9090) to see the change.\n",
"\n",
"## Dynamic Resources\n",
"\n",
"Flask is built on top of the Werkzeug URL router.\n",
"Klein (a Twisted Flask) is built on the same router.\n",
"This allows defining resources which parametrize on the URL."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"@route('/greet/<user>')\n",
"def greet(request, user):\n",
" return 'Hello {}!'.format(user.upper())"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Click on [the link](http://localhost:9090/greet/say-my-name) to see a personalized greeting. Edit the URL to change the personalization."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Reversing a String -- Exercise"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"@route('/reverse/<content>')\n",
"def reverse(request, content):\n",
" # Fix me\n",
" return content"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Click on [the link](http://localhost:9090/reverse/desrever) to check whether you reversed correctly."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Using Web APIs\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"from twisted.internet import defer"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"@route('/immediate')\n",
"async def immediate(request):\n",
" result = await defer.succeed('Right now')\n",
" return \"It's ready {}\".format(result)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's see [if it's ready](http://localhost:9090/immediate)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"inputHidden": false,
"outputHidden": false
},
"outputs": [],
"source": [
"from twisted.internet import task\n",
"import operator\n",
"\n",
"def add(a, b):\n",
" return task.deferLater(reactor, 1, operator.add, a, b)\n",
"\n",
"@route(\"/apiclient/add/<int:a>/<int:b>\")\n",
"async def addroute(request, a, b):\n",
" deferred = add(a, b)\n",
" number = str(await deferred)\n",
" return \"Your result is: \" + number"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"We can [calculate sums](http://localhost:9090/apiclient/add/3/4) now.\n",
"Notice the artifically induced delay -- though addition is fast,\n",
"we delay it by a second to get to see how slow arithmetic feels like."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"import json\n",
"import treq\n",
"\n",
"async def subtract(a, b):\n",
" data = json.dumps([b]).encode('utf-8')\n",
" response = await treq.post('http://localhost:1234/negate', data=data)\n",
" content = await response.content()\n",
" minus_b = json.loads(content.decode('utf-8'))\n",
" return a + minus_b\n",
"\n",
"@route('/subtract/<int:a>/<int:b>')\n",
"async def route_subtract(request, a, b):\n",
" return 'Answer: {}'.format(await subtract(a, b))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Canonical [subtraction](http://localhost:9090/subtract/7/3)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Exercise -- Calculate an affine function"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"import json\n",
"import treq\n",
"\n",
"## Calculate a*b+c\n",
"async def affine(a, b, c):\n",
" # Use http://localhost:1234/multiply\n",
" # It expects a JSON array of two\n",
" content = await response.content()\n",
" a_times_b = json.loads(content.decode('utf-8'))\n",
" return 0\n",
"\n",
"@route('/affine/<int:a>/<int:b>/<int:c>')\n",
"async def route_subtract(request, a, b, c):\n",
" return 'Answer: {}'.format(await affine(a, b, c))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Test that you get 21 [here](http://localhost:9090/affine/4/5/1)"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"## Environment Part 2\n",
"\n",
"We had some fun greeting, reversing, and calculating solutions but what is really going on here?\n",
"\n",
"* Event loop\n",
"* Tornado\n",
"* Twisted reactor\n",
"* Jupyter Kernels\n",
"* Twisted-on-Tornado\n",
"\n",
"Annotated first cell:\n",
"\n",
"```\n",
"from tornado.platform.twisted import install\n",
"reactor = install()\n",
"```\n",
"\n",
"Installing the reactor is best done as early as possible.\n",
"Yes, even before other imports!\n",
"For historical reasons, importing `twisted.internet.reactor`\n",
"automatically installs the default reactor if none is installed.\n",
"\n",
"```\n",
"from twisted.web.server import Site\n",
"from twisted.internet import endpoints\n",
"from klein import route, resource\n",
"```\n",
"\n",
"After installing the reactor, we can safely import everything we need.\n",
"\n",
"```\n",
"description = \"tcp:9090\"\n",
"ep = endpoints.serverFromString(reactor, description)\n",
"```\n",
"\n",
"Endpoints are a mini-language describing how to reach network services.\n",
"The `tcp:9090` indicates listening for TCP connections on port 9090.\n",
"By default, it listens on all addresses.\n",
"\n",
"```\n",
"s = Site(resource())\n",
"```\n",
"\n",
"A `Site` wraps a `Resource` with something that can be served over HTTP.\n",
"Here, we get the default Klein resource.\n",
"Usually, it is better to use custom Klein resources for production.\n",
"The default one is great for prototyping.\n",
"\n",
"```\n",
"ep.listen(s)\n",
"```\n",
"\n",
"Finally, we connect the site to a TCP listener.\n",
"\n",
"## Deferreds\n",
"\n",
"So far, to transform deferreds into values we used `await`.\n",
"There is one other primitive operation that is valuable -- `gatherResults`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Naive implementation of a-b-c\n",
"async def negate(num):\n",
" data = json.dumps([num]).encode('utf-8')\n",
" response = await treq.post('http://localhost:1234/negate', data=data)\n",
" content = await response.content()\n",
" return json.loads(content.decode('utf-8'))\n",
"\n",
"@route('/slowsubtract/<int:a>/<int:b>/<int:c>')\n",
"async def slowsubtract(request, a, b, c):\n",
" import time\n",
" before = time.time()\n",
" minus_b = await negate(b)\n",
" minus_c = await negate(c)\n",
" res = a + minus_b + minus_c\n",
" elapsed = int(time.time() - before)\n",
" return \"Answer: {} (elapsed {})\".format(res, elapsed)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Click on [this](http://localhost:9090/slowsubtract/10/2/3) first and settle in with a nice cup of tea.\n",
"\n",
"This is a typical problem in real life --\n",
"needing two unrelated inputs in order to compose an answer.\n",
"In fact, micro-service architectures make it more frequent.\n",
"\n",
"The fix is to send both requests at the same time --\n",
"and double the value we get from waiting."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"@route('/fastsubtract/<int:a>/<int:b>/<int:c>')\n",
"async def fastsubtract(request, a, b, c):\n",
" import time\n",
" before = time.time()\n",
" d_minus_b = defer.ensureDeferred(negate(b))\n",
" d_minus_c = defer.ensureDeferred(negate(c))\n",
" minus_b, minus_c = await defer.gatherResults([d_minus_b, d_minus_c])\n",
" res = a + minus_b + minus_c\n",
" elapsed = int(time.time() - before)\n",
" return \"Answer: {} (elapsed {})\".format(res, elapsed)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"With [this](http://localhost:9090/fastsubtract/10/2/3) we barely have time for a sip!"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Exercise -- compute the squared length of the hypotenuse"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"# Reminder: c**2 = a*a + b*b\n",
"@route('/sqhype/<int:a>/<int:b>')\n",
"async def sqhype(request, a, b):\n",
" ## ??\n",
" pass\n",
"\n",
"@route('/fastsqhype/<int:a>/<int:b>')\n",
"async def fastsqhype(request, a, b):\n",
" ## ??\n",
" pass"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Both [this](http://localhost:9090/sqhype/3/4/) and [this](http://localhost:9090/fastsqhype/3/4/) should result in 25 -- but the second should be quite a bit faster. Show elapsed time for bonus points!"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Digression -- overwhelming services\n",
"\n",
"The techniques we introduced above can easily overwhelm a service.\n",
"Twisted has been known to be fast enough to take out routers.\n",
"While a great testament to Twisted's speed, this is suboptimal behavior.\n",
"\n",
"The solution is not to slow Twisted down --\n",
"it is to be explicit in how many concurrent requests we want a server to sustain."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import time\n",
"\n",
"mult_control = defer.DeferredSemaphore(2)\n",
"\n",
"async def limitedmultiply(a, b):\n",
" data = json.dumps([a, b]).encode('utf-8')\n",
" response = await mult_control.run(treq.post, 'http://localhost:1234/multiply', data=data)\n",
" content = await response.content()\n",
" res = json.loads(content.decode('utf-8'))\n",
" return res\n",
"\n",
"@route('/3dsqnorm/<int:a>/<int:b>/<int:c>')\n",
"async def sqnorm(request, a, b, c):\n",
" before = time.time()\n",
" dlist = [defer.ensureDeferred(limitedmultiply(x, x))\n",
" for x in (a, b, c)]\n",
" reslist = await defer.gatherResults(dlist)\n",
" result = sum(reslist)\n",
" elapsed = int(time.time() - before)\n",
" return \"Answer: {} (elapsed {})\".format(result, elapsed)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The [result](http://localhost:9090/3dsqnorm/1/2/3) comes after 10 seconds: not 5, since it makes sure not to overwhelm the service with more than two concurrent requests."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Crowd-sourced Sums\n",
"\n",
"Let's let people give us interesting sums!"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"import cgi\n",
"\n",
"feed = []\n",
"waiting = []\n",
"\n",
"@route('/sum/add/<int:a>/<int:b>')\n",
"def addsum(request, a, b):\n",
" sum = a + b\n",
" res = (a, b, sum)\n",
" if waiting:\n",
" waiting.pop(0).callback(res)\n",
" else:\n",
" feed.append(res)\n",
" return 'Thanks'\n",
"\n",
"@route('/sum')\n",
"async def getsum(request):\n",
" if feed:\n",
" res = feed.pop(0)\n",
" else:\n",
" d = defer.Deferred()\n",
" waiting.append(d)\n",
" res = await d\n",
" return '{} + {} = {}'.format(*res)\n",
"\n",
"@route('/sum/admin')\n",
"def adminfact(request):\n",
" return cgi.escape(repr(dict(waiting=waiting, feed=feed)))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can use [this](http://localhost:9090/sum/add/3/7) to add a sum, [this](http://localhost:9090/sum/) to get it, and [this](http://localhost:9090/sum/admin) to watch the state of the queue."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Exercise -- give a sum, get a sum"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"@route('/sum/tit-for-tat/<string:a>')\n",
"async def tftsum(request, a):\n",
" # Make sure not to get your own sum!\n",
" pass"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Caching\n",
"\n",
"Twisted, here, runs in one process.\n",
"This allows niceties like the sum relayer above.\n",
"It also allows simple caching."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"from twisted.python import failure\n",
"\n",
"negations = {}\n",
"progress = {}\n",
"\n",
"async def cachednegate(a):\n",
" if a in negations:\n",
" return negations[a]\n",
" if a in progress:\n",
" d = defer.Deferred()\n",
" progress[a].append(d)\n",
" res = await d\n",
" return res\n",
" progress[a] = []\n",
" data = json.dumps([a]).encode('utf-8')\n",
" try:\n",
" response = await treq.post('http://localhost:1234/negate', data=data)\n",
" content = await response.content()\n",
" res = json.loads(content.decode('utf-8'))\n",
" except:\n",
" for d in progress.pop(a):\n",
" d.errback(failure.Failure())\n",
" raise\n",
" else:\n",
" negations[a] = res\n",
" for d in progress.pop(a):\n",
" d.callback(res)\n",
" return res\n",
"\n",
"@route('/cachingnegate/<int:a>')\n",
"async def cachingnegate(request, a):\n",
" res = await cachednegate(a)\n",
" return 'Answer: {}'.format(res)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Try negating [five](http://localhost:9090/cachingnegate/5). Then click the link again, and see it load much faster.\n",
"\n",
"A real-life cache would have expiry (luckily for us, math is immutable) and a way to control the maximum memory. Caching is hard!"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Exercise -- cache multiplications"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"@route('/cachingmultiply/<int:a>/<int:b>')\n",
"async def cachingmultiply(request, a, b):\n",
" # ???\n",
" # Hint: as a first pass, avoid the \"in progress\" cache\n",
" pass"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Testing\n",
"\n",
"All this calling of real live network services is fun, but in order to make our software reliable, we have to develop a test suite that can run things quickly, and won't fail randomly if our wifi is slow. So, how does this all fit together with some unit tests?\n",
"\n",
"Normally, we'd just do `trial ./your_module.py` or `trial your.package.test`, but in order to get Twisted-style unit tests to run in the context of a Notebook we'll use this handy little function:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"from twisted.trial.unittest import SynchronousTestCase\n",
"from twisted.trial.runner import TestLoader\n",
"from twisted.trial.reporter import VerboseTextReporter\n",
"\n",
"def runTests(cls):\n",
" TestLoader().loadClass(cls).run(VerboseTextReporter(realtime=True))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now, before we start testing anything complicated, let's start with a simple test that fails, so we can see a traceback:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from twisted.trial.unittest import SynchronousTestCase\n",
"@runTests\n",
"class TestSomething(SynchronousTestCase):\n",
" def test_something(self):\n",
" self.assertEqual(1, 2)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Then let's make it pass, to ensure we can see that it will report a success:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from twisted.trial.unittest import SynchronousTestCase\n",
"@runTests\n",
"class TestSomething(SynchronousTestCase):\n",
" def test_something(self):\n",
" self.assertEqual(2, 2)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now that we're warmed up, let's demonstrate how tests can control when Deferreds fire. Let's make a fake API facade that implements an `add` method which we can give one result to. Each time we ask it to `add`, it will add a `Deferred` to a queue, and then when when we give it a result, it will give that to each result in the order that it was added."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"from twisted.internet.defer import Deferred\n",
"\n",
"class FakeAPI(object):\n",
" def __init__(self):\n",
" self.deferreds = []\n",
" self.addends = []\n",
" \n",
" def giveOneResult(self):\n",
" a, b = self.addends.pop(0)\n",
" self.deferreds.pop(0).callback(a + b)\n",
" \n",
" def add(self, a, b):\n",
" self.deferreds.append(Deferred())\n",
" self.addends.append((a, b))\n",
" return self.deferreds[-1]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You can use it like so:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"fakeAPI = FakeAPI()\n",
"async def asyncmethod():\n",
" print(\"Result:\", await fakeAPI.add(3, 4))\n",
"\n",
"from twisted.internet.defer import ensureDeferred\n",
"print(\"calling...\")\n",
"coroutine = asyncmethod()\n",
"print(\"ensuring...\")\n",
"ensureDeferred(coroutine)\n",
"print(\"giving result...\")\n",
"fakeAPI.giveOneResult()\n",
"print(\"Done!\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In a test, that would look like this:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"@runTests\n",
"class TestFakeAPI1(SynchronousTestCase):\n",
" def test_fake(self):\n",
" fake = FakeAPI()\n",
" result = fake.add(3, 4)\n",
" self.assertEqual(result, 7)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"But of course, that won't work - we need to extract the result. For this, Twisted provides the test method `successResultOf`:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"@runTests\n",
"class TestFakeAPI2(SynchronousTestCase):\n",
" def test_fake(self):\n",
" fake = FakeAPI()\n",
" result = fake.add(3, 4)\n",
" self.assertEqual(self.successResultOf(result), 7)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"But, in order for that to work, we need to *give* it a result first:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"@runTests\n",
"class TestFakeAPI3(SynchronousTestCase):\n",
" def test_fake(self):\n",
" fake = FakeAPI()\n",
" result = fake.add(3, 4)\n",
" fake.giveOneResult()\n",
" self.assertEqual(self.successResultOf(result), 7)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Finally, we might want to make sure that there _is_ no result at the appropriate point in the test:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"@runTests\n",
"class TestFakeAPI4(SynchronousTestCase):\n",
" def test_fake(self):\n",
" fake = FakeAPI()\n",
" result = fake.add(3, 4)\n",
" fake.giveOneResult()\n",
" self.assertNoResult(result)\n",
" self.assertEqual(self.successResultOf(result), 7)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Not there, apparently:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"@runTests\n",
"class TestFakeAPI5(SynchronousTestCase):\n",
" def test_fake(self):\n",
" fake = FakeAPI()\n",
" result = fake.add(3, 4)\n",
" self.assertNoResult(result)\n",
" fake.giveOneResult()\n",
" self.assertEqual(self.successResultOf(result), 7)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"That's good for testing everything from the ground up, but, we just spent all this time building this web API; both things that call web requests, with treq, and things that implement web APIs, with klein. How do we test those for real, without having to spin up big expensive servers?\n",
"\n",
"Treq comes with the `treq.testing` module you can use for this exact purpose."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from treq.testing import StubTreq\n",
"from klein import resource\n",
"\n",
"@runTests\n",
"class APITest1(SynchronousTestCase):\n",
" def test_api(self):\n",
" myTreq = StubTreq(resource())\n",
" deferredResponse = myTreq.get('http://localhost:8080/greet/bob')\n",
" response = self.successResultOf(deferredResponse)\n",
" self.assertEqual(response.code, 200)\n",
" text = self.successResultOf(treq.content(response))\n",
" self.assertEqual(text, b\"Hello BOB!\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As a convenience, `treq.testing` causes simple responses that come back immediately to fire their `Deferred`s right away. But what if the logic were a bit more complex? For example, what if we had this \"quadruple\" API that could quadruple a number? (As we all know, quadrupling is accomplished by adding a number to itself, doing that again, and then adding those two numbers together):\n",
"\n",
"Since we need to pass this route a reference to the API it's going to use, we're going to use a class to hold it; this is very much like top-level routes except we have to make our own `Klein`:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"from klein import Klein\n",
"class Quadruplex(object):\n",
" def __init__(self, api):\n",
" self.api = api\n",
"\n",
" router = Klein()\n",
" @router.route(\"/quadruple/<int:x>\")\n",
" async def quadruple(self, request, x):\n",
" return str(await self.api.add(await self.api.add(x, x),\n",
" await self.api.add(x, x)))\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now, to test it:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"@runTests\n",
"class APITest2(SynchronousTestCase):\n",
" def test_api(self):\n",
" myFakeAPI = FakeAPI()\n",
" quad = Quadruplex(myFakeAPI)\n",
" myTreq = StubTreq(quad.router.resource())\n",
" deferredResponse = myTreq.get('http://localhost:8080/quadruple/4')\n",
" # answer all those API calls!\n",
" response = self.assertNoResult(deferredResponse)\n",
" myFakeAPI.giveOneResult()\n",
" myFakeAPI.giveOneResult()\n",
" myFakeAPI.giveOneResult()\n",
" response = self.successResultOf(deferredResponse)\n",
" self.assertEqual(response.code, 200)\n",
" text = self.successResultOf(treq.content(response))\n",
" self.assertEqual(text, b\"16\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The problem here is that although a result has been given, it is stuck in buffers inside `StubTreq`'s virtual network, and needs to be flushed out. This is what the `StubTreq.flush` method is for:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"@runTests\n",
"class APITest2(SynchronousTestCase):\n",
" def test_api(self):\n",
" myFakeAPI = FakeAPI()\n",
" quad = Quadruplex(myFakeAPI)\n",
" myTreq = StubTreq(quad.router.resource())\n",
" deferredResponse = myTreq.get('http://localhost:8080/quadruple/4')\n",
" # answer all those API calls!\n",
" response = self.assertNoResult(deferredResponse)\n",
" myFakeAPI.giveOneResult()\n",
" myFakeAPI.giveOneResult()\n",
" myFakeAPI.giveOneResult()\n",
" myTreq.flush()\n",
" response = self.successResultOf(deferredResponse)\n",
" self.assertEqual(response.code, 200)\n",
" text = self.successResultOf(treq.content(response))\n",
" self.assertEqual(text, b\"16\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Exercise: now that you know how to write tests for klein routes, write a test for max."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"@router.route(\"/maximum/<int:x>/<int:y>\")\n",
"def maximum(request, x, y):\n",
" \"Exercise: do a test-driven implementation of max\"\n",
"\n",
"@runTests\n",
"class MaxTest2(SynchronousTestCase):\n",
" \n",
" def test_max(self):\n",
" self.fail(\"...\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## More Resources\n",
"\n",
"* [Twisted Finger Tutorial](https://twistedmatrix.com/documents/current/core/howto/tutorial/index.html)\n",
"* [Twisted book](http://shop.oreilly.com/product/0636920025016.do)\n",
"* [Twisted Web \"60 seconds\" howtos](https://twistedmatrix.com/documents/current/web/howto/web-in-60/index.html)\n",
"* [Twisted HOWTOs](https://twistedmatrix.com/documents/current/core/howto/)\n",
"* [Twisted slow poetry tutorial](http://krondo.com/an-introduction-to-asynchronous-programming-and-twisted/)\n",
"* [Deferred tutorial](https://moshez.files.wordpress.com/2009/03/deferred-pres.pdf)\n",
"\n",
"## Questions?\n",
"\n",
"A question is something to which you do not know the answer."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.0b2"
}
},
"nbformat": 4,
"nbformat_minor": 1
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment