Skip to content

Instantly share code, notes, and snippets.

@edunham

edunham/twistednotes.rst

Last active August 29, 2015 13:59
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save edunham/10587717 to your computer and use it in GitHub Desktop.
Save edunham/10587717 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
twistednotes.rst

Notes from the PyCon 2014 Twisted BOF

TLDR (from Ying's notes):

  • Python 3 porting:
    • Needed: Which modules are ported, which dependencies are not ported
    • Twisted.python.dist3 (?)
    • If not porting, bring review queue down so porting tickets can be reviewed
  • GitHub integration
    • PRs not accepted yet, but can be made into a patch that Trac accepts. RADIX will document.
    • Contribute to braid to figure out how to move twisted infra to Git (which breaks Trac)
    • Braid takes git PRs, can spin up the whole thing on VM
    • See wiki for remaining work
    • Maybe don't care about trac source browser
    • Talk to Jesse to get some rackspace devops to do this
  • Twisted desired features:
    • Treq maybe should be in Twisted
      • JP volunteers to move multi-part uploader
      • Are integration/performance tests cost effective to run normally? (Maybe for Agent)
    • Logging branch - RALPH volunteers to look at branch
    • Tubes
      • generic fanout and fanin possible? if so api is probably ok
      • needs documentation in framing module
      • eliminate all inheritance (pump)
      • docs are wrong

Longer version:

Moving Twisted to python 3:

  • Dependency graph
    • Use modulegraph or snakefood or something
    • color code ported / needs to be ported / can't port because unported dependencies
  • Trial is not ported yet. How do tests run? Poorly.
  • python3 contains a list of what's officially ported
  • PSF might give a grant about it
  • Work on reducing review queue
  • Can't move to GitHub until queue gets down to no outsatnding branches
  • report 25 shows newbies what tickets someone without commit can review
  • newbies can review anything by a committer
  • Take gh PR link and append .patch to the end, and you get a patch which might be eligible by Twisted's review process
  • PR can be submitted but WON'T be accepted
  • Better to say that we ignore PRs so people don't get confused or get their hopes up
  • GH PRs re-open themselves occasionally
  • Twisted review process is branch changes, not commits
  • Glyph wants to move to GH but won't work on it or stop the project
  • Problem is that people try to fix it then disappear
  • GH project is now online thanks to Tom
  • Can't switch to Git because twistedmatrix.com source browser will die. Can't upgrade trac because Reasons.
  • Get copy of entire site working with Git; submit PR to GH site
  • Documentation strategy is ask Tomprince questions on IRC and write down the answers

* URL rewrite magic to point at Github might be possible to fix Trac's source viewer issues, and has allegedly been done before by Buildbot (says Tom) Resolution: Might be possible to pay Rackspace to do it

What do people want to see in Twisted that would make them use it?

Agents API (and everybody laughs)

  • goal was simple, expressive HTTP api implementation in twisted
  • Agent is not supposed to be highest-level API
  • Maybe treq should be included in core...
  • runs integration tests against httpbin.org whenever it's bulit though
  • Tests should use in-memory agent, and integration tests live elsewhere
  • httpbin dep is not a blocker so treq can be ported to twisted
  • dreid does not want to do the work; possibly volunteer hawkowl?
  • multi-part file body producer can be moved up
  • good tests and doesn't return deferreds from test cases
  • 99% of users need it so merging it in would be batteries-included; HTTP stuff should be ridiculously easy in Twisted
  • Good, simple HTTP client has always been the plan, if treq didn't belong in twisted then agents didn't either
  • Compliance suite would be good but too expensive to maintain
  • Agent needs integration tests.
  • Can do SSL over in-memory transports to test connecting a bunch of APIs at once
  • Fakes should be verified

Logging

  • What's the question?

    There's a new logging library in a branch, with unfortunate history: Somebody did it wrong and built a Big New Feature where they shouldn't, and now it's a Big Unreviewable Patch Logging system was developed for calendar server; tests were retrofitted in then it was ported into Twisted and split into a bunch of modules to be reviewable chunks Ticket has a big comment explaining expectations... Hypothetically it's now in chunks that can be reviewed in correct workflow It specifies what log events are and how they get produced...

  • Mere mortals expect text logs; they offend the people at the table
  • Module allows back and ofrth between text and structured logging without losing structure -- logger should eventually default to using json, but this system has algorithm for non-json
  • Ralph Meijer will look at the ticket during sprints and also review it
  • Glyph wants to change all the internal logging like .noisy
  • We're stuck with loglevels because everyone wants them and they're in an RFC
  • Have to change logging APIs but need something to change them to...
  • Set up new logging system by composing a series of objects, so get backwards-compat by leaving certain modules out
  • logbook is a sane system that solves some interesting related problems
  • Standard library logging can sort of do structured logs if you hack on the handlers

Are tubes done yet?

  • Glyph has gone 2 sessions without fully rewriting so it might be done sometime this decade
  • Could make significant progress at the sprint
  • There's documentation with port forwarder and flow control, and tests exist and work
  • API and examples look pretty sensible and have been around awhile
  • API might be ready for production

* Separate change and also necessary: Need to make sure generic fan-out and fan-in are possible before committing to current API -- "the chat problem" or any broadcast Flow control in Twisted is better because it's technically possible, and that's about where the bar is. Supposed to make backpressure etc. invisible to processors but work okay -- trying to reduce both boilerplate and magic. This also addresses linereceiver -- adds framing logic instead of having a bunch of separate solutions * Naming convention might need to be revisited though * Branch not up for review yet because Glyph needs to be less busy with TLS, lots of undocumented code in framing module, needs at least prototype of fan-out (no obligation to merge after review), fix the documentation that's wrong. And Glyph wants to eliminate ALL inheritance instead of just most: * Right now it inherits from pump and has null implementation superclass -- ought instead to be a null implementation decorator as a wrapper red/orange/blue layer problem -- go watch the talk (find link) * Could make people write the "def whatever; pass" and then add the decorator later * Need a class or decorator that takes a function and returns the thing that it should be?? * (don't want to make a decorator because it magics away the implementation) * Glyph's plan was to set attribute on function using wrapper and use global mutable state * Goal: Get tubes into review during sprint because then we could fix iagent to be tube instead of body producer * It'll still have Agent's problem where if you totally forget, it's not clear what to do

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment