Hello everyone! I have been working on the great GSoC project for Mitmproxy this summer. The task was to create the language with new syntax for working with mitmproxy commands in more comfortable way.
What is it about?
Mitmproxy has a modular core. All its functionality is implemented in completely self-contained
addons (extensions in other words). Users interact with addons only through commands and options. An example of interaction using commands: cut.save @focus response.content ~/flows.
Here cut.save is a command name, which is pinned to Python function under the cover.
@focus, response.content and ~/flows are its arguments:
@focus - view flow selector, which is resolved into the list of flows under the cover, response.content represents the content of flow's response, ~/flows is a path to some file.
After typing this command in the Mitmproxy status bar and pressing Enter button, a user interacts with Cut addon and saves response content of the current HTTP flow to the ~/flows file.
This was the only way of сommands appearance: command + args. It wasn't enough for Mitmproxy and we wanted more: nested commands, async commands, pipes like in Bash, array with square brackets and other awesome stuff.
For example:
client.replay(view.select("~h google.com"))view.select "~h google.com" | client.replay
console.command [
@focus "|" export.file
console.choose(Format export.formats())
]
and so on.
More about the language:
-
The documentation I wrote at the very beginning of GSoC18 -> here.
-
Two issues with the hottest discussion about the language I've ever seen. mitmproxy/mitmproxy#3074 mitmproxy/mitmproxy#3087
What work was done?
- I found appropriate parser for creating new language. I chose David Beazley's PLY. My thoughts about it are here -> mitmproxy/mitmproxy#3159.
- I provided the support for parentheses, nested commands, async commands and pipes. I also implemented commands history to navigate through last executed commands like in Ubuntu terminal. I created 6 GSoC PRs this summer -> https://github.com/mitmproxy/mitmproxy/pulls?q=is%3Apr+author%3Akajojify+label%3Agsoc
- All listed PRs have
gsoclabel and detailed description inside. - All language changes went into language branch.
What code got merged? 5 PRs: https://github.com/mitmproxy/mitmproxy/pulls?q=is%3Apr+is%3Amerged+author%3Akajojify+label%3Agsoc
What code wasn't merged? 1 PR: https://github.com/mitmproxy/mitmproxy/pulls?utf8=%E2%9C%93&q=is%3Apr+is%3Aunmerged+author%3Akajojify+label%3Agsoc+
What’s left to do? The area we went into is very deep, but as for now we need to fit our language to the documentation I wrote at the beginning of GSoC:
- fix broken commands. There are 2-3 commands, which don't work after changes I made;
- finish working on unmerged PR. This PR concerns syntax highlighting and autocompletion for the new language;
- implement a few more features like sink and source commands, pointer and deferred commands etc. according to the docs.
It seems, this is everything I want to share with you. I hope to see your comments into Mitmproxy issues or PRs working hard on one of the coolest open source tools.
All the best, Roman