gsoc-proposal.md

#Extending dogfort for real world use

##Synopsis Nodejs provides a fast, lightweight platform for web servers, popularly used in real time applications. Although its possible to make clojurescript applications targeting nodejs, the current experience is far from pleasant. Partly due to node’s heavy use of callbacks, and partly because most existing node libraries are heavily object oriented and imperative. Bodil Stokke’s dogfort (https://github.com/bodil/dogfort) is a nice proof of concept , inspired by ring and compojure, that abstracts out quite a few of these issues.

Dogfort uses ‘promises’ built on top on Node’s EventEmitter class abstracted by Bodil’s redlobster library, instead of chaining callbacks. This allows event handlers to return values which makes a ring inspired api (producing response maps from request maps) possible.

However before it can be put to real world use, Dogfort lacks a few crucial features necessary for any web server framework. This includes (and is not limited to):

• sessions
• cookies
• compatibility with existing node modules
• authentication
• access control
• benchmarks
• tests

In addition, since dogfort runs on top of node, adding an abstraction layer over socket.io would be a must to have feature. Socket.io provides a clean api for two way communication between clients and server over various transports, including websockets, http long-polling, jsonp polling etc.

So the aim of the project is to complete the features mentioned above, whilst keeping the api as idiomatic as possible, so clojure devs can utilise node for writing server side code with a finite number of facepalms.

##Qualifications I converted to clojure 7 months back after watching Rich Hickey’s “Are we there yet?” and reading Chas Emerick et al’s “Clojure Programming” and have used it wherever I can since then. This includes my Final Year Project(ongoing) at NUS (National University of Singapore) on visualizing flight data, and some freelance web work. Before clojure, I used to be a nodejs bolshevik, and worked on a number of hobby/school projects including a multiplayer snake, a collaborative annotator, a realtime anonymous social network all heavily utilizing nodejs. Therefore, its probably not surprising that I find this project to be incredibly interesting.

##Community Benefits Most clojurescript work has so far been focused on the browser, and there are few projects that provide abstractions for using clojurescript over a node runtime. This project attempts to change that by providing a micro framework, based on the same interfaces as ring (which is idiomatic and popular amongst clojurians) on the node runtime. Given the rapid progress on javascript performance in recent times, and node’s gradual maturity, the node runtime could very well approach the jvm in terms of performance.

##Design Issues Since the project aims to provide an idiomatic api layer over node, a number of design decisions will need to be taken. These include (though not limited to):

• Stateful vs Functional Sessions: The ring approach to handle sessions in a pure way is pass the the store as a value in the request map, and let the handler manipulate the store and return a new store assoc-ed in the response map. Another approach, followed by lib-noir is to allow stateful manipulation to a “global” session store, that can be accessed from anywhere in the code. Though the former is idiomatic, it involves more boilerplate. Ideally dogfort should support both approaches.

• Architecture for websockets/longpolling: The ring architecture is great for conventional http requests and responses, but wouldn’t work out of the box for two way real time communication. Since a chunk of the project aims to provide an abstraction over socket.io, I’ll be looking at http-kit’s websocket api, and ztellman’s aleph as possible role models for interfaces.

• Async Workflows: Because of node’s single process event loop, any expensive operation, especially IO has to be non-blocking. Node’s answer is callbacks, which this project aims to limit. Promises solve part of the problem but are not as wieldy when it comes to tasks like asynchronously map or reduce over a collection, for which the need of a library like async.js arises. ztellman’s lamina provides an elegant solution in the form of channels and pipelines. I realize implementing something like lamina from scratch is non-trivial in a node based environment, however, having a cljs sugar over async.js which in the least allows asynchronous seq operations, should be easier. Unless Bodil upgrades redlobster to add similar functionality, I'll be adding this functionality to redlobster

• Compatibility with existing npm modules: Node has some very mature packages, prominently including everyauth, jade etc. Developers should be able to use such packages with ease. This would require an adapter over connect as many modules are designed to work with connect.(Connect is a commonly used middleware framework for node).

##Schedule

(= :numbers :weeks)

1. Finalise design decisions, especially the architecture for the socket.io layer. This component is quite important because it impacts the design of common middleware (like sessions) for both transport types.

2. Implement socket.io layer

3. Debug + Write tests for socket.io layer.

4. Implement session middleware. Under the hood should be mostly similar to ring's session middleware, except that the calls to the session store would be asynchronous. The session store itself should be a protocol, which should allow any store to be easily used. I plan on implementing a simple in memory session store, along with a Redis session store using node_redis under the hood.

5. Implement middleware for cookies. This should be straight forward, modelled on ring's cookie middleware

6. Debug + Write tests for session and cookie middleware

7. Implement connect adapter. Connect middleware share some similarities to ring middleware such that connect middleware adds fields to request and response objects. Thus such an adapter would need to allow request, response objects created by connect to be accessible to dogfort's request, response maps.

8. Debug + write tests for connect adapter.

9. Implement authentication middleware. This would be a great test for the connect adapter, as I'm planning to use everyauth here. I haven't ironed out what the interface would look like, but in the least, I'm planning on some simple cljs sugar over everyauth.

10. Continue work on auth middleware. I expect this to take more time.

11. Write tests for auth middleware.

12. Work on benchmarks, performance tweaks. Compare performance to express, raw node implementations.

13. Refactor, tweak, fix bugs, finish off loose ends

14. (recur 13)

##Deliverables

• An enhanced version of dogfort with the features mentioned above.
• A possibly enhanced version of redlobster. See Asyn Workflow in Design Issues above.

##Other Commitments No other significant summer plans. My semester begins in mid August though, so I'll be less productive then. That's also why I've kept the last few weeks light.

##Other Information Why Clojure? Love the language.

How did you hear about Clojure and GSoC? I knew about GSOC a while back, and noticed clojure participated last year on the gsoc website

Have you developed or contributed to open source projects? Not signficantly yet. Most of my projects on github are solo hobby projects. Looking forward to contribute more :)

Are you applying as a student to another GSoC organization? Nope