Project Idea for Rails Girls Summer of Code

RGSoC_OPKF

Building a web application for research and visualization of a large
public dataset of EU farm-subsidies


About the dataset from http://farmsubsidy.org/:

Subsidies paid to farmers and others under the European Union’s Common
Agricultural Policy amount to approximately €55 billion a year, more
than 40% of European Union’s entire annual budget, or around €100 a year
for each EU citizen. Using citizens rights to access government
information the organization tries to obtain detailed information to
payments and recipients of farm subsidies in every EU member state. We
make this data available in a way that is useful to European citizens.
Right now farmsubsidy.org tracks €221.4 billion in payments to
21'938'181 recipients.

At the moment the data is represented on the organizations website in
long lists in different categories like "county" or "type of subsidies
scheme". There is one single search input which just performs a
sting-search across multiple data-columns. The current site is missing
more advanced research tools and visualizations that would make the wast
amount of data more accessible to everyday users, journalists and
activists alike.

The python sourcecode of the website is accessible on github:
https://github.com/eutransparency/Farm-Subsidy
The raw data is available in CSV-Files and the project offers an API in
JSON, XML and YAML.


- - - -

What we intend to do:

The aim is not to replace the existing site. The organizational tasks
required would be way beyond the scope of a "summer of code" project. Our
application will be a companion project to the existing
farmsubsidy.org-site.

We plan to start from the raw dataset and develop a focused
web-application on top of it. It will offer an alternative view on the
existing data using visualization techniques like interactive
treemap-diagrams [
http://philogb.github.io/jit/static/v20/Jit/Examples/Treemap/example1.html
]. We will also look into more advanced search options like a drill-down
interface or facetted search [ https://github.com/sunspot/sunspot ].

Since the dataset is really huge it might be necessary to restrict our
project initially to a subset like the last two or three years.
Questions like this will give me the opportunity to gather experience
about working with large datasets and getting a feeling for the
limitations of a database.

We will develop the application as an open source tool so it can be
re-used by other projects. The core functionality of the application is
searching, browsing and visualizing a large number of financial records.
This is in no way specific to European farm subsidies.
For example there is a sister-project that deals with subsidies to
fishing vessels (http://fishsubsidy.org/). My code could probably be
re-used by this and other projects.

We are about to research projects like http://openspending.org/
[https://github.com/openspending/openspending] to find out where our
project would fit into the spectrum of open-source / open-goverment-data
web-applications.


- - - -

Who will support us?

Stefan Wehrmeyer from the "Open Knowledge Foundation Deutschland" is
working directly with farmsubsidies.org. Since he lives in my hometown,
Berlin, we have contacted him and he agreed to help us with obtaining and
interpreting the raw data. He will act as an adviser on handling large
amounts of public/open government data. Stefan is also a very talented
and experienced javascript front-end developer, so we hope to get some
tips and pointers from him about the existing JS-visualization libraries.

Stefan: stefanwehrmeyer.com / https://github.com/stefanw
Open Knowledge Foundation Deutschland: http://okfn.de/


Urs Kleinert is woking as ruby-developer for a climate-protection
foundation based in Zurich, Switzerland. In their Berlin office,
together with a team of three more developers, they build an maintain a
large multi-party rails-application. [
http://www.myclimate.org/portrait/team.html ] He has been programming
rails-apps since version 1.2 (around 2007). Urs agreed to act as a coach
for all kind of ruby/rails related questions. He and his team also will
introduce me to their everyday workflow techniques, stuff like
story-driven project-planing, TDD or deployment-strategies.

The Berlin office of the myclimate-webdevelopment-team is located at the
co-up coworking-space [ http://co-up.de/ ]. The owners of the shared
office space have generously agreed to support the "Rails Girls Summer
of Code" by providing free desk space for the time of our internship.