We are working on a set of "data quality" tools, to check the quality of open data publications.
These tools are currently very much WIP, here:
- Dashboard: https://github.com/okfn/data-quality-dashboard/tree/feature/refactor
- Example: http://uk-25k.datadashboards.io
- CLI: https://github.com/okfn/data-quality-cli/tree/feature/refactor
- Dataset (UK 25k spend data): https://github.com/okfn/data-quality-uk-25k-spend/tree/feature/refactor
You do not need to learn all these tools. They just provide context for what we are doing, which is:
- Check sets of data that are published, and make a list of the data sources
- Assess the quality of the data published
- Show the results of our quality assessment in a dashboard app
The dataset repository above contains a script (id_data.py) to build out two files: publishers.csv and sources.csv. It does this in a very particular way due to the type of data we are collecting there.
We want to create a different, generic script that does similar: builds publishers.csv and sources.csv from any CKAN instance.
So, the key differences from the above script are:
- Get all resources from a CKAN instance, not just a particular type (so, unlike the 25k spend data above)
- Write a generic script/class/whatever that just takes the URL of the CKAN instance, and builds out the list of
publishers.csv, from the instance's Organizations, andsources.csvfrom the instances resources.
- Given a CKAN instance, use its API to...
- Build a list of publishers followng this schema ( https://github.com/okfn/data-quality-uk-25k-spend/blob/feature/refactor/data/publishers.csv )
- Build a list of sources following this schema ( https://github.com/okfn/data-quality-uk-25k-spend/blob/feature/refactor/data/sources.csv )
Fields from the schema above that do not make sense in this generic implementation - that is fine, just note them.
The output should be:
- A repository on GitHub with:
- The script/module that implements the "CSV database"
- An example database using the Queensland government CKAN instance: https://data.qld.gov.au/
- Using the standard Open Knowledge license for such apps ( https://github.com/okfn/data-quality-dashboard/blob/master/LICENSE )
- A README to describe the usage of the script/module
We should then be able to take this code, and build the publishers.csv and sources.csv tables for any other CKAN instance.
Answering questions
Q: Should the script be resilient, and do verbose error handling as per this example?
A: It is not a requirement, no. You have to balance resilience, code quality, and time constraints accordingly.
Q: A SOLR query example from the example repo - is this important, etc?
A: This is part of the very specific use case in that repo (filtering out data that does not match certain criteria), and should not at all be relevant here. Here, we just want to get all resources, so there is no need for special queries of this type.
Q: Should development be in Python 3?
A: All new code at Open Knowledge is compatible across Python 2 and 3. You can read about our coding standards, and see an example implementation of those standards. So, while we do have a clear preference for 2/3 code, I also do understand that this can be a little tricky in a short project like this, if you have not had this experience before. So, you will not be "penalised" for targeting a single Python runtime, 2 or 3, as long as this is stated clearly in the README.
Q: Is the mentioned "example" database just the two CSV files?
A: Yes, that is correct.