While browsing the internet I stumbled upon an interesting REST countries API, it has a list of all the countries around the world with relevant data like regions, languages, currencies, bordering countries, timezones, location and some other cool stuff.
While using it, I was quite amazed how well it worked but I wondered how it could be better. I identified the following limitations:
-
Data traversal - I can only traverse the data in one direction, country per country but what if I want to know all the subregions in Europe? Which currencies do they use in African countries? Or the full name of bordering countries? To answer those questions, I have to do multiple requests or loop over the whole country list
-
Advanced searching - It's possible to search by a couple of parameters but no advanced search scenario's like "give me all the countries with a population of 10 million or less" or "give me all the countries on the UTC timezone".
-
Over fetching - It's possible to filter the output so that you don't have to query all country-related data but you can't filter out nested data like the symbol of a currency.
I don't think that the maintainers of REST countries should address these limitations. They are quite specific and natural to any traditional REST service. However, these limitations can easily be overcome by combining the power of Neo4j and GraphQL which effectively converts this REST service to a graph service.
Traversing a data tree in different directions is only natural for a graph database like Neo4j. Fetching only what you need is one of the core benefits of GraphQL. Advanced searching, filtering, sorting and pagination comes for free by using the awesome open-source neo4j-graphql.js library.
I took the following steps to effectively transform REST countries to a graph service. However, the same theory could be applied to the REST service that you have in mind. You can explore the final result of the transformation in the graph countries GitHub repo.
To get started you will need to insert all the data from the REST service in the Neo4j graph database. Before you do that, take a step back and think about how you will structure your graph, just drawing some circles and lines with pen and paper or a whiteboard should do the trick, once you are done with that you can take your sketch to the next level with a tool like draw.io
Deciding what should be a node, a property or a relationship can be tricky, what helps is to look at how the REST service is structured. Many-to-many or one-to-many relationships like objects in an array deserve separate nodes. Try to name all your different types of nodes and relationships as soon as possible.
Most importantly, don't worry if you can't model your whole graph at this stage, graph databases are flexible and can easily be changed or extended at a later stage.
There are multiple ways to insert data into a Neo4j instance. If you have a big data set, on the scale of 100mb+, you probably want to use something like Neo4j admin import. If the data will only be inserted via a user interface, start with modelling a GraphQL schema and follow the Neo4j GraphQL Quickstart.
In my use case, I already had all the necessary data and it was relatively small. With a NodeJS script, I looped over all the available countries and used the Neo4j JavaScript driver to execute my Cypher queries. I could have written my GraphQL schema first but I decided to use the full flexibility of custom Cypher queries
https://gist.github.com/ed7e4255a43c9692373af94bb7115b76
The above query creates two different types of nodes and relationships. The MERGE clause assures that I won't have redundant nodes or relationships. Since the data set is so small, I have the luxury to drop the database and insert all the nodes and relationships again every time I execute the script, making it really easy to make changes to my data model.
Now how do we expose a Neo4j database as a simple API to other applications? With the neo4j-graphql.js library, you can easily create a GraphQL endpoint, it will handle resolvers and mutations for you and even the schema can be auto-generated by inferring GraphQL type definitions from your Neo4j database.
Let's walk through the setup:
Create a new directory and initiate a new node project
https://gist.github.com/669daf6029887ad5fc75651d93c7567e
Install all the necessary dependencies
https://gist.github.com/9bcd720089564dac45b8281ef552e5c6
Create a .env file at the root of your project with the connection details of your Neo4j database
https://gist.github.com/07662dc0fc9f2333a5e069d9630e062f
Create an index.js file at the root of your project with the code to infer a GraphQL API from your Neo4j database
https://gist.github.com/def2cdc1cf80d9f70d9c5e2f493ba9d2
Start your GraphQL API:
https://gist.github.com/8d1cc94d396406d056d8094a0619dc79
Now you can explore your GraphQL API on http://localhost:8080. Don't forget to check out the docs, the search, filter and sorting options are extensive. If you like to extend your GraphQL schema with custom Cypher queries, you can always copy the inferred schema and use the @cypher directive
https://gist.github.com/310d1dfacfbfea703e96ca6826979c6c
Try to get some insights into your data by performing some graph algorithms. I found the following queries quite interesting:
The longest shortest path between all the bordering countries in the world is the following:
https://gist.github.com/d59e5de87862991ce5b5d38baf073154
Top 10 languages that are used in different countries, notice how this correlates with the largest kingdoms in our recent history
https://gist.github.com/848a0ed67556182ff51c543da22f1fb8
You can also discover interesting patterns by just browsing your graph, for example, countries with Portuguese as there official language never border each other.
On the oposite side, Arab speaking countries always have a bordering country that also have Arab as an official language if you exclude the island nations of Bahrain and the Comores.
Thanks to the generated GraphQL API, I was able to overcome the limitations of the REST API:
- I can traverse the data tree in any direction, what are the subregions of Europe? Here you go. Which currencies do they use in African countries? No problem. What is the full name of the bordering countries? Don't make it so easy.
- The search, filter and sorting options are amazing. Give me all the countries with a population of 10 million or less. All done and sorted by population. Give me all the countries on the UTC timezone. This is the result.
- Because of the nature of GraphQL, I can query exactly what I want so no under or over fetching.
Next to an easy to use API, I can do some data analytics through graph algorithms to get some more insights into my domain which was impossible through the REST service.