Skip to content

Instantly share code, notes, and snippets.

@thakursaurabh1998

thakursaurabh1998/WorkProofAOSSIE.md

Last active August 31, 2019 05:06
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save thakursaurabh1998/5b72cc8f2a59e3e2c461f3c57db8d004 to your computer and use it in GitHub Desktop.
Save thakursaurabh1998/5b72cc8f2a59e3e2c461f3c57db8d004 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
WorkProofAOSSIE.md

CarbonFootprint-API

Google Summer of Code '19 Final Submission Report

Student - Saurabh Thakur

Links

Google Summer of Code '19:

This year more focus was shifted on making the frontend of the dashboard more pleasing by adding more features as well as making the project more appealing and easy for the people who develop or want to develop to this open source project by improving the CI/CD of the project as well as development environment setup.

This years summer of code brought quite a few interesting changes to the CarbonFootprint-API project. I tried to make this project a little more up to date with new UI, refactored code, more tests and better DevOps.

It was really interesting to see how well the project is maintained because the transition was smooth, whether I was developing new things or I was refactoring them.
So I started off by working on the client side of the application. Correcting bugs in the UI and also started working on updating version of React.
During working with React I was facing an issue which was causing me a lot of frustration, the issue was the build time it was taking to show me the changes I made after changing the JSX code. In short words hot reloading was really slow. This was due to the webpack config which was used to build the client assets. This webpack config was not suitable for development life cycle, so to solve this problem I started finding solutions. I found two main solutions:

  1. One was to create a separate webpack config for development and one for production level builds. This would need a lot of insight in webpack as well as would become an addition which needed to be manage.
  2. To use create-react-app (CRA).

I started working on the latter solution. This decision wasn't random and there were proper reasons of doing that which we are going to discuss below:

  • CRA managed the webpack config for dev and prod environments which was what we needed, but it also give a lot of other useful features.
  • It provides with react-scripts which can help improve the dev lifecycle.
  • It consists of multiple scripts, from starting the development server, to running the production build.
  • And all of this comes out of the box, so a developer doesn't need to worry about the configuration of the project.
  • It also solved another issue which we were facing was the usage of css styling in the JSX files.
  • I had to configure my babel separately so that it can take account of these changes and do not break. CRA also handled this out of the box.

After doing all the necessary changes to all the files the client side was finally updated and the MR was merged.

I was also working on another solution which I proposed was to add a new feature of taking entry of daily user activities, from which we can calculate and track the Carbon emission of that user. This data could be shown to the user in the form of heatmap, just like gitlab shows the previous commits of a user.
Creating the heatmap was also a challenge in itself, the library which I found after researching on the internet to visualize the data was d3. The final result looked like this:

The profile page now shows a heatmap which displays the CO2 emission of a user over past few months.
Hovering on a tile shows the emission of the day with the date

Next what I did was to add eslint, which was a long awaited desire of this project.

Swagger was also introduced to write documentation of the existing apis. It helped with understanding the response details of the api as well as it gave an option to test run the api from the documentation page only so that a developer can see the response of the api without setting up the whole project.

Various sections that can be explored on Swagger Docs page

The major work done was to improve the CI/CD of the project.
Docker was used for containerisation of the application. Previously setting up the development environment was very difficult and tedious task, installing mongodb, redis as well as configuring the project with all the dependencies. This was really confusing and time consuming.
Now with the help of docker-compose all of this becomes easy and the project gets up and running within minutes.
All the configuration is done in docker-compose files which are separate for development and production purposes.

Also the deployment of the project have some major changes. Earlier we used to go to the server and pull the changes and then redeploy manually whenever something was merged to production but now with gitlab-runner everything is automated and the application can be deployed with a click of a single button from the gitlab merge request page.

Technical Details

  • Docker - To containerize the application for development and deployment purposes.
  • Docker Compose - For runnin multiple services in container interacting with each other.
  • Gitlab Runner - Gitlab runner is used for deployment of the project in the production server.
  • ESLint - ESLint helps provide linting for the project and can also be used to enforce code structure and style.
  • Swagger Documentation - Swagger made API documentation and first time usage really easy, so we added it to the platform
  • D3 JS - D3 is used in creating the user carbon emission heatmap in user profile.

Merge Requests

  • MR !134 (Improve responsiveness of profile page)(merged)

    • The profile page components were out of alignment and were also not suitable for responsive usage across different devices.
    • Most of the components were corrected, and the profile page was refactored for better UI and UX.

  • MR !138 (User's daily activity collection and CO2 emission visualization)(merged)

    • Enhances how the user interacts with our platform.
    • Providing user the interface to add the daily activity to keep track of the Carbon Dioxide emission.
    • Also displaying all the previous year data in the form of a heatmap.

  • MR !139 (Google maps API dependency removed and Microsoft Bing Maps API added)(merged)

    • Refactored code uses Bing maps API to use functions like Distance Matrix and Locations API.
    • Module dependency of @google/maps also removed.
    • Docs updated with the new instructions to create key of Microsoft Bing Map API.
    • Configuration file examples also updated.
    • All the tests related to the maps API are now passing which can be seen here in the respective commit pipeline job

  • MR !140 (pipeline error fix, correction in assertion value)(merged)

    • Correction of wrong assertion value which was leading to failing of tests.

  • MR !141 (Documentation change: adding gitlab env config)(merged)

    • A lot of steps were missing to add the environment variables and setting up the development environment of the project.
    • Adding steps to add gitlab config env variables in documentation.

  • MR !143 (Initialized swagger)(merged)

    • Made api documentations for different API paths
    • Serving the docs on /api/docs path from where the user can directly make API calls and check the response structure without doing any input
    • Making use of JSDocs
    • Custom documentation css styles for better UI of the API doc page
    • Tests can be made on local development servers as well as production API

  • MR !144 (Region value changed to default)(merged)

    • This MR was done to correct the failing of the pipeline.

  • MR !149 (readme doc changes)(merged)

    • Add the services used in the documentation, i.e. redis, mongo for the clarity of services to developers.

  • MR !150 (CORS was configured on the server side)(merged)

    • For cross site requests during development and other purposes, cors is implemented on the server side.
    • Only whitelists specific urls.

  • MR !151 (React v16.8)(merged)

    • React version bump to latest 16.8.
    • Directory structure and implementation of client side changed.
    • Webpack build was removed and create-react-app was used to make development lifecycle fast and efficient.
    • Previously load time after changing a jsx file was ~5 seconds.
    • Was decreased drastically on usage of CRA with help of react development builds and dev server provided within CRA.
    • Deployment script was also changed and react-scripts build was used to generate static asset files.

  • MR !152 (HOTFIX)(merged)

    • This was an hotfix of an old bug in the API code.

  • MR !153 (Changing urls to https from http)(merged)

    • Many libraries were imported as a script directly using a URL and not as a package.
    • In these URLs http was used which was raising an error when they were called from a https page.
    • This was resolved in this merge request.

  • MR !155 (Removing constant mongoose deprecation warning)(merged)

    • Mongoose constantly threw an error of deprecation of promises of its library.
    • This was fixed by changing the promise property of mongoose with native javascript promise.

  • MR !156 (gitbook build permission access)(merged)

    • Fix of an old bug in the deploy script.
    • The script wasn't able to build the gitbook documentation and the documentation wasn't opening because of the missing sudo keyword.
    • Issue was fixed and gitbook docs were built on production server.

  • MR !157 (removing unnecessary package and files)(merged)

    • Older webpack config removed.
    • Packages which weren't used anymore were removed and package.json updated.

  • MR !158 (Adding ESLint)(merged)

    • All the javascript files weren't following any guidelines for writing code. There was no linting present. The code was highly unstructured and varied on different files.
    • ESLint config file was added with airbnb configs as a base config.
    • All the files were reformatted and changes were made like arrow functions, use of let and const, etc.
    • Lint tests were also added in gitlab-ci to test the linting in each code push.

  • MR !159 (Documentation fixes)(merged)

    • A lot of docs needed to be updated because of the changes of client side changes after intro of CRA.

  • MR !160 (FIX: mapdata dependent endpoints)(merged)

    • Old bug fix of map data.
    • There was a logical error in the code when calculating emissions.

  • MR !161 (FIX: default color when no data available)(merged)

    • Heatmap was showing wrong color whent there was no user data present.
    • Changed the default colour to grey.

  • MR !162 (Tests divided to more structured way using describe and more tests)(merged)

    • Tests were not classified so I grouped them according to their API endpoints.
    • The tests status printed in the CI are now more readable and are grouped.
    • More tests were added for some new API endpoints as well as previous uncovered parts.

  • MR !164 (Dockerfile init with docker-compose)(open)

    • The app is containerized using Docker.
    • Docker compose used for fast setup of development environment as well in production for the services used in API (i.e. redis).
    • Setting up of project becomes very easy with only a single command.
    • The deployment of the project is also changed which takes use of the Gitlab CI/CD which is declared in the gitlab-ci.yml file.
    • On server we have a gitlab-runner daemon running which listens for any deployment trigger from the gitlab platform.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment