Skip to content

Instantly share code, notes, and snippets.

@Tushar3099
Last active August 17, 2021 15:31
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save Tushar3099/9cef95d1a5e78a0a69d3e385de390e32 to your computer and use it in GitHub Desktop.
Save Tushar3099/9cef95d1a5e78a0a69d3e385de390e32 to your computer and use it in GitHub Desktop.
GSoC'21 Work Product Submission

GSoC’21 Work Product Submission

Abstract

Bench-Routes is a monitoring tool that monitors various performance-based metrics of the mentioned API endpoints. Since monitoring each endpoint on a large scale is an arduous task, bench-routes come to the rescue. These API endpoints monitored through Bench-Routes can be very large in numbers(thousands of endpoints). The scraped data is stored in TSDB as object storage. This project mainly focused on revamping the data model of the scraping mechanism and scheduling the scraping jobs concurrently. Also, changing the configuration of API and recommence the querying from the new TSDB.

General Info

Organisation: BenchRoutes
Name: Tushar Guha Neogi
Email: tusharneogi3099@gmail.com
Github: Tushar3099
Mentor: Harkishen Singh, Aquib Baig

Project Ideas

Revamping Configurations Of BenchRoutes

Previous State

BenchRoutes require API configuration from the user for monitoring the endpoints. The configuration is actually parsed from a file config.yml. The api details are provided by the user in this file according to the prescribed format. The earlier config structure on which BenchRoute was working was not much refactored and has many nesting which was not necessary. We are required to make a new config design and a new parser package that will parse the new config design.

Current State

I have made a new parser package that parses the data from config.yml according to the new design. You can have a look at the new design from the Configuration section of the Scheduler doc. I have written tests considering most of the edge cases. The parser also validates the API config according to the design.

Commit link

https://github.com/bench-routes/bench-routes/commit/2c4998c174527ff39fc2362746354c42a9111439


Scheduling Jobs for scraping endpoint metrics

Previous State

Bench-route monitor API endpoints by launching go-routines for each-endpoints with different workers concurrently. However, there are multiple problems with this system. Some of them are, relaunching of excessive go-routines, no independent scrape interval for each of the endpoints, inefficient jitter calculation method etc. Refer to the Scheduler doc for a detailed description. We need to redesign the architecture of scraping the metrics.

Current State

After discussion with the mentor, we came up with a new composite-pattern design. Now each endpoint is mapped to individual Jobs. The jobs can be termed as iteratively running go-routines that are launched concurrently once the server is started. These jobs are then scheduled using a scheduler which keeps track of the last execution time and schedules accordingly. Also, there is high-level module abstraction that controls the low-level modules. The calculation of jitter metrics is also done using the ping metric.

For this implementation, I have built multiple packages: job, module, scheduler and evaluate. Each package has unit tests and also integration tests. For a detailed description of the architecture and package functionality refer to the Scheduler doc.

Commit link

https://github.com/bench-routes/bench-routes/commit/43e68363f8c9e2ff134e9ef4a4c44d2df82cc8ab


Adding querier and API integration

Previous State

Since the base architecture of BenchRoute has changed due to the last two ideas, the earlier APIs need to be changed with new APIs according to the client requirements. Also, a new TSDB has been integrated with the new Designs, hence querying data from TSDB should also be changed. Once all these tasks are completed, all different modules should be integrated to form the final product.

Current State

I have made a new package querier which does the querying from the TSDB. It searches the range endpoints in the most optimized complexity of log(n) using binary search. It returns the response with multiple details about the query like evaluation-time, query-type, values which is an array of values of ping/jitter/monitor according to the query type etc. I have also written tests for the package. Apart from this, I have made a new api.go file which is integrated with the querier and contains all the APIs required by the client-side with proper error handling. Refer to the commit link for actual code.

Commit link

https://github.com/bench-routes/bench-routes/commit/f8ed0155ca38228ce57d112dac636e1eaa130c1a


Clean up Code(Not a Project idea)

Current State

Although it's not a project idea, I have included it because a lot has been cleaned and refactored in this PR which is required to know if someone is following the code from the earlier version to this version. It simply cleans all the trivial code that was is used anymore and updates the tests of all the modules and makes the entire version ready for release.

Commit link

https://github.com/bench-routes/bench-routes/commit/e13b49aea27584b73d1cfebca491e92cc509b2b3


My Contributions

All of my contributions in BenchRoutes can be found through this link : https://github.com/bench-routes/bench-routes/commits?author=Tushar3099&since=2021-03-06


Final Work Product

The final work product can be found through the following link : https://github.com/bench-routes/bench-routes/commit/8372515df7b491e4253bd23773bb515ec7f1689e

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment