We worked on developing a distributed task execution engine for Apache Airavata. This feature was brought to our attention by the team from the Apache Airavata. Essentially, under the current design, jobs are pushed to certain typed RabbitMQ queues, and workers that are compatible with a certain kinda of job, subscribe to these queues. We realized that there workers are not the same and we need to maximize the throughput of jobs and workers. We use the max flow algorithm in order to do the assignment and batch message the queue to distribute the jobs.
We created a buffer that was batch processed with the scheduler periodically and pushed to a RabbitMQ queue. We also Couchbase to keep track of the workflow graph structures.
Our code was pulled to their prototyping repo: https://github.com/airavata-courses/spring17-workload-management/tree/hackillinois
Our mentors from IU were really helpful and useful in understanding the code workflow and building out our design.