NRNB: Graphspace
Implement AWS DevOps Pipeline for Image Deployment for Graphspace
Deep Poharkar (@deep-poharkar)
Project is focused on continuing the work started in GSoC 2023 for the NRNB Org and GraphSpace project, specifically targeting the deployment aspect that remained incomplete. The goal is to create an AWS DevOps pipeline to automate the deployment process of an image to an EC2 instance.
The task involves creating an AWS DevOps pipeline that can efficiently retrieve an image and deploy it to an EC2 instance. The steps might include:
- Setting up an AWS account and configuring necessary permissions: AWS Account Creation: If the contributor doesn't have an AWS account, they'll need to sign up for one. They can visit the AWS website (https://aws.amazon.com/) and create a new account. Some services may have free tiers for new accounts, which can be utilized for this project. IAM (Identity and Access Management): Once the AWS account is set up, the contributor needs to configure Identity and Access Management (IAM) to define roles and permissions. IAM allows creating users, groups, and roles with specific permissions to access AWS resources. Permissions must be set to allow the pipeline to interact with required AWS services like EC2, S3, or ECR (Elastic Container Registry).
- Creating a pipeline script to fetch the image from a designated repository: Source Control Management (SCM): The project's code repository, likely hosted on platforms like GitHub, GitLab, or Bitbucket, will contain the pipeline configuration and scripts. Pipeline Configuration: Using a CI/CD service (e.g., AWS CodePipeline or Jenkins), a pipeline will be configured. This pipeline will have stages defining actions to be performed, such as retrieving the image from a specific repository (e.g., Docker Hub, AWS ECR, or another container registry).
- Automating the deployment process to the EC2 instance: EC2 Instance Configuration: The EC2 instance should be set up and configured appropriately. This includes selecting the instance type, installing necessary software or dependencies, and ensuring security configurations (e.g., security groups, key pairs). Deployment Automation: The pipeline script will automate the deployment process. Once the image is fetched, it should be deployed to the EC2 instance. This might involve using tools like AWS CLI or SDKs to interact with the EC2 service to start a new instance or update an existing one with the latest image.
Project Proposal - (Implement AWS DevOps Pipeline for Image Deployment for Graphspace)
- Understood the Graphspace codebase.
- Discussed the project requirements with the mentor.
- Set up communication channels with the organization for collaboration.
- Explored various AWS services required for the project.
- Refined the strategic plan for project completion.
- Researched on the IAM permission policies needed for the project.
- Provided the organization with the necessary IAM permissions, ensuring least privilege.
- Started working on the project using own IAM user account as a test.
- Created IAM roles for the required AWS services.(Demo)
- Set up AWS CodeBuild to build the Docker image of Graphspace using the Dockerfile.
- Created a build project and connected it to GitHub.(Demo)
- Defined environment variables and settings for the build process.
- Wrote the buildspec.yml (Code)
- Configured CodeBuild to push the new Docker image to the ECR after a successful build.(Demo)
- Verified the build and deployment process to ensure everything functioned correctly.(Demo)
- Installed all dependencies required for the project to run on the EC2 instance.
- Created an IAM role for EC2.
- Received an AWS account from organisation to work.
- Pivoted the project to use Elastic Container Service (ECS).
- Created an IAM role for ECS.
- Modified the buildspec.yml to push the Docker images to the separate ECR repositories.(Demo)
- Created a task definition for each ECS services.(Demo)
- Configured the ECS service to run the Graphspace container.(Demo)
- Auto Scaling and Load Balancing for the ECS service.
- Deployed the Docker containers to the cloud.
- Configured monitoring and logging for the deployed Docker container using AWS CloudWatch.(Demo)
- Tested the deployment process thoroughly to ensure reliability and efficiency.
- Created a detailed report of the project.
- Submitted the final evaluation.
| Tasks | Planned | Completed |
|---|---|---|
| Setting up account and drafting permissions. | Yes | ✔️ |
| Setting Up CodeBuild and ECR | Yes | ✔️ |
| Coding an efficient buildspec.yml | Yes | ✔️ |
| Configured the ECS service to run Graphspace by writing task definitions. | Yes | ✔️ |
| Auto Scaling and Load Balancing for the ECS service. | Yes | ✔️ |
| Deployed the Docker containers to the cloud. | Yes | ✔️ |
| Monitoring and logging for the deployed Docker container using CloudWatch | Yes | ✔️ |
Presented the daily work to the mentors on regular google meets.
1. Mahesh Kasbe (primary mentor) - Github
2. Yash Agarwal (secondary mentor) - Github
I would like to extend my sincere gratitude to my mentors for giving me this incredible opportunity. My mentors, along with the organization members, have been very approachable and have provided me with constant support and guidance throughout the project. I look forward to contributing and learning more from the organization.