AWS S3 intelligent tearing storage simulation - snippets #python3 #docker #docker-compose #orchestation #cli #http #nginx

README.md

This file belongs to: github.com/realFranco.

emulate-s3-intelligent-tearing

Objetive of this project.

Create a set of vm using docker and interconnect each of them.

There will be a communication layer using a REST API.

As an MVP: Simulate the "intelligent tearing" service from s3.

Other elements could be practice bridges like github actions or gitlab pipelines.

Running example:

python3 main.py

Why a service named "cli"

Required for centralize services, execute tests, static code analysis and open the service connection

Execution example:

Wake up the all the stack

docker compose -f .docker/compose.yml build

# Ref: https://docs.docker.com/engine/reference/commandline/compose_up/
docker compose -f .docker/compose.yml up

# Starts the containers in the background and leaves them running.
docker compose -f .docker/compose.yml up --detach

# Get the name of the existing container (exited and running services)
docker ps -a

# Instance a bash shell in the container
docker exec -it <container name> [ /bin/bash | bash | sh ]

# Turn off the all the services, make sure to run this to release the resources provisioned into the host.
docker compose -f .docker/compose.yml down

How to manipulate the "cli" service

docker compose -f .docker/compose.yml build

docker compose -f .docker/compose.yml run --rm cli <your-command>

docker compose -f .docker/compose.yml down

Tools used for this snippet

docker compose version
>
Docker Compose version v2.15.1

docker version
>
Client:
 Cloud integration: v1.0.29
 Version:           20.10.22
 API version:       1.41
 Go version:        go1.18.9
 Git commit:        3a2c30b
 Built:             Thu Dec 15 22:28:41 2022
 OS/Arch:           darwin/arm64
 Context:           default
 Experimental:      true

Check if postgres is ok from the cli

# Previously, install `postgres` in order to check if is ready for accepting connections.
pg_isready --host=$POSTGRES_HOST --username=$POSTGRES_USER --port=$POSTGRES_PORT
>
host:5432 - accepting connections

# Connect into the db from the `cli` container
psql -h $POSTGRES_HOST -p $POSTGRES_PORT -U $POSTGRES_USER -d $POSTGRES_DB

Check if redis is ok from the cli

# Previously, install `redis-cli` in order the check if is ready for accepting connections.
redis-cli -h $REDIS_HOST -p $REDIS_PORT PING

Check the rest api

curl -v http://0.0.0.0:8005

curl -v http://localhost:8005

curl -v http://localhost:8005/

curl -v http://0.0.0.0:8000

Run the service locally without docker

python3 -m venv env

source env/bin/activate

pip3 install -r src/requirements.txt

uvicorn src.main:app --host 0.0.0.0 --port 8000

# To remove the virtual env.
deactivate

compose.yml

# File location: .docker/compose.yml

version: '3.5'
services:
  cli:
    build:
      context: ./../.
      dockerfile: ./.docker/cli/Dockerfile
    volumes:
      - ./../:/app
    networks:
      - bahrain-international-circuit
    working_dir: /app
    env_file:
      - .env

  db:
    image: postgres:15.2
    container_name: ${POSTGRES_HOST}
    env_file:
      - ./.env
    restart: on-failure
    volumes:
      - ./postgres:/var/lib/postgresql/data
      - ./db-postgres:/docker-entrypoint-initdb.d/
    networks:
      - bahrain-international-circuit
    ports:
      - 8001:5432
    depends_on:
      - cli

  mem-db:
    image: redis:7.0.9
    container_name: ${REDIS_HOST}
    restart: on-failure
    volumes:
      - ./redis:/data
    networks:
      - bahrain-international-circuit
    ports:
      - 8002:${REDIS_PORT}
    depends_on:
      - cli

  api-rest:
    container_name: api-rest  # @todo: put into a variable name.
    build:
      context: ./../.
      dockerfile: ./.docker/api-rest/Dockerfile
    volumes:
      - ./../:/app
    ports:
      - 8004:8000
    networks:
      - bahrain-international-circuit
    working_dir: /app
    env_file:
      - .env
    depends_on:
      - cli
      - db
      - mem-db

  http:
    image: nginx:1.23.3
    volumes:
      - ./http/nginx.conf:/etc/nginx/conf.d/default.conf
    ports:
      - 8005:80
    networks:
      - bahrain-international-circuit
    depends_on:
      - cli
      - db
      - mem-db
      - api-rest

networks:
  bahrain-international-circuit:
    driver: bridge

Dockerfile

# Location: .docker/api-rest/Dockerfile

from alpine:3.17.2

run apk update

run apk add \
    python3=3.10.10-r0 \
    py3-pip && \
    pip3 install --no-cache-dir --upgrade fastapi uvicorn[standard]

CMD ["uvicorn", "src.main:app", "--host", "0.0.0.0", "--port", "8000"]

main.py

"""
File location: src/main.py

http://127.0.0.1:8004/docs

http://127.0.0.1:8004/redoc

uvicorn main:app --reload
"""
from fastapi import FastAPI

app = FastAPI()


@app.get("/")
def read_root():
    return {"Hello": "World"}

nginx.conf

# File location: .docker/http/nginx.conf

server {
    listen 80;

    location / {
        proxy_pass http://api-rest:8000;
    }
}

output

At this point in the time, my AWS S3 Intelligent tearing simulation it is not hosted in a public repo.

The only result that I have for now it is a tweet, as you can see, all the orchestation was done correctly.

See: https://twitter.com/RFranqueza/status/1632720991527772160

I wrote bahrain-international-circuit as the network bridge name becuase I was doing the project while observing the Qualy from the F1 Grand Prix at Bahrein Circuit.

You can rename as you want it.