Skip to content

Instantly share code, notes, and snippets.

@Cdaprod

Cdaprod/My Dev Tools.md

Last active June 13, 2024 02:20
Show Gist options
  • Save Cdaprod/4a5f054d2781699b0cf02b71eb2cf955 to your computer and use it in GitHub Desktop.
Save Cdaprod/4a5f054d2781699b0cf02b71eb2cf955 to your computer and use it in GitHub Desktop.
Download ZIP
These steps provide a cohesive setup with MinIO, Weaviate, and Tailscale, along with the necessary tools for your DevOps and AIOps tasks.
Raw
My Dev Tools.md

Setting Up CLI and Python Integration for MinIO and Weaviate in a Hybrid Cloud Environment

Description:

This guide provides detailed instructions for configuring a cohesive multi-server interface architecture using Tailscale, MinIO, Weaviate, GitHub, Python, and Docker/KinD within a hybrid cloud environment. It includes steps to install necessary tools, create Python scripts for interacting with MinIO and Weaviate, and define bash functions and aliases for streamlined CLI operations. The provided scripts and commands ensure seamless file uploads to MinIO and indexing in Weaviate, enhancing your DevOps and AIOps workflows.

Implementation:

Here’s how to configure your environment with the specified settings for GitHub, MinIO, and Weaviate.

1. Ensure Python Environment is Set Up

Install Python and Virtual Environment Tools

# Update package list and install Python and virtual environment tools
sudo apt update
sudo apt install python3 python3-venv python3-pip -y

Create and Activate a Virtual Environment

# Create a virtual environment
python3 -m venv env

# Activate the virtual environment
source env/bin/activate

Install Required Python Packages

# Install MinIO, Weaviate client, and other necessary packages
pip install minio weaviate-client requests

2. Connect to MinIO

Here's how to connect to your already deployed MinIO instance using Python:

from minio import Minio

# Connect to MinIO instance
minio_client = Minio(
    "play.min.io:443",  # MinIO server address
    access_key="minioadmin",  # MinIO access key
    secret_key="minioadmin",  # MinIO secret key
    secure=True  # Using HTTPS
)

# Example: List all buckets
buckets = minio_client.list_buckets()
for bucket in buckets:
    print(bucket.name)

3. Connect to Weaviate

Here's how to connect to your Weaviate Cloud Console instance using Python:

from weaviate import Client

# Connect to Weaviate instance
weaviate_client = Client(
    "https://my-cluster-4z3bgtfd.weaviate.network",  # Weaviate server address
    additional_config={
        "auth_client_secret": {
            "secret": "your-secret-key"  # Replace with your Weaviate secret key if needed
        }
    }
)

# Example: Get the schema
schema = weaviate_client.schema.get()
print(schema)

4. Set Up GitHub CLI

Install GitHub CLI

# Install GitHub CLI on Debian-based systems
type -p curl >/dev/null || sudo apt install curl -y
curl -fsSL https://cli.github.com/packages/githubcli-archive-keyring.gpg | sudo dd of=/usr/share/keyrings/githubcli-archive-keyring.gpg
sudo chmod go+r /usr/share/keyrings/githubcli-archive-keyring.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/githubcli-archive-keyring.gpg] https://cli.github.com/packages stable main" | sudo tee /etc/apt/sources.list.d/github-cli.list > /dev/null
sudo apt update
sudo apt install gh -y

Authenticate with GitHub

# Authenticate GitHub CLI
gh auth login

# Set default GitHub username
gh config set -h github.com user cdaprod

5. Set Up Docker and Kubernetes in Docker (KinD)

Install Docker

# Install Docker on Debian-based systems
curl -fsSL https://get.docker.com -o get-docker.sh
sh get-docker.sh

Install KinD

# Install KinD
GO111MODULE="on" go install sigs.k8s.io/kind@v0.11.1

# Add Go bin directory to PATH
export PATH=$PATH:$(go env GOPATH)/bin

# Create a KinD cluster
kind create cluster

Example Shell Script to Install All Tools

Here is a comprehensive shell script to automate the installation of all tools:

#!/bin/bash

# Function to install Docker
install_docker() {
    curl -fsSL https://get.docker.com -o get-docker.sh
    sh get-docker.sh
}

# Function to install Tailscale
install_tailscale() {
    curl -fsSL https://tailscale.com/install.sh | sh
    sudo tailscale up
}

# Function to install GitHub CLI
install_github_cli() {
    type -p curl >/dev/null || sudo apt install curl -y
    curl -fsSL https://cli.github.com/packages/githubcli-archive-keyring.gpg | sudo dd of=/usr/share/keyrings/githubcli-archive-keyring.gpg
    sudo chmod go+r /usr/share/keyrings/githubcli-archive-keyring.gpg
    echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/githubcli-archive-keyring.gpg] https://cli.github.com/packages stable main" | sudo tee /etc/apt/sources.list.d/github-cli.list > /dev/null
    sudo apt update
    sudo apt install gh -y
    gh auth login
    gh config set -h github.com user cdaprod
}

# Function to install Python and virtual environment tools
install_python() {
    sudo apt update
    sudo apt install python3 python3-venv python3-pip -y
}

# Function to install KinD
install_kind() {
    GO111MODULE="on" go install sigs.k8s.io/kind@v0.11.1
    export PATH=$PATH:$(go env GOPATH)/bin
    kind create cluster
}

# Install Docker
install_docker

# Install Tailscale
install_tailscale

# Install GitHub CLI
install_github_cli

# Install Python and set up virtual environment
install_python
python3 -m venv env
source env/bin/activate
pip install minio weaviate-client requests

# Install KinD
install_kind

echo "All tools installed successfully!"

Save this script as install_tools.sh, give it execute permissions, and run it:

chmod +x install_tools.sh
./install_tools.sh

Example Python Script to Interact with MinIO and Weaviate

Here's an example Python script to upload a file to MinIO and create an object in Weaviate:

from minio import Minio
from weaviate import Client

# MinIO Configuration
minio_client = Minio(
    "play.min.io:443",  # MinIO server address
    access_key="minioadmin",  # MinIO access key
    secret_key="minioadmin",  # MinIO secret key
    secure=True  # Using HTTPS
)

# Weaviate Configuration
weaviate_client = Client(
    "https://my-cluster-4z3bgtfd.weaviate.network",  # Weaviate server address
    additional_config={
        "auth_client_secret": {
            "secret": "your-secret-key"  # Replace with your Weaviate secret key if needed
        }
    }
)

# Example: Upload a file to MinIO and create an object in Weaviate
def upload_and_index(file_path, bucket_name, class_name, schema):
    # Upload file to MinIO
    minio_client.fput_object(bucket_name, file_path, file_path)
    
    # Create an object in Weaviate
    weaviate_client.data_object.create({
        "file_name": file_path,
        "bucket": bucket_name
    }, class_name, schema)

# Usage
upload_and_index("example.txt", "my-bucket", "Document", {"file_name": "string", "bucket": "string"})

Certainly! You can create a set of bash functions and alias them for easier use from the command line. Here is an extended version that includes bash functions for uploading files to MinIO and indexing them in Weaviate, along with the necessary Python scripts.

1. Create the Python Script

First, create a Python script called minio_weaviate.py that includes the functions for uploading to MinIO and indexing in Weaviate.

minio_weaviate.py

from minio import Minio
from weaviate import Client
import sys
import json

# MinIO Configuration
minio_client = Minio(
    "play.min.io:443",  # MinIO server address
    access_key="minioadmin",  # MinIO access key
    secret_key="minioadmin",  # MinIO secret key
    secure=True  # Using HTTPS
)

# Weaviate Configuration
weaviate_client = Client(
    "https://my-cluster-4z3bgtfd.weaviate.network",  # Weaviate server address
    additional_config={
        "auth_client_secret": {
            "secret": "your-secret-key"  # Replace with your Weaviate secret key if needed
        }
    }
)

def upload_to_minio(file_path, bucket_name):
    # Upload file to MinIO
    minio_client.fput_object(bucket_name, file_path, file_path)
    print(f"Uploaded {file_path} to bucket {bucket_name}.")

def index_in_weaviate(file_path, bucket_name, class_name, schema):
    # Create an object in Weaviate
    weaviate_client.data_object.create({
        "file_name": file_path,
        "bucket": bucket_name
    }, class_name, schema)
    print(f"Indexed {file_path} in Weaviate class {class_name}.")

def upload_and_index(file_path, bucket_name, class_name, schema):
    upload_to_minio(file_path, bucket_name)
    index_in_weaviate(file_path, bucket_name, class_name, schema)

if __name__ == "__main__":
    action = sys.argv[1]
    file_path = sys.argv[2]
    bucket_name = sys.argv[3]
    if action == "upload":
        upload_to_minio(file_path, bucket_name)
    elif action == "index":
        class_name = sys.argv[4]
        schema = json.loads(sys.argv[5])
        index_in_weaviate(file_path, bucket_name, class_name, schema)
    elif action == "upload_and_index":
        class_name = sys.argv[4]
        schema = json.loads(sys.argv[5])
        upload_and_index(file_path, bucket_name, class_name, schema)
    else:
        print("Invalid action. Use 'upload', 'index', or 'upload_and_index'.")

2. Create Bash Functions and Aliases

Add the following bash functions to your ~/.bashrc or ~/.bash_profile to create convenient aliases for the operations.

Add to ~/.bashrc or ~/.bash_profile

# Function to upload a file to MinIO
upload_to_minio() {
    python3 minio_weaviate.py upload "$1" "$2"
}

# Function to index a file in Weaviate
index_in_weaviate() {
    python3 minio_weaviate.py index "$1" "$2" "$3" "$4"
}

# Function to upload a file to MinIO and index it in Weaviate
upload_and_index() {
    python3 minio_weaviate.py upload_and_index "$1" "$2" "$3" "$4"
}

# Aliases for convenience
alias minio_upload=upload_to_minio
alias weaviate_index=index_in_weaviate
alias upload_index=upload_and_index

After adding these lines, reload your bash profile:

source ~/.bashrc
# or
source ~/.bash_profile

3. Usage Examples

Now you can use the following commands directly from your CLI.

Upload a file to MinIO

minio_upload example.txt my-bucket

Index a file in Weaviate

weaviate_index example.txt my-bucket Document '{"file_name": "string", "bucket": "string"}'

Upload a file to MinIO and index it in Weaviate

upload_index example.txt my-bucket Document '{"file_name": "string", "bucket": "string"}'

4. Full Setup Script

Here is the complete setup script including the creation of the Python script and the bash functions:

#!/bin/bash

# Create Python script
cat << 'EOF' > minio_weaviate.py
from minio import Minio
from weaviate import Client
import sys
import json

# MinIO Configuration
minio_client = Minio(
    "play.min.io:443",  # MinIO server address
    access_key="minioadmin",  # MinIO access key
    secret_key="minioadmin",  # MinIO secret key
    secure=True  # Using HTTPS
)

# Weaviate Configuration
weaviate_client = Client(
    "https://my-cluster-4z3bgtfd.weaviate.network",  # Weaviate server address
    additional_config={
        "auth_client_secret": {
            "secret": "your-secret-key"  # Replace with your Weaviate secret key if needed
        }
    }
)

def upload_to_minio(file_path, bucket_name):
    # Upload file to MinIO
    minio_client.fput_object(bucket_name, file_path, file_path)
    print(f"Uploaded {file_path} to bucket {bucket_name}.")

def index_in_weaviate(file_path, bucket_name, class_name, schema):
    # Create an object in Weaviate
    weaviate_client.data_object.create({
        "file_name": file_path,
        "bucket": bucket_name
    }, class_name, schema)
    print(f"Indexed {file_path} in Weaviate class {class_name}.")

def upload_and_index(file_path, bucket_name, class_name, schema):
    upload_to_minio(file_path, bucket_name)
    index_in_weaviate(file_path, bucket_name, class_name, schema)

if __name__ == "__main__":
    action = sys.argv[1]
    file_path = sys.argv[2]
    bucket_name = sys.argv[3]
    if action == "upload":
        upload_to_minio(file_path, bucket_name)
    elif action == "index":
        class_name = sys.argv[4]
        schema = json.loads(sys.argv[5])
        index_in_weaviate(file_path, bucket_name, class_name, schema)
    elif action == "upload_and_index":
        class_name = sys.argv[4]
        schema = json.loads(sys.argv[5])
        upload_and_index(file_path, bucket_name, class_name, schema)
    else:
        print("Invalid action. Use 'upload', 'index', or 'upload_and_index'.")
EOF

# Add bash functions and aliases to .bashrc
cat << 'EOF' >> ~/.bashrc

# Function to upload a file to MinIO
upload_to_minio() {
    python3 minio_weaviate.py upload "$1" "$2"
}

# Function to index a file in Weaviate
index_in_weaviate() {
    python3 minio_weaviate.py index "$1" "$2" "$3" "$4"
}

# Function to upload a file to MinIO and index it in Weaviate
upload_and_index() {
    python3 minio_weaviate.py upload_and_index "$1" "$2" "$3" "$4"
}

# Aliases for convenience
alias minio_upload=upload_to_minio
alias weaviate_index=index_in_weaviate
alias upload_index=upload_and_index
EOF

# Reload .bashrc
source ~/.bashrc

echo "Setup completed successfully!"

This complete setup script will create the necessary Python script and add the bash functions to your .bashrc file, allowing you to use the new commands immediately.

By following these steps, you will have a cohesive setup with MinIO, Weaviate, and Tailscale, along with the necessary tools for your DevOps and AIOps tasks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment