tmosleyIII/stellar_rpc_external_datastore_guide.md

## stellar_rpc_external_datastore_guide.md

      
    Raw
  

              stellar_rpc_external_datastore_guide.md
            
          
    Stellar RPC External Datastore Setup Guide

This guide walks you through setting up stellar-rpc with external datastore support to fetch historical ledgers from Google Cloud Storage.
Prerequisites


Docker installed
Google Cloud SDK (gcloud) installed
Access to a GCS bucket containing Stellar ledger data

Step 1: Install Google Cloud SDK

If you don't have the Google Cloud SDK installed:
# For Debian/Ubuntu
echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" | sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key --keyring /usr/share/keyrings/cloud.google.gpg add -
sudo apt-get update && sudo apt-get install google-cloud-cli

# For macOS
brew install --cask google-cloud-sdk

# For other systems, visit: https://cloud.google.com/sdk/docs/install
Step 2: Set Up Google Cloud Authentication

Option A: Application Default Credentials (Easiest for local testing)

# Login to your Google account
gcloud auth login

# Create application default credentials
gcloud auth application-default login

# Verify the credentials file was created
ls -la ~/.config/gcloud/application_default_credentials.json
Option B: Service Account (Recommended for production)


Go to Google Cloud Console - Service Accounts
Click "Create Service Account"
Name it stellar-rpc-reader and add description
Grant the Storage Object Viewer role
Click "Create Key" → "JSON" and download the file
Save it as gcs-credentials.json in your working directory

Option C: Test if Bucket is Public (Optional)

# Test if the bucket is publicly accessible
gsutil ls gs://obsrvr-stellar-ledger-data-testnet-data/landing/ledgers/testnet/

# If this works without authentication, you might not need credentials
Step 3: Create Configuration Files

Create stellar-rpc-datastore.toml

cat > stellar-rpc-datastore.toml << 'EOF'
# Stellar RPC configuration with external datastore
# For stellar/stellar-rpc:23.0.0-rc2-127

# Basic configuration
ENDPOINT = "0.0.0.0:8000"
ADMIN_ENDPOINT = "0.0.0.0:8001"
NETWORK_PASSPHRASE = "Test SDF Network ; September 2015"
HISTORY_ARCHIVE_URLS = ["https://history.stellar.org/prd/core-testnet/core_testnet_001"]
STELLAR_CORE_BINARY_PATH = "/usr/bin/stellar-core"
CAPTIVE_CORE_CONFIG_PATH = "/config/stellar-core.cfg"
CAPTIVE_CORE_STORAGE_PATH = "/data/captive-core"
DB_PATH = "/data/stellar_rpc.sqlite"

# Logging
LOG_LEVEL = "info"
LOG_FORMAT = "text"

# Set retention to minimum value (1 ledger) to minimize local storage
# The datastore will handle historical queries beyond this
HISTORY_RETENTION_WINDOW = 1

# Fee stats windows must be <= history retention window
SOROBAN_FEE_STATS_RETENTION_WINDOW = 1
CLASSIC_FEE_STATS_RETENTION_WINDOW = 1

# Enable external datastore for historical ledgers
SERVE_LEDGERS_FROM_DATASTORE = true

# Datastore configuration
[datastore_config]
type = "GCS"

[datastore_config.params]
# Obsrvr testnet ledger data bucket
destination_bucket_path = "obsrvr-stellar-ledger-data-testnet-data/landing/ledgers/testnet"

[datastore_config.schema]
ledgers_per_file = 1
files_per_partition = 64000

# Buffered storage backend configuration
[buffered_storage_backend_config]
buffer_size = 100
num_workers = 10
retry_limit = 3
retry_wait = "5s"
EOF
Create stellar-core.cfg

cat > stellar-core.cfg << 'EOF'
# Stellar Core configuration for testnet
NETWORK_PASSPHRASE="Test SDF Network ; September 2015"

DATABASE="sqlite3:///data/stellar.db"

UNSAFE_QUORUM=true
FAILURE_SAFETY=0

[[HOME_DOMAINS]]
HOME_DOMAIN="testnet.stellar.org"
QUALITY="HIGH"

[[VALIDATORS]]
NAME="sdf_testnet_1"
HOME_DOMAIN="testnet.stellar.org"
PUBLIC_KEY="GDKXE2OZMJIPOSLNA6N6F2BVCI3O777I2OOC4BV7VOYUEHYX7RTRYA7Y"
ADDRESS="core-testnet1.stellar.org"
HISTORY="curl -sf https://history.stellar.org/prd/core-testnet/core_testnet_001/{0} -o {1}"

[[VALIDATORS]]
NAME="sdf_testnet_2"
HOME_DOMAIN="testnet.stellar.org"
PUBLIC_KEY="GCUCJTIYXSOXKBSNFGNFWW5MUQ54HKRPGJUTQFJ5RQXZXNOLNXYDHRAP"
ADDRESS="core-testnet2.stellar.org"
HISTORY="curl -sf https://history.stellar.org/prd/core-testnet/core_testnet_002/{0} -o {1}"

[[VALIDATORS]]
NAME="sdf_testnet_3"
HOME_DOMAIN="testnet.stellar.org"
PUBLIC_KEY="GC2V2EFSXN6SQTWVYA5EPJPBWWIMSD2XQNKUOHGEKB535AQE2I6IXV2Z"
ADDRESS="core-testnet3.stellar.org"
HISTORY="curl -sf https://history.stellar.org/prd/core-testnet/core_testnet_003/{0} -o {1}"
EOF
Step 4: Run Stellar RPC with Docker

Option A: With Google Cloud Credentials

# Create data directory
mkdir -p stellar-rpc-data

# Run with application default credentials
docker run -d \
  --name stellar-rpc-datastore \
  -p 8000:8000 \
  -p 8001:8001 \
  -v $(pwd)/stellar-rpc-datastore.toml:/config/stellar-rpc.toml \
  -v $(pwd)/stellar-core.cfg:/config/stellar-core.cfg \
  -v $(pwd)/stellar-rpc-data:/data \
  -e GOOGLE_APPLICATION_CREDENTIALS=/config/gcs-credentials.json \
  -v ~/.config/gcloud/application_default_credentials.json:/config/gcs-credentials.json:ro \
  stellar/stellar-rpc:23.0.0-rc2-127 \
  --config-path /config/stellar-rpc.toml
Option B: With Service Account Key

# If using a service account JSON file
docker run -d \
  --name stellar-rpc-datastore \
  -p 8000:8000 \
  -p 8001:8001 \
  -v $(pwd)/stellar-rpc-datastore.toml:/config/stellar-rpc.toml \
  -v $(pwd)/stellar-core.cfg:/config/stellar-core.cfg \
  -v $(pwd)/stellar-rpc-data:/data \
  -e GOOGLE_APPLICATION_CREDENTIALS=/config/gcs-credentials.json \
  -v $(pwd)/gcs-credentials.json:/config/gcs-credentials.json:ro \
  stellar/stellar-rpc:23.0.0-rc2-127 \
  --config-path /config/stellar-rpc.toml
Option C: Without Credentials (if bucket is public)

# Try without credentials first
docker run -d \
  --name stellar-rpc-datastore \
  -p 8000:8000 \
  -p 8001:8001 \
  -v $(pwd)/stellar-rpc-datastore.toml:/config/stellar-rpc.toml \
  -v $(pwd)/stellar-core.cfg:/config/stellar-core.cfg \
  -v $(pwd)/stellar-rpc-data:/data \
  stellar/stellar-rpc:23.0.0-rc2-127 \
  --config-path /config/stellar-rpc.toml
Step 5: Monitor Startup

# Watch the logs for startup
docker logs -f stellar-rpc-datastore

# Look for messages like:
# "starting Stellar RPC"
# "stellar-core version: stellar-core 23.0.0.rc4"
# No fatal errors

# Ctrl+C to stop following logs
Step 6: Test the RPC Endpoint

Test Basic Health

# Check if RPC is responding
curl -X POST http://localhost:8000 \
  -H 'Content-Type: application/json' \
  -d '{
    "jsonrpc": "2.0",
    "method": "getLatestLedger",
    "id": 1
  }'
Expected response:
{
  "id": "1",
  "jsonrpc": "2.0",
  "result": {
    "id": "...",
    "protocolVersion": 23,
    "sequence": 502464
  }
}
Test Historical Ledger Retrieval (Datastore)

# Get a single historical ledger (this should come from the datastore)
curl -X POST http://localhost:8000 \
  -H 'Content-Type: application/json' \
  -d '{
    "jsonrpc": "2.0",
    "method": "getLedgers",
    "params": {
      "startLedger": 10000,
      "pagination": {
        "limit": 1
      }
    },
    "id": 1
  }' | jq
Test Recent Ledger (Local Storage)

# Get a recent ledger (should be served from local storage)
curl -X POST http://localhost:8000 \
  -H 'Content-Type: application/json' \
  -d '{
    "jsonrpc": "2.0",
    "method": "getLedgers",
    "params": {
      "startLedger": 500000,
      "pagination": {
        "limit": 1
      }
    },
    "id": 1
  }' | jq
Test Pagination

# Get multiple ledgers
curl -X POST http://localhost:8000 \
  -H 'Content-Type: application/json' \
  -d '{
    "jsonrpc": "2.0",
    "method": "getLedgers",
    "params": {
      "startLedger": 10000,
      "pagination": {
        "limit": 5
      }
    },
    "id": 1
  }' | jq


Step 7: Verify Datastore Integration

Check that the system is actually using the external datastore:
# Try to access a very old ledger that definitely won't be in local storage
curl -X POST http://localhost:8000 \
  -H 'Content-Type: application/json' \
  -d '{
    "jsonrpc": "2.0",
    "method": "getLedgers",
    "params": {
      "startLedger": 1000,
      "pagination": {
        "limit": 1
      }
    },
    "id": 1
  }' | jq '.result.ledgers[0].sequence'
If this returns ledger 1000, your datastore integration is working!
Troubleshooting

Common Issues


Authentication Errors:
error="dialing: google: error getting credentials"


Verify your credentials file exists and is readable
Try without credentials if the bucket is public


Config Validation Errors:
"history-retention-window must be positive"


Ensure HISTORY_RETENTION_WINDOW = 1 (not 0)


Captive Core Errors:
"setting BUCKET_DIR_PATH is disallowed for Captive Core"


Remove BUCKET_DIR_PATH from stellar-core.cfg
Use CAPTIVE_CORE_STORAGE_PATH in main config instead


Fee Stats Window Errors:
"Fee stat analysis window (50) cannot exceed history retention window (1)"


Set fee stats retention windows to 1 or lower


Useful Commands

# Stop and remove container
docker stop stellar-rpc-datastore
docker rm stellar-rpc-datastore

# View container logs
docker logs stellar-rpc-datastore

# Restart with new config
docker restart stellar-rpc-datastore

# Check container status
docker ps | grep stellar-rpc

# Clean up
docker stop stellar-rpc-datastore
docker rm stellar-rpc-datastore
rm -rf stellar-rpc-data
Configuration Reference

Key Configuration Options


SERVE_LEDGERS_FROM_DATASTORE: Enable external datastore (true/false)
HISTORY_RETENTION_WINDOW: Local ledger retention (minimum 1)
datastore_config.type: Datastore type ("GCS")
datastore_config.params.destination_bucket_path: GCS bucket path
buffered_storage_backend_config.buffer_size: In-memory ledger cache size
buffered_storage_backend_config.num_workers: Concurrent fetch workers

Performance Tuning

For better performance, adjust these values based on your needs:
[buffered_storage_backend_config]
buffer_size = 200      # Increase for more caching
num_workers = 20       # Increase for more concurrent fetches
retry_limit = 5        # Increase for unreliable networks
retry_wait = "10s"     # Increase retry delay
Next Steps

With your stellar-rpc instance running with datastore support, you can:

Integrate it with your applications using the JSON-RPC API
Query historical ledger data spanning the full network history
Build applications that need access to older transaction/event data
Scale horizontally without worrying about local storage limits

For production deployments, consider:

Using proper service account authentication
Setting up monitoring and alerting
Configuring appropriate resource limits
Using a reverse proxy for SSL termination
Implementing backup strategies for your local database
No results found