Skip to content

Instantly share code, notes, and snippets.

View cortex.yaml
- name: elegy-api # API name (required)
kind: RealtimeAPI
predictor:
type: python
path: predict.py # path to a python file with a PythonPredictor class definition, relative to the Cortex root (required)
config: # Arbitrary properties, accessible in predictor through config parameter
model_s3_path: xxxxxx
View predictor.py
import elegy
from lib import s3_download
class PythonPredictor:
def __init__(self, config):
self.model_path = s3_download(config.model_s3_path) # Download model from S3
self.model = elegy.model.load(self.model_path) # Initialize model
def predict(self, payload):
return self.model.predict(payload["input"])
View server-side-example.py
class PythonPredictor:
def __init__(self, config):
self.model = # Download model from any source (can also be configured in API spec).
def predict(self, payload):
return self.model(payload) # payload is now a list of payloads
View serverside.yaml
- name: <string> # API name (required)
kind: RealtimeAPI
predictor:
type: python
path: <string> # path to a python file with a PythonPredictor class definition, relative to the Cortex root (required)
server_side_batching: # (optional)
max_batch_size: <int> # the maximum number of requests to aggregate before running inference
batch_interval: <duration> # the maximum amount of time to spend waiting for additional requests before running inference on the batch of requests
View translator-example-api.yaml
- name: translator
kind: RealtimeAPI
predictor:
type: python
path: predictor.py
models:
dir: "s3://models.huggingface.co/bert/Helsinki-NLP/"
cache_size: 5
compute:
gpu: 1
View translator-example-cluster.yaml
# EKS cluster name for cortex (default: cortex)
cluster_name: cortex
# AWS region
region: us-east-1
# instance type
instance_type: g4dn.xlarge
# minimum number of instances (must be >= 0)
View example-cortex.yaml
- name: your-api
kind: RealtimeAPI
predictor:
type: python
path: predictor.py
compute:
cpu: 1
View metafflow-predictor.py
# Import whatever dependencies your API needs
import os
from metaflow import DataArtifact
os.environ['METAFLOW_USER'] = 'cortex'
class PythonPredictor:
def __init__(self, config):
# Initialize your model here
View metaflow_train.py
import os
from metaflow import FlowSpec, current, step
def select_model(a, b):
if a > b:
return a
return b
class TrainingFlow(FlowSpec):
View metaflow.py
import os
from metaflow import FlowSpec, current, step
import cortex
def select_model(a, b):
if a > b:
return a
return b