| Provider | Configuration | Cost ($/h) |
|----------|------------------------|------------|
| GCP | n1-standard-16 with P4 | 1.36 |
| Azure | NV6 with M60 | 1.14 |
| AWS | g3.4xlarge with M60 | 1.14 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# {% include 'template/license_header' %} | |
from typing import Tuple | |
from typing_extensions import Annotated | |
import pandas as pd | |
from sklearn.model_selection import train_test_split | |
from zenml import step | |
import random |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
""" | |
To be used with the ZenML SagemakerOrchestrator | |
https://github.com/zenml-io/zenml/blob/main/src/zenml/integrations/aws/orchestrators/sagemaker_orchestrator.py | |
as a small duck typed juke for the sagemaker network config object: | |
https://github.com/aws/sagemaker-python-sdk/blob/master/src/sagemaker/network.py#L24 | |
""" | |
from __future__ import absolute_import | |
from typing import Union, Optional, List |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
"""Materializer for Pandas CSV.""" | |
import os | |
import tempfile | |
from typing import Any, Type, Union | |
import pandas as pd | |
import numpy as np | |
from zenml.artifacts import DataArtifact, SchemaArtifact, StatisticsArtifact |
| Provider | Configuration | Cost ($/h) | Spot Cost ($/h) | Cost Savings |
|----------|------------------------|------------|-----------------|--------------|
| GCP | n1-standard-16 with P4 | 1.36 | 0.38 | 72% |
| Azure | NV6 with M60 | 1.14 | 0.20 | 82% |
| AWS | g3.4xlarge with M60 | 1.14 | 0.34 | 70% |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
training_pipeline.run( | |
backend=OrchestratorGCPBackend( | |
preemptible=True, # reduce costs by using preemptible instance | |
machine_type='n1-standard-4', | |
gpu='nvidia-tesla-k80', | |
gpu_count=1, | |
) | |
) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import pandas as pd | |
from time import time | |
from pprint import pprint | |
import simplejson | |
import json | |
import ujson | |
def _loads(data, serializer): | |
t_start = time() |