Skip to content

Instantly share code, notes, and snippets.

Avatar
💭
Trying to ML

Morgan McGuire morganmcg1

💭
Trying to ML
View GitHub Profile
@morganmcg1
morganmcg1 / DBPlot.py
Created Jun 21, 2022
Code for DBPlot
View DBPlot.py
# Decision Boundary Utils
-# Modified from https://github.com/tmadl/highdimensional-decision-boundary-plot
-class DBPlot(BaseEstimator):
- def __init__(
- self,
- estimator=KNeighborsClassifier(n_neighbors=10),
- acceptance_threshold=0.03,
- n_decision_boundary_keypoints=60,
- n_connecting_keypoints=None,
- n_interpolated_keypoints=None,
@morganmcg1
morganmcg1 / tables_logging.py
Created Mar 8, 2022
Logging images to Tables with references
View tables_logging.py
# First table/ table for first iteration/epoch
original_table = ... # format [id, wandb.Image object, ...]
artifact.add(original_table, "my_table")
# Log the table as artifact. (optionally)Log the same table in dashboard if you want to
run.log_artifact(artifact)
run.log({"my_table": original_table})
# Use the logged artifact to get the reference of table
run.use_artifact(artifact)
View yea-test
#!/usr/bin/env python
"""Test a batch of import telemetry
---
id: 0.imports.04-batch4
tag:
shard: imports
plugin:
- wandb
depend:
View yea_test
paperspace@pspx59qcl:~/client$ yea run functional_tests/imports/04-batch4.py
INFO: Timed out waiting for server to start... http://localhost:37211 1645464371.6856294
INFO: Timed out waiting for server to start... http://localhost:37211 1645464372.6893241
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 159, in _new_conn
conn = connection.create_connection(
File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 84, in create_connection
raise err
File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 74, in create_connection
sock.connect(sa)
@morganmcg1
morganmcg1 / artifacts_test.ipynb
Created Aug 26, 2021
artifacts download fails
View artifacts_test.ipynb
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@morganmcg1
morganmcg1 / gist:0e4344df49fe3b43243505992ce998d5
Last active Aug 9, 2021
gpt-j generation error stacktrace
View gist:0e4344df49fe3b43243505992ce998d5
2021-08-09 13:33:24.972717: E external/org_tensorflow/tensorflow/compiler/xla/pjrt/pjrt_stream_executor_client.cc:1981] Execution of replica 2 failed: Resource exhausted: Attempting to reserve 4.44G at the bottom of memory. That was not possible. There are 9.62G free, 0B reserved, and 2.65G reservable.
2021-08-09 13:33:24.972816: E external/org_tensorflow/tensorflow/compiler/xla/pjrt/pjrt_stream_executor_client.cc:1981] Execution of replica 5 failed: Resource exhausted: Attempting to reserve 4.44G at the bottom of memory. That was not possible. There are 9.62G free, 0B reserved, and 2.65G reservable.
2021-08-09 13:33:24.972875: E external/org_tensorflow/tensorflow/compiler/xla/pjrt/pjrt_stream_executor_client.cc:1981] Execution of replica 3 failed: Resource exhausted: Attempting to reserve 4.44G at the bottom of memory. That was not possible. There are 9.62G free, 0B reserved, and 2.65G reservable.
2021-08-09 13:33:24.972954: E external/org_tensorflow/tensorflow/compiler/xla/pjrt/pjrt_stream_executor_client.c
@morganmcg1
morganmcg1 / deepchem_wandb.py
Last active Jul 28, 2021
DeepChem W&B Minimal Examples
View deepchem_wandb.py
#!/usr/bin/env python
"""Test Optuna integration
---
id: 0.0.4
check-ext-wandb: {}
assert:
- :wandb:runs_len: 1
- :wandb:runs[0][project]: integrations_testing
- :wandb:runs[0][config][a]: 2
- :wandb:runs[0][config][b]: testing
@morganmcg1
morganmcg1 / create_tfrecords_prosecraft.py
Last active Jul 26, 2021
Modified create_tfrecords from GPT-Neo repo
View create_tfrecords_prosecraft.py
import argparse
import os
from pathlib import Path
import ftfy
import tensorflow as tf
from lm_dataformat import Reader
from tokenizers import Tokenizer
from transformers import GPT2TokenizerFast
from tqdm import tqdm
View download_s3_data.py
import boto3
import os
import pickle
my_bucket = "prosecraft-manuscript-archives"
my_file = "manuscript-samples-2021-05-06.zip"
s3 = boto3.resource('s3')
obj = s3.Object(my_bucket, my_file)
body = obj.get()['Body'].read()
View upload_to_s3.py
import boto3, os
my_bucket = "prosecraft-language-models"
folder_name = "manuscript-samples"
for f in files:
boto3.Session().resource('s3').Bucket(my_bucket).Object(
os.path.join(folder_name, f"{f.split('/')[-2]}_{f.split('/')[-1]}")).upload_file(f)