Skip to content

Instantly share code, notes, and snippets.

View zeryx's full-sized avatar

zeryx zeryx

View GitHub Profile
[{ "ClaimId ":5355879, "CciliId ":9807378, "PetPolicyId ":1019191, "OwnerId ":847019, "Ded ":250.0000, "RiderInEffect ":None, "DaysInEffect ":None, "dateclosed ":None, "Inserted ": "2020-04-29T06:23:44.277 ", "DateOfLoss ": "2020-04-29T04:00:00 ", "DaysEnrollLoss ":1196, "ItemDescription ": "DALLAS: EXAMINATION- RECHECK ", "TrialFlag ":1, "PolicyPetName ": "DALLAS ", "InvoicePetName ": "DALLAS ", "DiagnosisOne ": "DIAGNOSTICS ", "DiagnosisTwo ": " ", "PawprintStatus ": "NoPrex ", "SalesTaxFlag ":0, "Subtotal ":113.7500, "ActualExtendedPrice ":36.0000, "DiscountFlag ":0, "PaymentFlag ":0, "LineItemDupFlag ":0, "DuplicateClaimId ":None, "TaxCovered ":0, "Eligible ":0.0, "PastCondList ": "ABRASION,GASTROENTERITIS,RETAINED DECIDUOUS TOOTH ", "TransactionDate ": "2020-04-29T04:00:00 ", "EnrollmentDate ": "2017-01-31T20:12:59.857 ", "TrialDate ": "2017-01-19T15:25:15.203 ", "FirstSymptomDate ": "2020-04-15T04:00:00 ", "RelatedClaim ":5314280, "AgeDays ":1196, "PolicyStartDate ": "2017-01-19T00:00:00 "},{ "ClaimId
import torch as th
from torchvision import models, transforms
from PIL import Image
import numpy as np
resnet18 = models.resnet18(pretrained=True)
# This stage below removes the last module (aka the softmax layer) to provide raw data for fine-tuning/transfer learning
modules=list(resnet18.children())[:-1]
model = th.nn.Sequential(*modules)
th.save(model, "/tmp/resnet18-headless.th")
@zeryx
zeryx / requirements.txt
Created March 13, 2020 19:33
Example of how to use concurrency to process inputs and work jobs in paralle for batch processes
algorithmia>=1.0.0,<2.0
six
tensorflow-gpu==1.2.0

Steps required to deploy a new dependency package

Langpacks

  1. Create new langpacks branch, from latest master
    • eg: git checkout -b ALGO-xxx-package-name
  2. create new directory in libraries
    • eg: package-name-version
  3. create new Dockerfile in new package directory
  4. copy existing dependency template to new directory
  • eg: cp -LrR templates/package-name-version-1 templates/package-name-version

Already done

  • ResNet 18 - Image Classification - ImageNet (not available for tensorflow)
  • ResNet 50 - Image Classification - ImageNet

Suggestions

  • MobileNet v2 - Image Classification
  • Why
@zeryx
zeryx / auxillary.py
Created December 3, 2019 23:02
image classification with recursion example
import tensorflow as tf
import Algorithmia
import os
import re
import numpy as np
TEMP_COLLECTION = 'data://.session/'
MODEL_FILE = "data://zeryx/InceptionNetDemo/classify_image_graph_def.pb"
CONVERSION_FILE = "data://zeryx/InceptionNetDemo/imagenet_synset_to_human_label_map.txt"
@zeryx
zeryx / async_algorithm.py
Created December 3, 2019 22:56
image classification with asyncronous processing
import Algorithmia
from .auxillary import load_model, id_to_string
import tensorflow as tf
import asyncio
import os, re
import numpy as np
# Global variables here
client = Algorithmia.client()
@zeryx
zeryx / auxillary.py
Created December 3, 2019 22:52
basic algorithm, made in python 3.6
import tensorflow as tf
import Algorithmia
import os
import re
import numpy as np
TEMP_COLLECTION = 'data://.session/'
MODEL_FILE = "data://zeryx/InceptionNetDemo/classify_image_graph_def.pb"
CONVERSION_FILE = "data://zeryx/InceptionNetDemo/imagenet_synset_to_human_label_map.txt"

Algorithm Parallelism and Threading Tutorial

This tutorial is primarily targetted at an advanced audience looking for solutions to improve runtime performance of already existing algorithms. If you'd like to find the introductory tutorials please check this page first.

You might even be making requests to multiple downstream algorithms from within your current algorithm, reusing building blocks to make the most out of the platform. For example:

import Algorithmia
import numpy as np
@zeryx
zeryx / algo_orchestrator.py
Created November 26, 2019 21:30
An orchestration algorithm that asynronously waits for the outputs of 3 different algorithms in batch to finish before returning a result
import Algorithmia
from multiprocessing import Manager, Pool
# API calls will begin at the apply() method, with the request body passed as 'input'
# For more details, see algorithmia.com/developers/algorithm-development/languages
ALGO_1 = "algorithmiahq/DeepFashion/1.3.0"
ALGO_2 = "algorithmiahq/multistageclassifierpetsdemo/0.1.0"
ALGO_3 = "character_recognition/tesseract/0.3.0"
# This threadpool is shared between all 3 algorithm requests, this keeps the number of active children restricted and easily controlled