theSekyi

Architecting for the cloud

• Architect solutions to be scalable and reilient
• Business requirements involve lowering costs / enhancing user experience
• Keep an eye on technical needs during development and operation

---

3 Major Questions To Ask

1. Where is the company coming from

theSekyi

# STEP 1: Load

# Load documents using LangChain's DocumentLoaders
# This is from https://langchain.readthedocs.io/en/latest/modules/document_loaders/examples/csv.html

from langchain.document_loaders.csv_loader import CSVLoader
loader = CSVLoader(file_path='./example_data/mlb_teams_2012.csv')
data = loader.load()

theSekyi

Setting up github and bitbucket on the same computer (Windows)

Guide for Windows

mix3d asked for some help using this guide with windows so here we go. This was tested with Windows 10. Run all commands in Git Bash once it's installed.

Github will be the main account and bitbucket the secondary.

Git for Windows

• Download and install Git for Windows
  • In the installer, select everything but decide if you want a desktop icon (2nd step)

theSekyi

#IRC Reference

Not intended as a guide for newbies, more like a "cheat sheet" for the somewhat experienced IRC user, especially one who wields some power over a channel.

##The Basics

• /join #channel
  • Joins the specified channel.
• /part #channel
  • Leaves the specified channel.

theSekyi

import pandas as pd
import csv

import openrouteservice
from openrouteservice.directions import directions
import uuid
import time


em = "#"

theSekyi

def static_features(t0):
    """Get features such as current weekday and time 
    """
    weekday = t0.dt.dayofweek
    
    return weekday

t0 = pd.to_datetime('2019-01-07 06:05:00')
a = static_features(t0)

theSekyi

Traceback (most recent call last):
  File "/opt/conda/bin/luigi", line 10, in <module>
    sys.exit(luigi_run())
  File "/opt/conda/lib/python3.6/site-packages/luigi/cmdline.py", line 9, in luigi_run
    run_with_retcodes(argv)
  File "/opt/conda/lib/python3.6/site-packages/luigi/retcodes.py", line 70, in run_with_retcodes
    with luigi.cmdline_parser.CmdlineParser.global_instance(argv):
  File "/opt/conda/lib/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "/opt/conda/lib/python3.6/site-packages/luigi/cmdline_parser.py", line 52, in global_instance

theSekyi

import numpy as np
import matplotlib.pyplot as plt 
from math import pi,sin





def sinc(my_list_of_x):
    '''

theSekyi

DEBUG: Checking if CleanData(out_dir=data/interim/) is complete
INFO: Informed scheduler that task   CleanData_data_interim__ced0aed8fc   has status   PENDING
INFO: Done scheduling tasks
INFO: Running Worker with 1 processes
DEBUG: Asking scheduler for work...
DEBUG: Pending tasks: 1
INFO: [pid 11417] Worker Worker(salt=790270365, workers=1, host=lols-MacBook-Pro.local, username=socrates, pid=11417) running   CleanData(out_dir=data/interim/)
ERROR: [pid 11417] Worker Worker(salt=790270365, workers=1, host=lols-MacBook-Pro.local, username=socrates, pid=11417) failed    CleanData(out_dir=data/interim/)
Traceback (most recent call last):
  File "/Users/socrates/.local/share/virtualenvs/pipelines-O1OuTu0F/lib/python3.7/site-packages/luigi/task.py", line 851, in getpaths

theSekyi

$ docker-compose up orchestrator
code-challenge-2019_dask-scheduler_1 is up-to-date
code-challenge-2019_luigid_1 is up-to-date
Recreating code-challenge-2019_orchestrator_1 ... done
Attaching to code-challenge-2019_orchestrator_1
orchestrator_1    | DEBUG: Checking if MakeDatasets(no_remove_finished=False, out_dir=/usr/share/data/interim/) is complete
orchestrator_1    | DEBUG: Checking if DownloadData(no_remove_finished=False, fname=wine_dataset, out_dir=/usr/share/data/raw/, url=https://github.com/datarevenue-berlin/code-challenge-2019/releases/download/0.1.0/dataset_sampled.csv) is complete
orchestrator_1    | INFO: Informed scheduler that task   MakeDatasets_False__usr_share_data__006671ec2e   has status   PENDING
orchestrator_1    | INFO: Informed scheduler that task   DownloadData_wine_dataset_False__usr_share_data__79bc385f2e   has status   PENDING
orchestrator_1    | INFO: Done scheduling tasks