avinash-mishra

#! /usr/bin/env python

#
# mysql2cassandra.py
#   Dump a MySQL result set to file and then import into a Cassandra column family
# 
# Configuration
#   mysql_params		[host, port, user, password, db]		MySQL conenction parameters
#   mysql_columns		[colname, colname2, ...]				Columns for building MySQL query
#																	The column that will hold values of the row key in the Cassandra column family must be first

avinash-mishra

Problem:
$ sudo apt-get update
..........
..........
Err:13 http://extras.ubuntu.com/ubuntu xenial/main Sources                                                        
  404  Not Found [IP: 91.189.92.152 80]
Ign:17 http://extras.ubuntu.com/ubuntu xenial/main amd64 Packages                                                 
Ign:20 http://extras.ubuntu.com/ubuntu xenial/main i386 Packages                                                  
Ign:22 http://extras.ubuntu.com/ubuntu xenial/main all Packages                                                   
Ign:23 http://extras.ubuntu.com/ubuntu xenial/main Translation-en_US                                              

avinash-mishra

How to create a MultiNode - MultiBroker Cluster for Kafka on AWS

PreRequisites :

1. Kafka Binary files : http://kafka.apache.org/downloads.html

2. Atleast 2 AWS machines : AWS EMR or EC2 will be preferable

3. A Kafka Manager Utility to watch up the Cluster : https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem

avinash-mishra

Cassandra 2.x Cluster on AWS EC2

Install java 8

# Remove java 7
sudo yum remove -y java

# Install basic packages
sudo yum install -y git

avinash-mishra

[
   {
   "time": "03:53:25 AM",
   "milliseconds_since_epoch": 1362196405309,
   "date": "03-02-2013",
   "name": "alphabet"
   },
   {
   "time": "03:53:26 AM",
   "milliseconds_since_epoch": 1362196405300,

avinash-mishra

"""A setuptools based setup module.
See:
https://packaging.python.org/en/latest/distributing.html
https://github.com/pypa/sampleproject
"""

# Always prefer setuptools over distutils
from setuptools import setup, find_packages
# To use a consistent encoding
from codecs import open

avinash-mishra

"""
Install dependencies before running : https://dash.plot.ly/installation
"""

import dash
from dash.dependencies import Input, Output
import dash_core_components as dcc
import dash_html_components as html
import pandas as pd
pd.core.common.is_list_like = pd.api.types.is_list_like

avinash-mishra

S3 Syncing

• Export out all your S3 data
• Source: http://docs.aws.amazon.com/cli/latest/reference/s3/sync.html

# sync from bucket to local
aws s3 sync <bucket> <target_folder> <options>
aws s3 sync s3://mybucket . --acl public-read

aws s3 sync --region ap-northeast-1 s3://[移動元バケット名] s3://[移動先バケット名]

avinash-mishra

Machine Learning

• データ加工を無料で学べる「データサイエンス100本ノック」　データサイエンティスト協会がGitHubに公開：データサイエンスの「基礎的なデータ処理の修行場」として - ＠IT
• 「機械学習の最先端」を効率的に情報収集！　おすすめのメルマガ3選：AI・機械学習の独学リソース - ＠IT
• データサイエンス初学者向け学習環境「データサイエンス100本ノック（構造化データ加工編）」が公開：CodeZine（コードジン）
• 機械学習で使われる評価関数まとめ - Qiita
• ML/DL Engineering Made Easy with PyTorch’s Ecosystem Tools | by PyTorch | PyTorch | Medium
• ライブラリーを使わずにPythonでニューラルネットワークを構築してみる - Qiita
• 深層強化学習と汎用化が重要、ロボットへの応用が面白い　Googleブレイン Shane Gu氏の基調講演「Deep Learning Digital Conference」 | ロボスタ
• [新たな活性化関数「FReLU」誕生&解説！ - Qiita](https://qiita.com/omiita/items/bfbba77559762405698