Charmy Feng fcharmy
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| FROM python:3.7 | |
| RUN wget https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-4.0.2.tgz | |
| RUN tar -zxvf mongodb-*.tgz | |
| RUN rm mongodb-*.tgz | |
| RUN mkdir -p /data/db | |
| ENV PATH="/mongodb-linux-x86_64-4.0.2/bin:${PATH}" | |
| EXPOSE 27017 | |
| ENTRYPOINT ["mongod"] |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import logging | |
| from sqlalchemy import create_engine, MetaData, Table | |
| from sqlalchemy.engine import reflection | |
| from sqlalchemy.orm import mapper, sessionmaker | |
| logger = logging.getLogger(__name__) | |
| def get_session(host=None, port=3306, db=None, |
fcharmy
/ sync_git.sh
Created
September 13, 2018 09:02
auto sync github repository for whole repo or given certain directories
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #!/usr/bin/env bash | |
| # To keep updating codes from git repository | |
| # | |
| # feng.charmy@gmail.com | |
| # Nothe: please make sure git config has read permission of repository | |
| # | |
| # Usage ./sync_dags.sh git@github.com:user/repo.git branch_name\ | |
| # -t interval_in_seconds\ | |
| # -d sparse_checkout_directory(optional) -d more_sparse_checkout_directory ..\ | |
| # -e execute_command_after_sync |
fcharmy
/ ig_scrapy_spider.py
Last active
September 26, 2018 10:04
Crawl IG hashtag posts including post link, main images url, thumbnail, is_video, username, likes, caption, posted time, user profile image url. Only show the spider implementation, pipeline need to be implement separately.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import json | |
| import urllib.parse as urlparse | |
| from scrapy import Item, Field, Spider, Request | |
| from scrapy.exceptions import CloseSpider | |
| # ------ Instagram settings ------ | |
| IG_SERVER = "https://www.instagram.com" | |
| IG_PATH = '/explore/tags/' | |
| IG_POST_PATH = "p/" |
fcharmy
/ prec_rec_f1_request.json
Last active
November 2, 2018 09:19
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| { | |
| "metric": [ | |
| "prec_rec_f1" | |
| ], | |
| "image": [ | |
| { | |
| "image": { | |
| "im_id": "xxx", | |
| "width": 512, | |
| "height": 512 |
fcharmy
/ prec_rec_f1_response.json
Created
November 2, 2018 09:16
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| { | |
| "overview": { | |
| "concept_group_names": [ | |
| "category", | |
| "sleeve_length", | |
| "pattern" | |
| ], | |
| "metrics": { | |
| "prec_rec_f1": { | |
| "type": "table", |
fcharmy
/ acc_1_request.json
Created
November 2, 2018 09:20
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| { | |
| "metric": [ | |
| "acc_1" | |
| ], | |
| "image": [ | |
| { | |
| "image": { | |
| "im_id": "xxx", | |
| "width": 512, | |
| "height": 512 |
fcharmy
/ acc_1_response.json
Created
November 2, 2018 09:21
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| { | |
| "overview": { | |
| "concept_group_names": [ | |
| "category", | |
| "sleeve_length", | |
| "pattern" | |
| ] | |
| }, | |
| "results": [ | |
| { |
fcharmy
/ confusion_matrix_request.json
Last active
November 2, 2018 09:23
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| { | |
| "metric": [ | |
| "confusion_matrix" | |
| ], | |
| "image": [ | |
| { | |
| "image": { | |
| "im_id": "xxx", | |
| "width": 512, | |
| "height": 512 |
fcharmy
/ pr_curve_request.json
Created
November 2, 2018 09:24
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| { | |
| "metric": [ | |
| "pr_curve" | |
| ], | |
| "image": [ | |
| { | |
| "image": { | |
| "im_id": "xxx", | |
| "width": 512, | |
| "height": 512 |
OlderNewer