Thomas Mukei

## submit.md

      
              1 file
            
          
              0 forks
            
          
              0 comments
            
          
              0 stars
            
          
                Mukei
                / submit.md
            
            
              Created
              June 13, 2022 14:27
                — forked from tanaikech/submit.md
            
              
                Downloading Shared Files on Google Drive Using Curl
              
          
    Downloading Shared Files on Google Drive Using Curl

When the shared files on Google Drive is downloaded, it is necessary to change the download method by the file size. The boundary of file size when the method is changed is about 40MB.
File size < 40MB

CURL

filename="### filename ###"
fileid="### file ID ###"
curl -L -o ${filename} "https://drive.google.com/uc?export=download&amp;id=${fileid}"

  
## 20220206040024_memo.md

      
              1 file
            
          
              0 forks
            
          
              0 comments
            
          
              0 stars
            
          
                Mukei
                / 20220206040024_memo.md
            
            
              Last active
              February 6, 2022 08:44
            
          
    filter result based on images names containing "base" in their names

index = [index for index, im_similar in enumerate(im_similar) if "base" in im_similar]

im_similar = [im_similar[i] for i in index]
distances = [distances[0][i] for i in index]


## gcloud_cheat_sheet.md

      
              1 file
            
          
              0 forks
            
          
              0 comments
            
          
              0 stars
            
          
                Mukei
                / gcloud_cheat_sheet.md
            
            
              Created
              December 17, 2019 06:07
                — forked from pydevops/gcloud-cheat-sheet.md
            
              
                gcp gcloud cheat sheet
              
          
    Table of Contents


References
Other cheatsheets
multiple gcloud config configurations

switch gcloud context with gcloud config


auth
info
projects
zones


## cookbook_kaggle_in_colab.py
#source https://www.kaggle.com/general/64962

!pip install kaggle

#Authenticating with Kaggle API
from googleapiclient.discovery import build
import io, os
from googleapiclient.http import MediaIoBaseDownload
from google.colab import auth
auth.authenticateuser() driveservice = build('drive', 'v3')

## kaggle_dataset load.py
# To access kaggle datasets
!pip install kaggle

# Import data from Kaggle
# Colab's file access feature
from google.colab import files

# Retrieve uploaded file
uploaded = files.upload()

## conda_environment_rename.sh
#One workaround is to create clone environment, and then remove original one:
#(remember about deactivating current environment with deactivate on Windows and source deactivate on macOS/Linux)

conda create --name new_name --clone old_name --offline #use --offline flag to disable the redownload of all your packages
conda remove --name old_name --all # or its alias: `conda env remove --name old_name`

#There are several drawbacks of this method:
# time consumed on copying environment's files,
# temporary double disk usage.

## install_packages.sh
#
#  Original solution via StackOverflow:
#    http://stackoverflow.com/questions/35802939/install-only-available-packages-using-conda-install-yes-file-requirements-t
#

#
#  Install via `conda` directly.
#  This will fail to install all
#  dependencies. If one fails,
#  all dependencies will fail to install.

## useful_pandas_snippets.py
# List unique values in a DataFrame column
# h/t @makmanalp for the updated syntax!
df['Column Name'].unique()

# Convert Series datatype to numeric (will error if column has non-numeric values)
# h/t @makmanalp
pd.to_numeric(df['Column Name'])

# Convert Series datatype to numeric, changing non-numeric values to NaN
# h/t @makmanalp for the updated syntax!

## docker_cookbook.md

      
              1 file
            
          
              1 fork
            
          
              0 comments
            
          
              1 star
            
          
                Mukei
                / docker_cookbook.md
            
            
              Last active
              September 11, 2019 15:02
            
              
                [Docker Cookbook] #docker #cookbook
              
          
    original source: https://medium.com/statuscode/dockercheatsheet-9730ce03630d
Installation

Linux

curl -sSL https://get.docker.com/ | sh


## s3_copy_snippets.sh
#!
# Using aws s3 cp will require the --recursive parameter to copy multiple files.
aws s3 cp s3://myBucket/dir localdir --recursive

#The aws s3 sync command will, by default, copy a whole directory. It will only copy new/modified files.
aws s3 sync s3://mybucket/dir localdir
#Just experiment to get the result you want.

#You can combine --exclude and --include options to copy only objects that match a pattern, excluding all others.
aws s3 cp s3://mybucket/logs/ s3://mybucket2/logs/ --recursive --exclude "*" --include "*.log"
	#source https://www.kaggle.com/general/64962

	!pip install kaggle

	#Authenticating with Kaggle API
	from googleapiclient.discovery import build
	import io, os
	from googleapiclient.http import MediaIoBaseDownload
	from google.colab import auth
	auth.authenticateuser() driveservice = build('drive', 'v3')
	# To access kaggle datasets
	!pip install kaggle

	# Import data from Kaggle
	# Colab's file access feature
	from google.colab import files

	# Retrieve uploaded file
	uploaded = files.upload()
	#One workaround is to create clone environment, and then remove original one:
	#(remember about deactivating current environment with deactivate on Windows and source deactivate on macOS/Linux)

	conda create --name new_name --clone old_name --offline #use --offline flag to disable the redownload of all your packages
	conda remove --name old_name --all # or its alias: `conda env remove --name old_name`

	#There are several drawbacks of this method:
	# time consumed on copying environment's files,
	# temporary double disk usage.
	#
	# Original solution via StackOverflow:
	# http://stackoverflow.com/questions/35802939/install-only-available-packages-using-conda-install-yes-file-requirements-t
	#

	#
	# Install via `conda` directly.
	# This will fail to install all
	# dependencies. If one fails,
	# all dependencies will fail to install.
	# List unique values in a DataFrame column
	# h/t @makmanalp for the updated syntax!
	df['Column Name'].unique()

	# Convert Series datatype to numeric (will error if column has non-numeric values)
	# h/t @makmanalp
	pd.to_numeric(df['Column Name'])

	# Convert Series datatype to numeric, changing non-numeric values to NaN
	# h/t @makmanalp for the updated syntax!
	#!
	# Using aws s3 cp will require the --recursive parameter to copy multiple files.
	aws s3 cp s3://myBucket/dir localdir --recursive

	#The aws s3 sync command will, by default, copy a whole directory. It will only copy new/modified files.
	aws s3 sync s3://mybucket/dir localdir
	#Just experiment to get the result you want.

	#You can combine --exclude and --include options to copy only objects that match a pattern, excluding all others.
	aws s3 cp s3://mybucket/logs/ s3://mybucket2/logs/ --recursive --exclude "" --include ".log"