Skip to content

Instantly share code, notes, and snippets.

View WGET_Large_Files_GDrive.md

This document walks you through the steps to prepare a wget compatible link from a file that is located in your Google Drive.

Motivation: When working in Deep Learning, we often use Google Colab, Kaggle Kernels, or Cloud Instances for training our models on GPUs. But the problem that comes with it is we often have to upload all the necessary files required to get things up and running. This is particularly problematic when we have a large dataset and this cannot be uploaded/gathered directly (sometimes, scp does not work as well). We may have a dataset stored in our Google Drives. In situations like that, we generally create a wget compatible link from the file (typically the dataset) located in our Google Drive (this document only deals with Google Drive).

Steps:

  • Right click on the file (located in Google Drive) and click on "Share".
  • In the Link sharing on section, change the permissions of your file to "Anyone with the link can view" and copy the link.
  • Now, the link should resemble `
@dabeaz
dabeaz / README.txt
Created Oct 15, 2019
PyCon India 2019, Code from Keynote Presentation by @dabeaz
View README.txt
Code from PyCon India 2019 Keynote Talk
David Beazley (https://www.dabeaz.com)
======================================
This code is presented "as is" and represents what was live-coded
during my closing keynote presentation at PyCon India, Chennai,
October 13, 2009. I have made no changes to the files.
Requires: Python 3.6+, numpy, pygame
View gist:d88f9ef887f879c7be71fb40ae943a94
create table rank (id UInt64, rank_column LowCardinality(String), c0 AggregateFunction(sum, Int32))
engine = AggregatingMergeTree order by id;
create table nop (id UInt64, rank_column LowCardinality(String), c Int32) Engine= Null;
create materialized view nop_mv to rank as select id, rank_column, sumState(c) as c0 from nop group by id, rank_column
insert into nop select number id, toString(id % 150000), toInt32(rand() % 4234234) from numbers(100000000)
optimize table rank final;
@den-crane
den-crane / gist:f7382cd4f1f859ff6ac46afe7dc9925a
Created Oct 15, 2018
Populate AggregatingMergeTree through null table
View gist:f7382cd4f1f859ff6ac46afe7dc9925a
create table z(a date, b Int64) Engine=MergeTree Partition by toYYYYMM(a) order by a;
insert into z select today(), number from numbers(1000000000);
insert into z select yesterday(), number from numbers(1000);
create table mv_z_store(a date, max_b AggregateFunction(MAX,Int64)) ENGINE = AggregatingMergeTree Partition by toYYYYMM(a) order by a;
create table temp(a date, b Int64) Engine=Null;
create MATERIALIZED VIEW mv_z to mv_z_store AS SELECT a, maxState(b) AS max_b FROM temp GROUP BY a;
insert into temp select * from z;
drop table mv_z;
drop table temp;
@geohot
geohot / ransac_polyfit.py
Last active Mar 17, 2021
RANSAC polyfit. Fit polynomials with RANSAC in Python
View ransac_polyfit.py
def ransac_polyfit(x, y, order=3, n=20, k=100, t=0.1, d=100, f=0.8):
# Thanks https://en.wikipedia.org/wiki/Random_sample_consensus
# n – minimum number of data points required to fit the model
# k – maximum number of iterations allowed in the algorithm
# t – threshold value to determine when a data point fits a model
# d – number of close data points required to assert that a model fits well to data
# f – fraction of close data points required
besterr = np.inf
@jkleint
jkleint / timeseries_cnn.py
Created Jul 29, 2016
Example of using Keras to implement a 1D convolutional neural network (CNN) for timeseries prediction.
View timeseries_cnn.py
#!/usr/bin/env python
"""
Example of using Keras to implement a 1D convolutional neural network (CNN) for timeseries prediction.
"""
from __future__ import print_function, division
import numpy as np
from keras.layers import Convolution1D, Dense, MaxPooling1D, Flatten
from keras.models import Sequential
@karpathy
karpathy / min-char-rnn.py
Last active Jul 30, 2021
Minimal character-level language model with a Vanilla Recurrent Neural Network, in Python/numpy
View min-char-rnn.py
"""
Minimal character-level Vanilla RNN model. Written by Andrej Karpathy (@karpathy)
BSD License
"""
import numpy as np
# data I/O
data = open('input.txt', 'r').read() # should be simple plain text file
chars = list(set(data))
data_size, vocab_size = len(data), len(chars)
@ndarville
ndarville / webm.md
Last active May 19, 2021
4chan’s guide to converting GIF to WebM - https://boards.4chan.org/g/res/41212767
View webm.md

Grab ffmpeg from https://www.ffmpeg.org/download.html

It's a command line tool which means you will have to type things with your keyboard instead of clicking on buttons.

The most trivial operation would be converting gifs:

ffmpeg -i your_gif.gif -c:v libvpx -crf 12 -b:v 500K output.webm
  • -crf values can go from 4 to 63. Lower values mean better quality.
  • -b:v is the maximum allowed bitrate. Higher means better quality.
View gist:4525165
-- Create a function withn the security set to Definer so that it can insert
CREATE OR REPLACE FUNCTION cdb_insert_point(float, float) RETURNS integer
AS 'INSERT INTO example_table(the_geom) VALUES(ST_SetSRID(ST_MakePoint($1,$2),4326)) RETURNING cartodb_id;'
LANGUAGE SQL
SECURITY DEFINER
RETURNS NULL ON NULL INPUT;
--Grant access to the public user
GRANT EXECUTE ON FUNCTION cdb_insert_point(float,float) TO publicuser;
@mbostock
mbostock / .block
Last active May 27, 2021
Path Tween
View .block
license: gpl-3.0