Skip to content

Instantly share code, notes, and snippets.

View ftorto's full-sized avatar

ftorto ftorto

  • TOULOUSE, France
View GitHub Profile
@ftorto
ftorto / find.sh
Created April 6, 2017 12:57
Find files with only one line
find . -type f -exec awk 'END { if (NR > 1) print FILENAME }' {} \;
@ftorto
ftorto / parallel_limit.sh
Created February 24, 2017 07:21
Parallelize actions with max parallel count
function parallel_function {
# something
}
max_parallel_jobs=4
while true
do
parallel_function &
((start_date=start_date+1))
while [ $(jobs | wc -l) -ge $max_parallel_jobs ] ; do sleep 1 ; done
@ftorto
ftorto / diff.sh
Created February 24, 2017 06:52
Compare 2 lists
# List common lines in $1 and $2
function common {
comm --check-order -12 $1 $2
}
# List lines only present in $1
function onlyina {
comm --check-order -3 $1 $2 | sed '/\t/d'
}
input=$1
result="${input/find/replacement}"
@ftorto
ftorto / singleton.py
Created February 21, 2017 20:16
Singleton python
class Singleton(type):
_instances = {}
def __call__(cls, *args, **kwargs):
if cls not in cls._instances:
cls._instances[cls] = super(Singleton, cls).__call__(*args, **kwargs)
return cls._instances[cls]
#Python2
class MyClass(BaseClass):
__metaclass__ = Singleton
@ftorto
ftorto / git_recursive.sh
Last active February 9, 2017 15:13
Apply a command to all sub git directories
#Git recursive
# Apply a command to all sub git directories
gr() {
rootpath=`pwd`/
cmd=$*
if [ $# -eq 0 ]; then
echo "Multiline (Add empty line to end the input)"
@ftorto
ftorto / sum.sh
Created February 6, 2017 07:02
Sum all lines (1 value per line)
awk '{s+=$1} END {printf "%.0f", s}' mydatafile
@ftorto
ftorto / Spark.md
Last active January 31, 2017 12:10
Spark Transformations examples

Map

>>> rdd = sc.parallelize([1,2,3])
>>> rdd.map(lambda x: [x,x*10])
RDD: [1,2,3] -> [[1,10], [2,20], [3,30]]

>>> rdd.flatMap(lambda x: [x,x*10])
RDD: [1,2,3] -> [[1,10,2,20,3,30]]
@ftorto
ftorto / README.md
Created January 30, 2017 20:31
Python same namespace in different folders
Package-1/namespace/__init__.py
Package-1/namespace/module1/__init__.py
Package-2/namespace/__init__.py
Package-2/namespace/module2/__init__.py

with

  • Package-1
  • Package-2
@ftorto
ftorto / top10BigFiles.sh
Created January 24, 2017 10:26
Get the top10 of the biggest folder/files
du -hsx * | sort -rh | head -10