Skip to content

Instantly share code, notes, and snippets.

View bigorn0's full-sized avatar
💭
Enthousiast

Ugo Bechameil bigorn0

💭
Enthousiast
View GitHub Profile

How to split/redirect part of the logs based on level

apache-spark-stderr-and-stdout

Try this in log4j.properties passed to Spark (or modify default configuration under Spark/conf)

# Log to stdout and stderr
log4j.rootLogger=INFO, stdout, stderr
PRO TIP: Transfer git repo without `.git` folder using tar.gz | ssh `git archive HEAD | gzip -c | ssh <host> 'tar -zxvf -'
bash-4.3# curl -4 \
--insecure \
-sSf \
-w '\nLookup time:\t%{time_namelookup}\nConnect time:\t%{time_connect}\nAppCon time:\t%{time_appconnect}\nRedirect time:\t%{time_redirect}\nPreXfer time:\t%{time_pretransfer}\nStartXfer time:\t%{time_starttransfer}\n\nTotal time:\t%{time_total}\n' \
-o /dev/null
-XPUT https://my-endpoint