scr512

#!/bin/zsh
#Stats->Graphite
#10/27/16 Added boot wear monitoring. Removed Dashing stuff.
#02/27/16 Added username specific latency and operation collection. Pushing this data to both Graphite and Dashing
#03/29/15 Added in more accurate tracking of tier capacity via diskpools_to_json.py
#09/24/14 Added in queries for SmartPool capacity tracking
#08/22/14 Added in NFSv3 latency stats

#Let's first look and see if we have a stuck execution of the script. If the lock is older than 3 minutes we'll kill the PID and cleanup the lock.
find /tmp/send_stats_to_carbon.lck -mmin +3 -exec kill -9 `cat /tmp/send_stats_to_carbon.lck` {} \;

jvns

perf!

Yesterday we talked about perf. Let's start using perf!

I learned how to make flame graphs with perf today and it is THE BEST. I found this because Graydon Hoare pointed me to Brendan Gregg's excellent page on how to use perf.

Wait up! What's perf? I've talked about strace a lot before (in Debug your programs like they're closed source). strace lets you see which system calls a program is calling. But what if you wanted to know

• how many CPU instructions it ran?
• How many L1 cache misses there were?

vinitkumar

Architectural Questions

• Are responsibilities well defined?
• Are the collaborations well defined?
• Is coupling minimized?
• Can you identify potential duplication?
• Are interface definitions and constraints acceptable?
• Can modules access needed data—when needed?

jiffyclub

function condalist -d 'List conda environments.'
  for dir in (ls $HOME/miniconda3/envs)
    echo $dir
  end
end

function condactivate -d 'Activate a conda environment' -a cenv
  if test -z $cenv
    echo 'Usage: condactivate <env name>'
    return 1

jvns

A lot of these are outright stolen from Edward O'Campo-Gooding's list of questions. I really like his list.

I'm having some trouble paring this down to a manageable list of questions -- I realistically want to know all of these things before starting to work at a company, but it's a lot to ask all at once. My current game plan is to pick 6 before an interview and ask those.

I'd love comments and suggestions about any of these.

I've found questions like "do you have smart people? Can I learn a lot at your company?" to be basically totally useless -- everybody will say "yeah, definitely!" and it's hard to learn anything from them. So I'm trying to make all of these questions pretty concrete -- if a team doesn't have an issue tracker, they don't have an issue tracker.

I'm also mostly not asking about principles, but the way things are -- not "do you think code review is important?", but "Does all code get reviewed?".

gregrahn

Three comparison points:
Presto + RCFile vs Impala + RCFile vs Impala + Parquet

Note: Query time, CPU utilization, Disk read tput (KBRead)

Impala v1.1.1
Presto v0.52
================================================================================================================================
Presto + RCFile:
select ss_sold_date_sk, count(*) from store_sales_rcfile group by 1 order by 1 limit 2000;

granturing

public class HCatInputFormat extends InputFormat<SerializableWritable<Writable>, HCatRecord> {
	
	private final org.apache.hcatalog.mapreduce.HCatInputFormat input;
	
	public HCatInputFormat() {
		input = new org.apache.hcatalog.mapreduce.HCatInputFormat();
	}

	@Override
	public RecordReader<SerializableWritable<Writable>, HCatRecord> createRecordReader(

abshingate

<NotepadPlus>
    <UserLang name="Apache PIG" ext="pig" udlVersion="2.1">
        <Settings>
            <Global caseIgnored="yes" allowFoldOfComments="yes" foldCompact="no" forcePureLC="0" decimalSeparator="0" />
            <Prefix Keywords1="no" Keywords2="no" Keywords3="no" Keywords4="no" Keywords5="no" Keywords6="no" Keywords7="no" Keywords8="no" />
        </Settings>
        <KeywordLists>
            <Keywords name="Comments">00-- 01 02 03/* 04*/</Keywords>
            <Keywords name="Numbers, prefix1"></Keywords>
            <Keywords name="Numbers, prefix2"></Keywords>

ruo91

[hadoop@dev ~]$ start-all.sh
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
13/10/25 22:21:07 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /home/hadoop/2.2.0/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
localhost]
sed: -e expression #1, char 6: unknown option to `s'
HotSpot(TM): ssh: Could not resolve hostname HotSpot(TM): Name or service not known
64-Bit: ssh: Could not resolve hostname 64-Bit: Name or service not known
Java: ssh: Could not resolve hostname Java: Name or service not known

caged

#!/bin/sh
# usage ./create-psql-database DATABASE
#
DB_NAME=$1
FOLDER_NAME="nhgis0004"


# Drop and recreate database
dropdb --interactive "${DB_NAME}"
createdb --encoding UTF8 $DB_NAME