gistfile1.txt

I am running into a problem while trying to upgrade to mrjob 0.4.2. While a solution to this problem would be great, I am also interested in learning if there is a way to debug the wrapper script on emr. I am able to ssh to the emr hadoop cluster. However I have not figured out how to execute the wrapper script using the same environment that hadoop sets up when it is running it as part of the mapper process.

Here are the setup_cmds in my mrjob.conf file.

    setup_cmds: &setup_cmds
    - cd sudarshan-src-tree.tar.gz; make -f Makefile.emr
    - mkdir -p sudarshan-src-tree.tar.gz/srv-configs
    - ln -sf $(readlink -f sudarshan.yaml) sudarshan-src-tree.tar.gz/srv-configs/sudarshan.yaml
    - ln -sf $(readlink -f sudarshan-config-spark.yaml) sudarshan-src-tree.tar.gz/srv-configs/sudarshan-config-spark.yaml

The bash wrapper script fails while trying to run mkdir. Here is the failure from the hadoop log

**mkdir: cannot create directory `sudarshan-src-tree.tar.gz': Not a directory**

*Below is the Python wrapper mrjob 0.4.0*
--------------------------------------------------------------------------------------
writing wrapper script to /scratch/smg/mr_canary.smg.20131207.141556.835152/wrapper.py
WRAPPER: from fcntl import flock, LOCK_EX, LOCK_UN
WRAPPER: from subprocess import check_call, PIPE
WRAPPER: import sys
WRAPPER: 
WRAPPER: lock_file = open('/tmp/wrapper.lock.mr_canary.smg.20131207.141556.835152', 'a')
WRAPPER: flock(lock_file, LOCK_EX)
WRAPPER: 
WRAPPER: # run setup cmds:
WRAPPER: check_call('cd sudarshan-src-tree.tar.gz; make -f Makefile.emr', shell=True, stdout=open('/dev/null', 'w'))
WRAPPER: check_call('mkdir -p sudarshan-src-tree.tar.gz/srv-configs', shell=True, stdout=open('/dev/null', 'w'))
WRAPPER: check_call('ln -sf $(readlink -f sudarshan.yaml) sudarshan-src-tree.tar.gz/srv-configs/sudarshan.yaml', shell=True, stdout=open('/
dev/null', 'w'))
WRAPPER: check_call('ln -sf $(readlink -f sudarshan-config-spark.yaml) sudarshan-src-tree.tar.gz/srv-configs/sudarshan-config-spark.yaml', sh
ell=True, stdout=open('/dev/null', 'w'))
WRAPPER: 
WRAPPER: flock(lock_file, LOCK_UN)
WRAPPER: 
WRAPPER: # run the real mapper/reducer
WRAPPER: check_call(sys.argv[1:])
--------------------------------------------------------------------------------------
*Here is the Bash wrapper from mrjob 0.4.2*
--------------------------------------------------------------------------------------
writing wrapper script to /scratch/smg/mr_canary.smg.20131207.145408.625051/setup-wrapper.sh
WRAPPER: # store $PWD
WRAPPER: __mrjob_PWD=$PWD
WRAPPER: 
WRAPPER: # obtain exclusive file lock
WRAPPER: exec 9>/tmp/wrapper.lock.mr_canary.smg.20131207.145408.625051
WRAPPER: python -c 'import fcntl; fcntl.flock(9, fcntl.LOCK_EX)'
WRAPPER: 
WRAPPER: # setup commands
WRAPPER: {
WRAPPER:   export PYTHONPATH=$__mrjob_PWD/sudarshan-src-tree.tar.gz:$PYTHONPATH
WRAPPER:   cd sudarshan-src-tree.tar.gz; make -f Makefile.emr
WRAPPER:   mkdir -p sudarshan-src-tree.tar.gz/srv-configs
WRAPPER:   ln -sf $(readlink -f sudarshan.yaml) sudarshan-src-tree.tar.gz/srv-configs/sudarshan.yaml
WRAPPER:   ln -sf $(readlink -f sudarshan-config-spark.yaml) sudarshan-src-tree.tar.gz/srv-configs/sudarshan-config-spark.yaml
WRAPPER: } 0</dev/null 1>&2
WRAPPER: 
WRAPPER: # release exclusive file lock
WRAPPER: exec 9>&-
WRAPPER: 
WRAPPER: # run task from the original working directory
WRAPPER: cd $__mrjob_PWD
WRAPPER: "$@"
--------------------------------------------------------------------------------------------------
The job fails with the following error
Exception: Job on job flow j-2OZCEQUP4292N failed with status WAITING: Waiting after step failed
Probable cause of failure (from ssh://ec2-54-219-122-133.us-west-1.compute.amazonaws.com/mnt/var/log/hadoop/userlogs/job_201312071501_0001/
attempt_201312071501_0001_m_000000_3/syslog):
java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1
(while reading from s3://yelp-emr-dev/tmp/mr_canary.smg.20131207.145408.625051/files/README.txt)


I sshed to the emr hadoop cluster to look at the log files.
*Hadoop log*
------------------
hadoop@ip-10-171-115-160:/mnt/var/lib/hadoop/mapred/userlogs$ cat ./job_201312071501_0001/attempt_201312071501_0001_m_000005_0/stderr
cd sudarshan/util/cython; python setup_on_emr.py build_ext --inplace
running build_ext
building 'sudarshanyx' extension
creating build
creating build/temp.linux-x86_64-2.6
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.6 -c sudarshanyx.c -o build/temp.linux-x86_64-2.6/sudarshanyx.o -Wno-unused
gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions build/temp.linux-x86_64-2.6/sudarshanyx.o -o /mnt/var/lib/hadoop/mapred/taskTracker/hadoop/distcache/4312661691767056601_1162924978_1301649074/10.171.115.160/mnt/var/lib/hadoop/tmp/mapred/staging/hadoop/.staging/job_201312071501_0001/archives/sudarshan-src-tree.tar.gz/sudarshan/util/cython/sudarshanyx.so
touch sudarshan/logic/__init__.py
touch sudarshan/logic/dbobjs/__init__.py
**mkdir: cannot create directory `sudarshan-src-tree.tar.gz': Not a directory**
java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1
        at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:372)
        at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:586)
        at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:136)
        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
        at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:36)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:441)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:377)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1132)
        at org.apache.hadoop.mapred.Child.main(Child.java:249)
log4j:WARN No appenders could be found for logger (org.apache.hadoop.conf.Configuration).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.