Error log of LastCountryOfUser task qirh fwoloxrion-srerror

LastCountryOfUser task error

remote-task LastCountryOfUser --n-reduce-tasks 1 --host localhost   --remote-name analyticstack --user hadoop --repo https://github.com/codetigerco/edx-analytics-pipeline  --branch stage_edxapp --local-scheduler   --verbose   --wait --overwrite --interval-start 2017-01-01 --overwrite-n-days 16 --geolocation-data GeoIP.dat --override-config $HOME/edx-analytics-pipeline/config/devstack.cfg --skip-setup
Parsed arguments = Namespace(branch='stage_edxapp', extra_repo=None, host='localhost', job_flow_id=None, job_flow_name=None, launch_task_arguments=['LastCountryOfUser', '--n-reduce-tasks', '1', '--local-scheduler', '--overwrite', '--interval-start', '2017-01-01', '--overwrite-n-days', '16', '--geolocation-data', 'GeoIP.dat'], log_path=None, override_config='/edx/app/hadoop/edx-analytics-pipeline/config/devstack.cfg', package=None, private_key=None, remote_name='analyticstack', repo='https://github.com/codetigerco/edx-analytics-pipeline', secure_config=None, secure_config_branch=None, secure_config_repo=None, shell=None, skip_setup=True, sudo_user='hadoop', user='hadoop', vagrant_path=None, verbose=True, virtualenv_extra_args=None, wait=True, wheel_url=None, workflow_profiler=None)
Running commands from path = /edx/app/hadoop/pipeline/share/edx.analytics.tasks
Remote name = analyticstack
Running command = ['ssh', '-tt', '-o', 'ForwardAgent=yes', '-o', 'StrictHostKeyChecking=no', '-o', 'UserKnownHostsFile=/dev/null', '-o', 'KbdInteractiveAuthentication=no', '-o', 'PasswordAuthentication=no', '-o', 'User=hadoop', '-o', 'ConnectTimeout=10', 'localhost', "sudo -Hu hadoop /bin/bash -c 'cd /var/lib/analytics-tasks/analyticstack/repo && . $HOME/.bashrc && /var/lib/analytics-tasks/analyticstack/venv/bin/launch-task LastCountryOfUser --n-reduce-tasks 1 --local-scheduler --overwrite --interval-start 2017-01-01 --overwrite-n-days 16 --geolocation-data GeoIP.dat'"]
Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts.
^CTraceback (most recent call last):
  File "/var/lib/analytics-tasks/analyticstack/venv/bin/launch-task", line 6, in <module>
    from edx.analytics.tasks.launchers.local import main
  File "/var/lib/analytics-tasks/analyticstack/venv/local/lib/python2.7/site-packages/edx/analytics/tasks/launchers/local.py", line 22, in <module>
    import requests
  File "/var/lib/analytics-tasks/analyticstack/venv/local/lib/python2.7/site-packages/requests/__init__.py", line 53, in <module>
    from .packages.urllib3.contrib import pyopenssl
  File "/var/lib/analytics-tasks/analyticstack/venv/local/lib/python2.7/site-packages/requests/packages/urllib3/contrib/pyopenssl.py", line 53, in <module>
    import OpenSSL.SSL
  File "/var/lib/analytics-tasks/analyticstack/venv/local/lib/python2.7/site-packages/OpenSSL/__init__.py", line 8, in <module>
    from OpenSSL import rand, crypto, SSL
  File "/var/lib/analytics-tasks/analyticstack/venv/local/lib/python2.7/site-packages/OpenSSL/rand.py", line 11, in <module>
    from OpenSSL._util import (
  File "/var/lib/analytics-tasks/analyticstack/venv/local/lib/python2.7/site-packages/OpenSSL/_util.py", line 6, in <module>
    from cryptography.hazmat.bindings.openssl.binding import Binding
  File "/var/lib/analytics-tasks/analyticstack/venv/local/lib/python2.7/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 60, in <module>
    class Binding(object):
  File "/var/lib/analytics-tasks/analyticstack/venv/local/lib/python2.7/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 109, in Binding
    libraries=_get_libraries(sys.platform)
  File "/var/lib/analytics-tasks/analyticstack/venv/local/lib/python2.7/site-packages/cryptography/hazmat/bindings/utils.py", line 97, in build_ffi_for_binding
    extra_link_args=extra_link_args,
  File "/var/lib/analytics-tasks/analyticstack/venv/local/lib/python2.7/site-packages/cryptography/hazmat/bindings/utils.py", line 106, in build_ffi
    ffi.cdef(cdef_source)
  File "/var/lib/analytics-tasks/analyticstack/venv/local/lib/python2.7/site-packages/cffi/api.py", line 107, in cdef
    self._parser.parse(csource, override=override, packed=packed)
  File "/var/lib/analytics-tasks/analyticstack/venv/local/lib/python2.7/site-packages/cffi/cparser.py", line 167, in parse
    self._internal_parse(csource)
  File "/var/lib/analytics-tasks/analyticstack/venv/local/lib/python2.7/site-packages/cffi/cparser.py", line 173, in _internal_parse
    ast, macros, csource = self._parse(csource)
  File "/var/lib/analytics-tasks/analyticstack/venv/local/lib/python2.7/site-packages/cffi/cparser.py", line 128, in _parse
    ast = _get_parser().parse(csource)
  File "/var/lib/analytics-tasks/analyticstack/venv/local/lib/python2.7/site-packages/pycparser/c_parser.py", line 139, in parse
    debug=debuglevel)
  File "/var/lib/analytics-tasks/analyticstack/venv/local/lib/python2.7/site-packages/pycparser/ply/yacc.py", line 265, in parse
    return self.parseopt_notrack(input,lexer,debug,tracking,tokenfunc)
  File "/var/lib/analytics-tasks/analyticstack/venv/local/lib/python2.7/site-packages/pycparser/ply/yacc.py", line 921, in parseopt_notrack
    lookahead = get_token()     # Get the next token
  File "/var/lib/analytics-tasks/analyticstack/venv/local/lib/python2.7/site-packages/pycparser/c_lexer.py", line 77, in token
    self.last_token = self.lexer.token()
  File "/var/lib/analytics-tasks/analyticstack/venv/local/lib/python2.7/site-packages/pycparser/ply/lex.py", line 319, in token
    m = lexre.match(lexdata,lexpos)
KeyboardInterrupt
Connection to localhost closed.
Exiting with status = 1
(pipeline)hadoop@ip-172-30-0-176:~$ ls GeoIP.dat 
GeoIP.dat
(pipeline)hadoop@ip-172-30-0-176:~$ remote-task LastCountryOfUser --n-reduce-tasks 1 --host localhost   --remote-name analyticstack --user hadoop --repo https://github.com/codetigerco/edx-analytics-pipeline  --branch stage_edxapp --local-scheduler   --verbose   --wait --overwrite --interval-start 2017-01-01 --overwrite-n-days 16 --geolocation-data GeoIP.dat --override-config $HOME/edx-analytics-pipeline/config/devstack.cfg --skip-setup
Parsed arguments = Namespace(branch='stage_edxapp', extra_repo=None, host='localhost', job_flow_id=None, job_flow_name=None, launch_task_arguments=['LastCountryOfUser', '--n-reduce-tasks', '1', '--local-scheduler', '--overwrite', '--interval-start', '2017-01-01', '--overwrite-n-days', '16', '--geolocation-data', 'GeoIP.dat'], log_path=None, override_config='/edx/app/hadoop/edx-analytics-pipeline/config/devstack.cfg', package=None, private_key=None, remote_name='analyticstack', repo='https://github.com/codetigerco/edx-analytics-pipeline', secure_config=None, secure_config_branch=None, secure_config_repo=None, shell=None, skip_setup=True, sudo_user='hadoop', user='hadoop', vagrant_path=None, verbose=True, virtualenv_extra_args=None, wait=True, wheel_url=None, workflow_profiler=None)
Running commands from path = /edx/app/hadoop/pipeline/share/edx.analytics.tasks
Remote name = analyticstack
Running command = ['ssh', '-tt', '-o', 'ForwardAgent=yes', '-o', 'StrictHostKeyChecking=no', '-o', 'UserKnownHostsFile=/dev/null', '-o', 'KbdInteractiveAuthentication=no', '-o', 'PasswordAuthentication=no', '-o', 'User=hadoop', '-o', 'ConnectTimeout=10', 'localhost', "sudo -Hu hadoop /bin/bash -c 'cd /var/lib/analytics-tasks/analyticstack/repo && . $HOME/.bashrc && /var/lib/analytics-tasks/analyticstack/venv/bin/launch-task LastCountryOfUser --n-reduce-tasks 1 --local-scheduler --overwrite --interval-start 2017-01-01 --overwrite-n-days 16 --geolocation-data GeoIP.dat'"]
Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts.
DEBUG:stevedore.extension:found extension EntryPoint.parse('sqoop-import = edx.analytics.tasks.common.sqoop:SqoopImportFromMysql')
DEBUG:stevedore.extension:found extension EntryPoint.parse('run-vertica-sql-script = edx.analytics.tasks.warehouse.run_vertica_sql_script:RunVerticaSqlScriptTask')
DEBUG:stevedore.extension:found extension EntryPoint.parse('obfuscation = edx.analytics.tasks.export.obfuscation:ObfuscatedCourseTask')
DEBUG:stevedore.extension:found extension EntryPoint.parse('enrollment_validation = edx.analytics.tasks.monitor.enrollment_validation:CourseEnrollmentValidationTask')
DEBUG:stevedore.extension:found extension EntryPoint.parse('inc-enrollments-report = edx.analytics.tasks.warehouse.enrollment.incremental_enrollments:WeeklyIncrementalUsersAndEnrollments')
DEBUG:stevedore.extension:found extension EntryPoint.parse('total-enrollments-report = edx.analytics.tasks.warehouse.enrollment.total_enrollments:WeeklyAllUsersAndEnrollments')
DEBUG:stevedore.extension:found extension EntryPoint.parse('orders = edx.analytics.tasks.warehouse.financial.orders_import:OrderTableTask')
DEBUG:stevedore.extension:found extension EntryPoint.parse('problem_response = edx.analytics.tasks.insights.problem_response:LatestProblemResponseDataTask')
DEBUG:stevedore.extension:found extension EntryPoint.parse('push_to_vertica_lms_courseware_link_clicked = edx.analytics.tasks.warehouse.lms_courseware_link_clicked:PushToVerticaLMSCoursewareLinkClickedTask')
DEBUG:stevedore.extension:found extension EntryPoint.parse('video = edx.analytics.tasks.insights.video:InsertToMysqlAllVideoTask')
DEBUG:stevedore.extension:found extension EntryPoint.parse('ed_services_report = edx.analytics.tasks.warehouse.financial.ed_services_financial_report:BuildEdServicesReportTask')
DEBUG:stevedore.extension:found extension EntryPoint.parse('load-internal-database = edx.analytics.tasks.warehouse.load_internal_reporting_database:ImportMysqlToVerticaTask')
DEBUG:stevedore.extension:found extension EntryPoint.parse('export-student-module = edx.analytics.tasks.export.database_exports:StudentModulePerCourseAfterImportWorkflow')
DEBUG:stevedore.extension:found extension EntryPoint.parse('calendar = edx.analytics.tasks.insights.calendar_task:CalendarTableTask')
DEBUG:stevedore.extension:found extension EntryPoint.parse('load-warehouse = edx.analytics.tasks.warehouse.load_warehouse:LoadWarehouseWorkflow')
DEBUG:stevedore.extension:found extension EntryPoint.parse('cybersource = edx.analytics.tasks.warehouse.financial.cybersource:DailyPullFromCybersourceTask')
DEBUG:stevedore.extension:found extension EntryPoint.parse('load-d-user = edx.analytics.tasks.warehouse.load_internal_reporting_user:LoadInternalReportingUserToWarehouse')
DEBUG:stevedore.extension:found extension EntryPoint.parse('location-per-course = edx.analytics.tasks.insights.location_per_course:LastCountryOfUser')
DEBUG:stevedore.extension:found extension EntryPoint.parse('payment_reconcile = edx.analytics.tasks.warehouse.financial.reconcile:ReconcileOrdersAndTransactionsTask')
DEBUG:stevedore.extension:found extension EntryPoint.parse('enrollments-report = edx.analytics.tasks.warehouse.enrollment.enrollments:EnrollmentsByWeek')
DEBUG:stevedore.extension:found extension EntryPoint.parse('financial_reports = edx.analytics.tasks.warehouse.financial.finance_reports:BuildFinancialReportsTask')
DEBUG:stevedore.extension:found extension EntryPoint.parse('engagement = edx.analytics.tasks.insights.module_engagement:ModuleEngagementDataTask')
DEBUG:stevedore.extension:found extension EntryPoint.parse('events_obfuscation = edx.analytics.tasks.export.events_obfuscation:ObfuscateCourseEventsTask')
DEBUG:stevedore.extension:found extension EntryPoint.parse('dump-student-module = edx.analytics.tasks.export.database_exports:StudentModulePerCourseTask')
DEBUG:stevedore.extension:found extension EntryPoint.parse('export-events-by-course = edx.analytics.tasks.export.event_exports_by_course:EventExportByCourseTask')
DEBUG:stevedore.extension:found extension EntryPoint.parse('noop = edx.analytics.tasks.monitor.performance:ParseEventLogPerformanceTask')
DEBUG:stevedore.extension:found extension EntryPoint.parse('course_blocks = edx.analytics.tasks.insights.course_blocks:CourseBlocksApiDataTask')
DEBUG:stevedore.extension:found extension EntryPoint.parse('load-d-certificates = edx.analytics.tasks.warehouse.load_internal_reporting_certificates:LoadInternalReportingCertificatesToWarehouse')
DEBUG:stevedore.extension:found extension EntryPoint.parse('user-activity = edx.analytics.tasks.insights.user_activity:CourseActivityWeeklyTask')
DEBUG:stevedore.extension:found extension EntryPoint.parse('run-vertica-sql-scripts = edx.analytics.tasks.warehouse.run_vertica_sql_scripts:RunVerticaSqlScriptTask')
DEBUG:stevedore.extension:found extension EntryPoint.parse('paypal = edx.analytics.tasks.warehouse.financial.paypal:PaypalTransactionsByDayTask')
DEBUG:stevedore.extension:found extension EntryPoint.parse('grade-dist = edx.analytics.tasks.data_api.studentmodule_dist:GradeDistFromSqoopToMySQLWorkflow')
DEBUG:stevedore.extension:found extension EntryPoint.parse('enrollments_and_registrations_workflow-manifest = edx.analytics.tasks.warehouse.enrollment.enrollments_and_registrations_workflow_manifest:EnrollmentsandRegistrationsWorkflow')
DEBUG:stevedore.extension:found extension EntryPoint.parse('database-import = edx.analytics.tasks.insights.database_imports:ImportAllDatabaseTablesTask')
DEBUG:stevedore.extension:found extension EntryPoint.parse('catalog = edx.analytics.tasks.warehouse.course_catalog:CourseCatalogWorkflow')
DEBUG:stevedore.extension:found extension EntryPoint.parse('load-course-catalog = edx.analytics.tasks.warehouse.load_internal_reporting_course_catalog:PullCourseCatalogAPIData')
DEBUG:stevedore.extension:found extension EntryPoint.parse('enrollments = edx.analytics.tasks.insights.enrollments:ImportEnrollmentsIntoMysql')
DEBUG:stevedore.extension:found extension EntryPoint.parse('event-type-dist = edx.analytics.tasks.warehouse.event_type_dist:PushToVerticaEventTypeDistributionTask')
DEBUG:stevedore.extension:found extension EntryPoint.parse('tags-dist = edx.analytics.tasks.insights.tags_dist:TagsDistributionPerCourse')
DEBUG:stevedore.extension:found extension EntryPoint.parse('export-events = edx.analytics.tasks.export.event_exports:EventExportTask')
DEBUG:stevedore.extension:found extension EntryPoint.parse('course-enroll = edx.analytics.tasks.warehouse.enrollment.course_enroll:CourseEnrollmentChangesPerDay')
DEBUG:stevedore.extension:found extension EntryPoint.parse('data_obfuscation = edx.analytics.tasks.export.data_obfuscation:ObfuscatedCourseDumpTask')
DEBUG:stevedore.extension:found extension EntryPoint.parse('course_list = edx.analytics.tasks.insights.course_list:CourseListApiDataTask')
DEBUG:stevedore.extension:found extension EntryPoint.parse('load-d-user-course = edx.analytics.tasks.warehouse.load_internal_reporting_user_course:LoadUserCourseSummary')
DEBUG:stevedore.extension:found extension EntryPoint.parse('load-d-country = edx.analytics.tasks.warehouse.load_internal_reporting_country:LoadInternalReportingCountryToWarehouse')
DEBUG:stevedore.extension:found extension EntryPoint.parse('overall_events = edx.analytics.tasks.monitor.overall_events:TotalEventsDailyTask')
DEBUG:stevedore.extension:found extension EntryPoint.parse('load-f-user-activity = edx.analytics.tasks.warehouse.load_internal_reporting_user_activity:LoadInternalReportingUserActivityToWarehouse')
DEBUG:stevedore.extension:found extension EntryPoint.parse('student_engagement = edx.analytics.tasks.data_api.student_engagement:StudentEngagementTask')
DEBUG:stevedore.extension:found extension EntryPoint.parse('answer-dist = edx.analytics.tasks.insights.answer_dist:AnswerDistributionPerCourse')
DEBUG:stevedore.extension:found extension EntryPoint.parse('insert-into-table = edx.analytics.tasks.common.mysql_load:MysqlInsertTask')
DEBUG:stevedore.extension:found extension EntryPoint.parse('all_events_report = edx.analytics.tasks.monitor.total_events_report:TotalEventsReportWorkflow')
DEBUG:edx.analytics.tasks.launchers.local:Using override.cfg
2017-01-19 17:28:19,813 INFO 10396 [luigi-interface] worker.py:267 - Scheduled LastCountryOfUser(source=('hdfs://localhost:9000/data/',), expand_interval=2 days, 0:00:00, pattern=('.*tracking.log.*',), date_pattern=%Y%m%d, geolocation_data=GeoIP.dat, warehouse_path=hdfs://localhost:9000/edx-analytics-pipeline/warehouse/, interval=None) (PENDING)
2017-01-19 17:28:19,814 INFO 10396 [luigi-interface] worker.py:267 - Scheduled LastDailyIpAddressOfUserTask(source=('hdfs://localhost:9000/data/',), interval=2017-01-03-2017-01-19, expand_interval=2 days, 0:00:00, pattern=('.*tracking.log.*',), date_pattern=%Y%m%d, warehouse_path=hdfs://localhost:9000/edx-analytics-pipeline/warehouse/) (PENDING)
2017-01-19 17:28:28,000 INFO 10396 [luigi-interface] worker.py:267 - Scheduled PathSelectionByDateIntervalTask(source=('hdfs://localhost:9000/data/',), interval=2017-01-03-2017-01-19, expand_interval=2 days, 0:00:00, pattern=('.*tracking.log.*',), date_pattern=%Y%m%d) (DONE)
2017-01-19 17:28:28,013 WARNING 10396 [luigi-interface] worker.py:246 - Task ExternalURL(url=GeoIP.dat) is not complete and run() is not implemented. Probably a missing external dependency.
2017-01-19 17:28:28,014 INFO 10396 [luigi-interface] worker.py:267 - Scheduled ExternalURL(url=GeoIP.dat) (PENDING)
2017-01-19 17:28:36,148 INFO 10396 [luigi-interface] worker.py:267 - Scheduled PathSelectionByDateIntervalTask(source=('hdfs://localhost:9000/edx-analytics-pipeline/warehouse/last_ip_of_user',), interval=2017-01-01-2017-01-03, expand_interval=0:00:00, pattern=('.*?last_ip_of_user_(?P<date>\\d{4}-\\d{2}-\\d{2})',), date_pattern=%Y-%m-%d) (DONE)
2017-01-19 17:28:36,149 INFO 10396 [luigi-interface] worker.py:267 - Scheduled UncheckedExternalURL(url=hdfs://localhost:9000/edx-analytics-pipeline/warehouse/last_ip_of_user/dt=2017-01-18/last_ip_of_user_2017-01-18) (DONE)
2017-01-19 17:28:36,149 INFO 10396 [luigi-interface] worker.py:267 - Scheduled UncheckedExternalURL(url=hdfs://localhost:9000/edx-analytics-pipeline/warehouse/last_ip_of_user/dt=2017-01-17/last_ip_of_user_2017-01-17) (DONE)
2017-01-19 17:28:36,150 INFO 10396 [luigi-interface] worker.py:267 - Scheduled UncheckedExternalURL(url=hdfs://localhost:9000/edx-analytics-pipeline/warehouse/last_ip_of_user/dt=2017-01-16/last_ip_of_user_2017-01-16) (DONE)
2017-01-19 17:28:36,150 INFO 10396 [luigi-interface] worker.py:267 - Scheduled UncheckedExternalURL(url=hdfs://localhost:9000/edx-analytics-pipeline/warehouse/last_ip_of_user/dt=2017-01-15/last_ip_of_user_2017-01-15) (DONE)
2017-01-19 17:28:36,150 INFO 10396 [luigi-interface] worker.py:267 - Scheduled UncheckedExternalURL(url=hdfs://localhost:9000/edx-analytics-pipeline/warehouse/last_ip_of_user/dt=2017-01-14/last_ip_of_user_2017-01-14) (DONE)
2017-01-19 17:28:36,151 INFO 10396 [luigi-interface] worker.py:267 - Scheduled UncheckedExternalURL(url=hdfs://localhost:9000/edx-analytics-pipeline/warehouse/last_ip_of_user/dt=2017-01-13/last_ip_of_user_2017-01-13) (DONE)
2017-01-19 17:28:36,151 INFO 10396 [luigi-interface] worker.py:267 - Scheduled UncheckedExternalURL(url=hdfs://localhost:9000/edx-analytics-pipeline/warehouse/last_ip_of_user/dt=2017-01-12/last_ip_of_user_2017-01-12) (DONE)
2017-01-19 17:28:36,151 INFO 10396 [luigi-interface] worker.py:267 - Scheduled UncheckedExternalURL(url=hdfs://localhost:9000/edx-analytics-pipeline/warehouse/last_ip_of_user/dt=2017-01-11/last_ip_of_user_2017-01-11) (DONE)
2017-01-19 17:28:36,152 INFO 10396 [luigi-interface] worker.py:267 - Scheduled UncheckedExternalURL(url=hdfs://localhost:9000/edx-analytics-pipeline/warehouse/last_ip_of_user/dt=2017-01-10/last_ip_of_user_2017-01-10) (DONE)
2017-01-19 17:28:36,152 INFO 10396 [luigi-interface] worker.py:267 - Scheduled UncheckedExternalURL(url=hdfs://localhost:9000/edx-analytics-pipeline/warehouse/last_ip_of_user/dt=2017-01-09/last_ip_of_user_2017-01-09) (DONE)
2017-01-19 17:28:36,152 INFO 10396 [luigi-interface] worker.py:267 - Scheduled UncheckedExternalURL(url=hdfs://localhost:9000/edx-analytics-pipeline/warehouse/last_ip_of_user/dt=2017-01-08/last_ip_of_user_2017-01-08) (DONE)
2017-01-19 17:28:36,153 INFO 10396 [luigi-interface] worker.py:267 - Scheduled UncheckedExternalURL(url=hdfs://localhost:9000/edx-analytics-pipeline/warehouse/last_ip_of_user/dt=2017-01-07/last_ip_of_user_2017-01-07) (DONE)
2017-01-19 17:28:36,153 INFO 10396 [luigi-interface] worker.py:267 - Scheduled UncheckedExternalURL(url=hdfs://localhost:9000/edx-analytics-pipeline/warehouse/last_ip_of_user/dt=2017-01-06/last_ip_of_user_2017-01-06) (DONE)
2017-01-19 17:28:36,153 INFO 10396 [luigi-interface] worker.py:267 - Scheduled UncheckedExternalURL(url=hdfs://localhost:9000/edx-analytics-pipeline/warehouse/last_ip_of_user/dt=2017-01-05/last_ip_of_user_2017-01-05) (DONE)
2017-01-19 17:28:36,153 INFO 10396 [luigi-interface] worker.py:267 - Scheduled UncheckedExternalURL(url=hdfs://localhost:9000/edx-analytics-pipeline/warehouse/last_ip_of_user/dt=2017-01-04/last_ip_of_user_2017-01-04) (DONE)
2017-01-19 17:28:36,154 INFO 10396 [luigi-interface] worker.py:267 - Scheduled UncheckedExternalURL(url=hdfs://localhost:9000/edx-analytics-pipeline/warehouse/last_ip_of_user/dt=2017-01-03/last_ip_of_user_2017-01-03) (DONE)
2017-01-19 17:28:36,154 INFO 10396 [luigi-interface] interface.py:193 - Done scheduling tasks
2017-01-19 17:28:36,154 INFO 10396 [luigi-interface] worker.py:282 - [pid 10396] Worker Worker(salt=514950578, host=ip-172-30-0-176, username=hadoop, pid=10396) running   LastDailyIpAddressOfUserTask(source=('hdfs://localhost:9000/data/',), interval=2017-01-03-2017-01-19, expand_interval=2 days, 0:00:00, pattern=('.*tracking.log.*',), date_pattern=%Y%m%d, warehouse_path=hdfs://localhost:9000/edx-analytics-pipeline/warehouse/)
2017-01-19 17:28:42,475 INFO 10396 [luigi-interface] hadoop.py:242 - /edx/app/hadoop/hadoop/bin/hadoop jar /edx/app/hadoop/hadoop/share/hadoop/tools/lib/hadoop-streaming-2.3.0.jar -D mapred.job.name=LastDailyIpAddressOfUserTask(source=('hdfs://localhost:9000/data/',), interval=2017-01-03-2017-01-19, expand_interval=2 days, 0:00:00, pattern=('.*tracking.log.*',), date_pattern=%Y%m%d, warehouse_path=hdfs://localhost:9000/edx-analytics-pipeline/warehouse/) -D mapred.reduce.tasks=1 -mapper /usr/bin/python2.7 mrrunner.py map -reducer /usr/bin/python2.7 mrrunner.py reduce -file /var/lib/analytics-tasks/analyticstack/venv/local/lib/python2.7/site-packages/luigi/mrrunner.py -file /tmp/tmp9qSABm/packages.tar -file /tmp/tmp9qSABm/job-instance.pickle -input /data/tracking.log -output /edx-analytics-pipeline/marker/-969637439385692816-temp-2017-01-19T17-28-40.219523
2017-01-19 17:28:44,110 INFO 10396 [luigi-interface] hadoop.py:273 - 17/01/19 17:28:44 WARN streaming.StreamJob: -file option is deprecated, please use generic option -files instead.
2017-01-19 17:28:47,474 INFO 10396 [luigi-interface] hadoop.py:273 - 17/01/19 17:28:47 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
2017-01-19 17:28:47,919 INFO 10396 [luigi-interface] hadoop.py:273 - 17/01/19 17:28:47 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
2017-01-19 17:28:50,422 INFO 10396 [luigi-interface] hadoop.py:273 - 17/01/19 17:28:50 INFO mapred.FileInputFormat: Total input paths to process : 1
2017-01-19 17:28:50,564 INFO 10396 [luigi-interface] hadoop.py:273 - 17/01/19 17:28:50 INFO mapreduce.JobSubmitter: number of splits:2
2017-01-19 17:28:50,570 INFO 10396 [luigi-interface] hadoop.py:273 - 17/01/19 17:28:50 INFO Configuration.deprecation: mapred.job.name is deprecated. Instead, use mapreduce.job.name
2017-01-19 17:28:50,573 INFO 10396 [luigi-interface] hadoop.py:273 - 17/01/19 17:28:50 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
2017-01-19 17:28:51,343 INFO 10396 [luigi-interface] hadoop.py:273 - 17/01/19 17:28:51 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1484841390427_0006
2017-01-19 17:28:51,868 INFO 10396 [luigi-interface] hadoop.py:273 - 17/01/19 17:28:51 INFO impl.YarnClientImpl: Submitted application application_1484841390427_0006
2017-01-19 17:28:51,998 INFO 10396 [luigi-interface] hadoop.py:273 - 17/01/19 17:28:51 INFO mapreduce.Job: The url to track the job: http://ip-172-30-0-176:8088/proxy/application_1484841390427_0006/
2017-01-19 17:28:52,012 INFO 10396 [luigi-interface] hadoop.py:273 - 17/01/19 17:28:52 INFO mapreduce.Job: Running job: job_1484841390427_0006
2017-01-19 17:29:08,515 INFO 10396 [luigi-interface] hadoop.py:273 - 17/01/19 17:29:08 INFO mapreduce.Job: Job job_1484841390427_0006 running in uber mode : false
2017-01-19 17:29:08,515 INFO 10396 [luigi-interface] hadoop.py:273 - 17/01/19 17:29:08 INFO mapreduce.Job:  map 0% reduce 0%
2017-01-19 17:29:29,144 INFO 10396 [luigi-interface] hadoop.py:273 - 17/01/19 17:29:29 INFO mapreduce.Job:  map 50% reduce 0%
2017-01-19 17:29:30,171 INFO 10396 [luigi-interface] hadoop.py:273 - 17/01/19 17:29:30 INFO mapreduce.Job:  map 100% reduce 0%
2017-01-19 17:29:44,515 INFO 10396 [luigi-interface] hadoop.py:273 - 17/01/19 17:29:44 INFO mapreduce.Job:  map 100% reduce 100%
2017-01-19 17:29:57,744 INFO 10396 [luigi-interface] hadoop.py:273 - 17/01/19 17:29:57 INFO mapreduce.Job: Task Id : attempt_1484841390427_0006_r_000000_0, Status : FAILED
2017-01-19 17:29:57,778 INFO 10396 [luigi-interface] hadoop.py:273 - Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1
2017-01-19 17:29:57,780 INFO 10396 [luigi-interface] hadoop.py:273 - at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:320)
2017-01-19 17:29:57,781 INFO 10396 [luigi-interface] hadoop.py:273 - at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:533)
2017-01-19 17:29:57,792 INFO 10396 [luigi-interface] hadoop.py:273 - at org.apache.hadoop.streaming.PipeReducer.close(PipeReducer.java:134)
2017-01-19 17:29:57,792 INFO 10396 [luigi-interface] hadoop.py:273 - at org.apache.hadoop.io.IOUtils.cleanup(IOUtils.java:237)
2017-01-19 17:29:57,793 INFO 10396 [luigi-interface] hadoop.py:273 - at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:459)
2017-01-19 17:29:57,793 INFO 10396 [luigi-interface] hadoop.py:273 - at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
2017-01-19 17:29:57,793 INFO 10396 [luigi-interface] hadoop.py:273 - at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
2017-01-19 17:29:57,793 INFO 10396 [luigi-interface] hadoop.py:273 - at java.security.AccessController.doPrivileged(Native Method)
2017-01-19 17:29:57,794 INFO 10396 [luigi-interface] hadoop.py:273 - at javax.security.auth.Subject.doAs(Subject.java:422)
2017-01-19 17:29:57,794 INFO 10396 [luigi-interface] hadoop.py:273 - at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
2017-01-19 17:29:57,794 INFO 10396 [luigi-interface] hadoop.py:273 - at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
2017-01-19 17:29:58,798 INFO 10396 [luigi-interface] hadoop.py:273 - 17/01/19 17:29:58 INFO mapreduce.Job:  map 100% reduce 0%
2017-01-19 17:30:19,199 INFO 10396 [luigi-interface] hadoop.py:273 - 17/01/19 17:30:19 INFO mapreduce.Job:  map 100% reduce 100%
2017-01-19 17:30:32,463 INFO 10396 [luigi-interface] hadoop.py:273 - 17/01/19 17:30:32 INFO mapreduce.Job: Task Id : attempt_1484841390427_0006_r_000000_1, Status : FAILED
2017-01-19 17:30:32,466 INFO 10396 [luigi-interface] hadoop.py:273 - Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1
2017-01-19 17:30:32,466 INFO 10396 [luigi-interface] hadoop.py:273 - at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:320)
2017-01-19 17:30:32,467 INFO 10396 [luigi-interface] hadoop.py:273 - at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:533)
2017-01-19 17:30:32,467 INFO 10396 [luigi-interface] hadoop.py:273 - at org.apache.hadoop.streaming.PipeReducer.close(PipeReducer.java:134)
2017-01-19 17:30:32,467 INFO 10396 [luigi-interface] hadoop.py:273 - at org.apache.hadoop.io.IOUtils.cleanup(IOUtils.java:237)
2017-01-19 17:30:32,467 INFO 10396 [luigi-interface] hadoop.py:273 - at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:459)
2017-01-19 17:30:32,467 INFO 10396 [luigi-interface] hadoop.py:273 - at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
2017-01-19 17:30:32,469 INFO 10396 [luigi-interface] hadoop.py:273 - at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
2017-01-19 17:30:32,469 INFO 10396 [luigi-interface] hadoop.py:273 - at java.security.AccessController.doPrivileged(Native Method)
2017-01-19 17:30:32,470 INFO 10396 [luigi-interface] hadoop.py:273 - at javax.security.auth.Subject.doAs(Subject.java:422)
2017-01-19 17:30:32,479 INFO 10396 [luigi-interface] hadoop.py:273 - at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
2017-01-19 17:30:32,479 INFO 10396 [luigi-interface] hadoop.py:273 - at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
2017-01-19 17:30:33,480 INFO 10396 [luigi-interface] hadoop.py:273 - 17/01/19 17:30:33 INFO mapreduce.Job:  map 100% reduce 0%
2017-01-19 17:30:48,823 INFO 10396 [luigi-interface] hadoop.py:273 - 17/01/19 17:30:48 INFO mapreduce.Job:  map 100% reduce 100%
2017-01-19 17:31:01,044 INFO 10396 [luigi-interface] hadoop.py:273 - 17/01/19 17:31:01 INFO mapreduce.Job: Task Id : attempt_1484841390427_0006_r_000000_2, Status : FAILED
2017-01-19 17:31:01,047 INFO 10396 [luigi-interface] hadoop.py:273 - Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1
2017-01-19 17:31:01,047 INFO 10396 [luigi-interface] hadoop.py:273 - at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:320)
2017-01-19 17:31:01,047 INFO 10396 [luigi-interface] hadoop.py:273 - at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:533)
2017-01-19 17:31:01,047 INFO 10396 [luigi-interface] hadoop.py:273 - at org.apache.hadoop.streaming.PipeReducer.close(PipeReducer.java:134)
2017-01-19 17:31:01,048 INFO 10396 [luigi-interface] hadoop.py:273 - at org.apache.hadoop.io.IOUtils.cleanup(IOUtils.java:237)
2017-01-19 17:31:01,048 INFO 10396 [luigi-interface] hadoop.py:273 - at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:459)
2017-01-19 17:31:01,048 INFO 10396 [luigi-interface] hadoop.py:273 - at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
2017-01-19 17:31:01,048 INFO 10396 [luigi-interface] hadoop.py:273 - at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
2017-01-19 17:31:01,049 INFO 10396 [luigi-interface] hadoop.py:273 - at java.security.AccessController.doPrivileged(Native Method)
2017-01-19 17:31:01,051 INFO 10396 [luigi-interface] hadoop.py:273 - at javax.security.auth.Subject.doAs(Subject.java:422)
2017-01-19 17:31:01,051 INFO 10396 [luigi-interface] hadoop.py:273 - at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
2017-01-19 17:31:01,084 INFO 10396 [luigi-interface] hadoop.py:273 - at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
2017-01-19 17:31:02,085 INFO 10396 [luigi-interface] hadoop.py:273 - 17/01/19 17:31:02 INFO mapreduce.Job:  map 100% reduce 0%
2017-01-19 17:31:20,450 INFO 10396 [luigi-interface] hadoop.py:273 - 17/01/19 17:31:20 INFO mapreduce.Job:  map 100% reduce 100%
2017-01-19 17:31:34,628 INFO 10396 [luigi-interface] hadoop.py:273 - 17/01/19 17:31:34 INFO mapreduce.Job: Job job_1484841390427_0006 failed with state FAILED due to: Task failed task_1484841390427_0006_r_000000
2017-01-19 17:31:34,637 INFO 10396 [luigi-interface] hadoop.py:273 - Job failed as tasks failed. failedMaps:0 failedReduces:1
2017-01-19 17:31:34,930 INFO 10396 [luigi-interface] hadoop.py:273 - 17/01/19 17:31:34 INFO mapreduce.Job: Counters: 37
2017-01-19 17:31:34,930 INFO 10396 [luigi-interface] hadoop.py:273 - File System Counters
2017-01-19 17:31:34,930 INFO 10396 [luigi-interface] hadoop.py:273 - FILE: Number of bytes read=0
2017-01-19 17:31:34,931 INFO 10396 [luigi-interface] hadoop.py:273 - FILE: Number of bytes written=195423
2017-01-19 17:31:34,931 INFO 10396 [luigi-interface] hadoop.py:273 - FILE: Number of read operations=0
2017-01-19 17:31:34,931 INFO 10396 [luigi-interface] hadoop.py:273 - FILE: Number of large read operations=0
2017-01-19 17:31:34,931 INFO 10396 [luigi-interface] hadoop.py:273 - FILE: Number of write operations=0
2017-01-19 17:31:34,932 INFO 10396 [luigi-interface] hadoop.py:273 - HDFS: Number of bytes read=686974
2017-01-19 17:31:34,932 INFO 10396 [luigi-interface] hadoop.py:273 - HDFS: Number of bytes written=0
2017-01-19 17:31:34,932 INFO 10396 [luigi-interface] hadoop.py:273 - HDFS: Number of read operations=6
2017-01-19 17:31:34,932 INFO 10396 [luigi-interface] hadoop.py:273 - HDFS: Number of large read operations=0
2017-01-19 17:31:34,932 INFO 10396 [luigi-interface] hadoop.py:273 - HDFS: Number of write operations=0
2017-01-19 17:31:34,967 INFO 10396 [luigi-interface] hadoop.py:273 - Job Counters
2017-01-19 17:31:34,967 INFO 10396 [luigi-interface] hadoop.py:273 - Failed reduce tasks=4
2017-01-19 17:31:34,967 INFO 10396 [luigi-interface] hadoop.py:273 - Launched map tasks=2
2017-01-19 17:31:34,968 INFO 10396 [luigi-interface] hadoop.py:273 - Launched reduce tasks=4
2017-01-19 17:31:34,968 INFO 10396 [luigi-interface] hadoop.py:273 - Data-local map tasks=2
2017-01-19 17:31:34,968 INFO 10396 [luigi-interface] hadoop.py:273 - Total time spent by all maps in occupied slots (ms)=36941
2017-01-19 17:31:34,968 INFO 10396 [luigi-interface] hadoop.py:273 - Total time spent by all reduces in occupied slots (ms)=114468
2017-01-19 17:31:34,968 INFO 10396 [luigi-interface] hadoop.py:273 - Total time spent by all map tasks (ms)=36941
2017-01-19 17:31:34,969 INFO 10396 [luigi-interface] hadoop.py:273 - Total time spent by all reduce tasks (ms)=114468
2017-01-19 17:31:34,969 INFO 10396 [luigi-interface] hadoop.py:273 - Total vcore-seconds taken by all map tasks=36941
2017-01-19 17:31:34,969 INFO 10396 [luigi-interface] hadoop.py:273 - Total vcore-seconds taken by all reduce tasks=114468
2017-01-19 17:31:34,969 INFO 10396 [luigi-interface] hadoop.py:273 - Total megabyte-seconds taken by all map tasks=37827584
2017-01-19 17:31:34,969 INFO 10396 [luigi-interface] hadoop.py:273 - Total megabyte-seconds taken by all reduce tasks=117215232
2017-01-19 17:31:34,969 INFO 10396 [luigi-interface] hadoop.py:273 - Map-Reduce Framework
2017-01-19 17:31:34,970 INFO 10396 [luigi-interface] hadoop.py:273 - Map input records=1742
2017-01-19 17:31:34,970 INFO 10396 [luigi-interface] hadoop.py:273 - Map output records=167
2017-01-19 17:31:34,970 INFO 10396 [luigi-interface] hadoop.py:273 - Map output bytes=15809
2017-01-19 17:31:34,970 INFO 10396 [luigi-interface] hadoop.py:273 - Map output materialized bytes=16155
2017-01-19 17:31:34,970 INFO 10396 [luigi-interface] hadoop.py:273 - Input split bytes=182
2017-01-19 17:31:34,971 INFO 10396 [luigi-interface] hadoop.py:273 - Combine input records=0
2017-01-19 17:31:34,971 INFO 10396 [luigi-interface] hadoop.py:273 - Spilled Records=167
2017-01-19 17:31:34,971 INFO 10396 [luigi-interface] hadoop.py:273 - Failed Shuffles=0
2017-01-19 17:31:34,971 INFO 10396 [luigi-interface] hadoop.py:273 - Merged Map outputs=0
2017-01-19 17:31:34,971 INFO 10396 [luigi-interface] hadoop.py:273 - GC time elapsed (ms)=691
2017-01-19 17:31:34,971 INFO 10396 [luigi-interface] hadoop.py:273 - CPU time spent (ms)=2000
2017-01-19 17:31:34,972 INFO 10396 [luigi-interface] hadoop.py:273 - Physical memory (bytes) snapshot=411451392
2017-01-19 17:31:34,972 INFO 10396 [luigi-interface] hadoop.py:273 - Virtual memory (bytes) snapshot=4414054400
2017-01-19 17:31:34,972 INFO 10396 [luigi-interface] hadoop.py:273 - Total committed heap usage (bytes)=317587456
2017-01-19 17:31:34,972 INFO 10396 [luigi-interface] hadoop.py:273 - File Input Format Counters
2017-01-19 17:31:34,972 INFO 10396 [luigi-interface] hadoop.py:273 - Bytes Read=686792
2017-01-19 17:31:34,973 INFO 10396 [luigi-interface] hadoop.py:273 - 17/01/19 17:31:34 ERROR streaming.StreamJob: Job not Successful!
2017-01-19 17:31:34,973 INFO 10396 [luigi-interface] hadoop.py:273 - Streaming Command Failed!
2017-01-19 17:31:35,055 ERROR 10396 [luigi-interface] worker.py:304 - [pid 10396] Worker Worker(salt=514950578, host=ip-172-30-0-176, username=hadoop, pid=10396) failed    LastDailyIpAddressOfUserTask(source=('hdfs://localhost:9000/data/',), interval=2017-01-03-2017-01-19, expand_interval=2 days, 0:00:00, pattern=('.*tracking.log.*',), date_pattern=%Y%m%d, warehouse_path=hdfs://localhost:9000/edx-analytics-pipeline/warehouse/)
Traceback (most recent call last):
  File "/var/lib/analytics-tasks/analyticstack/venv/local/lib/python2.7/site-packages/luigi/worker.py", line 292, in _run_task
    task.run()
  File "/var/lib/analytics-tasks/analyticstack/venv/local/lib/python2.7/site-packages/edx/analytics/tasks/insights/location_per_course.py", line 163, in run
    super(LastDailyIpAddressOfUserTask, self).run()
  File "/var/lib/analytics-tasks/analyticstack/venv/local/lib/python2.7/site-packages/luigi/hadoop.py", line 612, in run
    self.job_runner().run_job(self)
  File "/var/lib/analytics-tasks/analyticstack/venv/local/lib/python2.7/site-packages/luigi/hadoop.py", line 482, in run_job
    run_and_track_hadoop_job(arglist)
  File "/var/lib/analytics-tasks/analyticstack/venv/local/lib/python2.7/site-packages/luigi/hadoop.py", line 318, in run_and_track_hadoop_job
    return track_process(arglist, tracking_url_callback, env)
  File "/var/lib/analytics-tasks/analyticstack/venv/local/lib/python2.7/site-packages/luigi/hadoop.py", line 302, in track_process
    raise HadoopJobError(message + 'Also, no tracking url found.', out, err)
HadoopJobError: ('Streaming job failed with exit code 1. Also, no tracking url found.', 'packageJobJar: [/var/lib/analytics-tasks/analyticstack/venv/local/lib/python2.7/site-packages/luigi/mrrunner.py, /tmp/tmp9qSABm/packages.tar, /tmp/tmp9qSABm/job-instance.pickle, /tmp/hadoop-hadoop/hadoop-unjar3875976745421125384/] [] /tmp/streamjob9193397795945862752.jar tmpDir=null\n', '17/01/19 17:28:44 WARN streaming.StreamJob: -file option is deprecated, please use generic option -files instead.\n17/01/19 17:28:47 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032\n17/01/19 17:28:47 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032\n17/01/19 17:28:50 INFO mapred.FileInputFormat: Total input paths to process : 1\n17/01/19 17:28:50 INFO mapreduce.JobSubmitter: number of splits:2\n17/01/19 17:28:50 INFO Configuration.deprecation: mapred.job.name is deprecated. Instead, use mapreduce.job.name\n17/01/19 17:28:50 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces\n17/01/19 17:28:51 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1484841390427_0006\n17/01/19 17:28:51 INFO impl.YarnClientImpl: Submitted application application_1484841390427_0006\n17/01/19 17:28:51 INFO mapreduce.Job: The url to track the job: http://ip-172-30-0-176:8088/proxy/application_1484841390427_0006/\n17/01/19 17:28:52 INFO mapreduce.Job: Running job: job_1484841390427_0006\n17/01/19 17:29:08 INFO mapreduce.Job: Job job_1484841390427_0006 running in uber mode : false\n17/01/19 17:29:08 INFO mapreduce.Job:  map 0% reduce 0%\n17/01/19 17:29:29 INFO mapreduce.Job:  map 50% reduce 0%\n17/01/19 17:29:30 INFO mapreduce.Job:  map 100% reduce 0%\n17/01/19 17:29:44 INFO mapreduce.Job:  map 100% reduce 100%\n17/01/19 17:29:57 INFO mapreduce.Job: Task Id : attempt_1484841390427_0006_r_000000_0, Status : FAILED\nError: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1\n\tat org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:320)\n\tat org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:533)\n\tat org.apache.hadoop.streaming.PipeReducer.close(PipeReducer.java:134)\n\tat org.apache.hadoop.io.IOUtils.cleanup(IOUtils.java:237)\n\tat org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:459)\n\tat org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)\n\tat org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)\n\tat java.security.AccessController.doPrivileged(Native Method)\n\tat javax.security.auth.Subject.doAs(Subject.java:422)\n\tat org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)\n\tat org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)\n\n17/01/19 17:29:58 INFO mapreduce.Job:  map 100% reduce 0%\n17/01/19 17:30:19 INFO mapreduce.Job:  map 100% reduce 100%\n17/01/19 17:30:32 INFO mapreduce.Job: Task Id : attempt_1484841390427_0006_r_000000_1, Status : FAILED\nError: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1\n\tat org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:320)\n\tat org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:533)\n\tat org.apache.hadoop.streaming.PipeReducer.close(PipeReducer.java:134)\n\tat org.apache.hadoop.io.IOUtils.cleanup(IOUtils.java:237)\n\tat org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:459)\n\tat org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)\n\tat org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)\n\tat java.security.AccessController.doPrivileged(Native Method)\n\tat javax.security.auth.Subject.doAs(Subject.java:422)\n\tat org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)\n\tat org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)\n\n17/01/19 17:30:33 INFO mapreduce.Job:  map 100% reduce 0%\n17/01/19 17:30:48 INFO mapreduce.Job:  map 100% reduce 100%\n17/01/19 17:31:01 INFO mapreduce.Job: Task Id : attempt_1484841390427_0006_r_000000_2, Status : FAILED\nError: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1\n\tat org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:320)\n\tat org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:533)\n\tat org.apache.hadoop.streaming.PipeReducer.close(PipeReducer.java:134)\n\tat org.apache.hadoop.io.IOUtils.cleanup(IOUtils.java:237)\n\tat org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:459)\n\tat org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)\n\tat org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)\n\tat java.security.AccessController.doPrivileged(Native Method)\n\tat javax.security.auth.Subject.doAs(Subject.java:422)\n\tat org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)\n\tat org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)\n\n17/01/19 17:31:02 INFO mapreduce.Job:  map 100% reduce 0%\n17/01/19 17:31:20 INFO mapreduce.Job:  map 100% reduce 100%\n17/01/19 17:31:34 INFO mapreduce.Job: Job job_1484841390427_0006 failed with state FAILED due to: Task failed task_1484841390427_0006_r_000000\nJob failed as tasks failed. failedMaps:0 failedReduces:1\n\n17/01/19 17:31:34 INFO mapreduce.Job: Counters: 37\n\tFile System Counters\n\t\tFILE: Number of bytes read=0\n\t\tFILE: Number of bytes written=195423\n\t\tFILE: Number of read operations=0\n\t\tFILE: Number of large read operations=0\n\t\tFILE: Number of write operations=0\n\t\tHDFS: Number of bytes read=686974\n\t\tHDFS: Number of bytes written=0\n\t\tHDFS: Number of read operations=6\n\t\tHDFS: Number of large read operations=0\n\t\tHDFS: Number of write operations=0\n\tJob Counters \n\t\tFailed reduce tasks=4\n\t\tLaunched map tasks=2\n\t\tLaunched reduce tasks=4\n\t\tData-local map tasks=2\n\t\tTotal time spent by all maps in occupied slots (ms)=36941\n\t\tTotal time spent by all reduces in occupied slots (ms)=114468\n\t\tTotal time spent by all map tasks (ms)=36941\n\t\tTotal time spent by all reduce tasks (ms)=114468\n\t\tTotal vcore-seconds taken by all map tasks=36941\n\t\tTotal vcore-seconds taken by all reduce tasks=114468\n\t\tTotal megabyte-seconds taken by all map tasks=37827584\n\t\tTotal megabyte-seconds taken by all reduce tasks=117215232\n\tMap-Reduce Framework\n\t\tMap input records=1742\n\t\tMap output records=167\n\t\tMap output bytes=15809\n\t\tMap output materialized bytes=16155\n\t\tInput split bytes=182\n\t\tCombine input records=0\n\t\tSpilled Records=167\n\t\tFailed Shuffles=0\n\t\tMerged Map outputs=0\n\t\tGC time elapsed (ms)=691\n\t\tCPU time spent (ms)=2000\n\t\tPhysical memory (bytes) snapshot=411451392\n\t\tVirtual memory (bytes) snapshot=4414054400\n\t\tTotal committed heap usage (bytes)=317587456\n\tFile Input Format Counters \n\t\tBytes Read=686792\n17/01/19 17:31:34 ERROR streaming.StreamJob: Job not Successful!\nStreaming Command Failed!\n')
2017-01-19 17:31:35,057 INFO 10396 [luigi-interface] notifications.py:96 - Skipping error email. Set `error-email` in the `core` section of the luigi config file to receive error emails.
2017-01-19 17:31:43,277 INFO 10396 [luigi-interface] worker.py:337 - Done
2017-01-19 17:31:43,278 INFO 10396 [luigi-interface] worker.py:338 - There are no more tasks to run at this time
2017-01-19 17:31:43,278 INFO 10396 [luigi-interface] worker.py:343 - There are 1 pending tasks possibly being run by other workers
2017-01-19 17:31:43,279 INFO 10396 [luigi-interface] worker.py:117 - Worker Worker(salt=514950578, host=ip-172-30-0-176, username=hadoop, pid=10396) was stopped. Shutting down Keep-Alive thread
Connection to localhost closed.