Suresh Saggar sureshsaggar
sureshsaggar
/ Setting up STUD network proxy
Last active
December 10, 2015 01:58
Setting up STUD network proxy
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Specify the configuration in a file and test the same via the "-t" option. | |
| root:~/stud# ./stud -t --config=ss_default_config.conf --ssl | |
| Trying to initialize SSL contexts with your certificates{core} | |
| Note: no DH parameters found in /root/stud/saggar.in.includesprivatekey.pem | |
| stud configuration looks ok. | |
| STUD creates two separate process i.e. do a ps from a different screen and you will see 2 pids: | |
| root:~/stud# ps -ef| grep stud |
sureshsaggar
/ bucket_mqtt_logs.pl
Last active
December 10, 2015 01:48
Perl script to categorize log files in separate output directories depending on the file suffix.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #!/usr/bin/perl | |
| use strict; | |
| use warnings; | |
| use File::Copy; | |
| my $usage = <<'END'; | |
| # Usage: perl $0 <input_logs_dir> <output_logs_dir> <suffix_regex?> | |
| # Example: perl bucket_mqtt_logs.pl ./snapshots/ ./snapshots/ 'log\.((\d{4})-(\d{2})-(\d{2}))-(\d{2})' | |
| END | |
sureshsaggar
/ Apache Oozie - JA009: java.io.IOException: Invalid job requirements.
Created
November 27, 2012 05:30
Apache Oozie - JA009: java.io.IOException: Invalid job requirements.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| 2012-11-26 18:46:14,525 WARN ActionStartXCommand:542 - USER[hduser] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000000-121126183401740-oozie-hdus-W] ACTION[0000000-121126183401740-oozie-hdus-W@mr-node] Error starting action [mr-node]. ErrorType [TRANSIENT], ErrorCode [JA009], Message [JA009: java.io.IOException: job_201211260754_0024(-1 memForMapTasks -1 memForReduceTasks): Invalid job requirements. | |
| at org.apache.hadoop.mapred.JobTracker.checkMemoryRequirements(JobTracker.java:4992) | |
| at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3752) | |
| at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3695) | |
| at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source) | |
| at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) | |
| at java.lang.reflect.Method.invoke(Method.java:601) | |
| at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578) | |
| at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393) | |
| at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389) |
sureshsaggar
/ gist:4129687
Created
November 22, 2012 06:33
Apache PIG - java.lang.UnsupportedOperationException: getJobTrackerAddrs is not supported
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1002: Unable to store alias XYZ_active_users | |
| at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1552) | |
| at org.apache.pig.PigServer.registerQuery(PigServer.java:540) | |
| at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:970) | |
| at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:386) | |
| at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:189) | |
| at org.apache.pig.PigServer.registerScript(PigServer.java:614) | |
| at org.apache.pig.PigServer.registerScript(PigServer.java:716) | |
| at org.apache.pig.PigServer.registerScript(PigServer.java:689) | |
| at com.bsb.hike.analytics.pig.BaseHikePigTask.run(BaseHikePigTask.java:148) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| $ hadoop fs -cat /tmp/agent-mqtt/FlumeData.1351679289119 | |
| ?64 bytes from www.google.com (114.121.18.9): icmp_req=4468 ttl=48 time=118 ms?64 bytes from www.google.com (114.121.18.9): icmp_req=4469 ttl=48 time=116 ms?64 bytes from www.google.com (114.121.18.9): icmp_req=4470 ttl=48 time=118 ms?64 bytes from www.google.com (114.121.18.9): icmp_req=4471 ttl=48 time=116 ms?64 bytes from www.google.com (114.121.18.9): icmp_req=4472 ttl=48 time=116 ms?64 bytes from www.google.com (114.121.18.9): icmp_req=4473 ttl=48 time=117 ms?64 bytes from www.google.com (114.121.18.9): icmp_req=4474 ttl=48 time=118 ms?64 bytes from www.google.com (114.121.18.9): icmp_req=4475 ttl=48 time=116 ms?64 bytes from www.google.com (114.121.18.9): icmp_req=4476 ttl=48 time=116 ms?64 bytes from www.google.com (114.121.18.9): icmp_req=4477 ttl=48 time=118 ms?G?\?C?a?????Ӷ |
sureshsaggar
/ Agent 01 - Using Exec Source & AVRO Sink
Created
November 1, 2012 13:32
Query: Saving data in _raw_ format in HDFS using FlumeNG?
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # Usage: | |
| # bin/flume-ng agent --conf ./conf/ -f conf/flume-webserver-agents.conf -Dflume.root.logger=DEBUG,console -n agent_hikemon | |
| # ------------------------------------------------------------------------------------ | |
| # Agent 01 - Using Exec Source & AVRO Sink | |
| # ------------------------------------------------------------------------------------ | |
| agent_hikemon.sinks = avroSink-consolidator | |
| agent_hikemon.sources = tailSource | |
| agent_hikemon.channels = MemoryChannel-WebServer | |
| agent_hikemon.channels.MemoryChannel-WebServer.type = memory |
sureshsaggar
/ Output from HDFS - Flume NG
Created
October 31, 2012 16:27
Output from HDFS - Flume NG
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| root@hadoop2-prod:~# /usr/local/hadoop/bin/hadoop fs -cat /tmp/agent-webserver/FlumeData.1351688879208 | |
| 64 bytes from www.google.com (72.15.18.10): icmp_req=275 ttl=48 time=111 ms | |
| 64 bytes from www.google.com (72.15.18.10): icmp_req=276 ttl=48 time=114 ms | |
| 64 bytes from www.google.com (72.15.18.10): icmp_req=277 ttl=48 time=114 ms | |
| 64 bytes from www.google.com (72.15.18.10): icmp_req=278 ttl=48 time=112 ms | |
| 64 bytes from www.google.com (72.15.18.10): icmp_req=279 ttl=48 time=114 ms | |
| 64 bytes from www.google.com (72.15.18.10): icmp_req=280 ttl=48 time=114 ms | |
| 64 bytes from www.google.com (72.15.18.10): icmp_req=281 ttl=48 time=114 ms | |
| 64 bytes from www.google.com (72.15.18.10): icmp_req=282 ttl=48 time=111 ms | |
| 64 bytes from www.google.com (72.15.18.10): icmp_req=283 ttl=48 time=114 ms |
sureshsaggar
/ Apache FlumeNG and HDFS Sink
Created
October 31, 2012 16:16
Apache FlumeNG and HDFS Sink
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # Directory: | |
| # root@localhost:~/flume/apache-flume-1.4.0-SNAPSHOT | |
| # Usage: | |
| # bin/flume-ng agent --conf ./conf/ -f conf/flume-agents-tests.conf -Dflume.root.logger=DEBUG,console -n agent_test_exec_TO_hdfs | |
| # ------------------------------------------------------------------------------------ | |
| # This workflow applies to some WebServer running flume agent and dumping data | |
| # back in HDFS. Here /tmp/ping.txt could be any log file. | |
| # ------------------------------------------------------------------------------------ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Step01: Verify that some suitable AVD exists: | |
| Sureshs-MacBook-Pro:android-sdk-macosx sureshsaggar$ tools/android list avd | |
| Available Android Virtual Devices: | |
| Name: AndroidApp | |
| Path: /Users/sureshsaggar/.android/avd/AndroidApp.avd | |
| Target: Google APIs (Google Inc.) | |
| Based on Android 2.3.3 (API level 10) | |
| ABI: armeabi | |
| Skin: HVGA |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Step 01: Find list of devices attached using adb: | |
| /Users/sureshsaggar/android-sdks/platform-tools/adb devices | |
| List of devices attached | |
| 222222222cb65fdf device | |
| Step 02: Set ANDROID_HOME | |
| export ANDROID_HOME=/Users/sureshsaggar/Downloads/android-sdk-macosx | |
| echo $ANDROID_HOME | |
| /Users/sureshsaggar/Downloads/android-sdk-macosx |