Marcin cinek810
cinek810
/ hive -success
Created
December 8, 2020 15:21
final success example for https://funinit.wordpress.com/2020/12/08/how-to-start-apache-hive-without-hdfs/
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # cat /home/vagrant/external_path/* | |
| 1,jane,doe,senior,mathematics | |
| 2,john,smith,junior,engineering | |
| # hive | |
| which: no hbase in (/home/vagrant/hadoop/hadoop-3.3.0/bin/:/home/vagrant/hadoop/apache-hive-2.3.7-bin/bin:/home/vagrant/hadoop/jre1.8.0_271/bin:/home/vagrant/hadoop/hadoop-3.3.0/bin/:/home/vagrant/hadoop/apache-hive-2.3.7-bin/bin:/home/vagrant/hadoop/jre1.8.0_271/bin:/mnt/slurm/bin:/sbin:/bin:/usr/sbin:/usr/bin:/home/vagrant/hadoop/hadoop-2.10.1//sbin:/home/vagrant/hadoop/hadoop-2.10.1//bin:/home/vagrant/hadoop/hadoop-2.10.1//sbin:/home/vagrant/hadoop/hadoop-2.10.1//bin) | |
| SLF4J: Class path contains multiple SLF4J bindings. | |
| SLF4J: Found binding in [jar:file:/home/vagrant/hadoop/apache-hive-2.3.7-bin/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class] | |
| SLF4J: Found binding in [jar:file:/home/vagrant/hadoop/hadoop-2.10.1/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] | |
| SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. |
cinek810
/ hive - gulava
Created
December 8, 2020 15:19
Error example for https://funinit.wordpress.com/2020/12/08/how-to-start-apache-hive-without-hdfs/
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #hive | |
| SLF4J: Class path contains multiple SLF4J bindings. | |
| SLF4J: Found binding in [jar:file:/home/cinek/hadoop/apache-hive-3.1.2-bin/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] | |
| SLF4J: Found binding in [jar:file:/home/cinek/hadoop/hadoop-3.1.4/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] | |
| SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. | |
| SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] | |
| Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V | |
| at org.apache.hadoop.conf.Configuration.set(Configuration.java:1357) | |
| at org.apache.hadoop.conf.Configuration.set(Configuration.java:1338) | |
| at org.apache.hadoop.mapred.JobConf.setJar(JobConf.java:518) |
cinek810
/ hive - initialize database
Created
December 8, 2020 15:17
https://funinit.wordpress.com/2020/12/08/how-to-start-apache-hive-without-hdfs/
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # cd tmp | |
| # ls | |
| metastore_db | |
| # rm -rf metastore_db/ | |
| # schematool -initSchema -dbType derby | |
| SLF4J: Class path contains multiple SLF4J bindings. | |
| SLF4J: Found binding in [jar:file:/home/vagrant/hadoop/apache-hive-2.3.7-bin/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class] | |
| SLF4J: Found binding in [jar:file:/home/vagrant/hadoop/hadoop-2.10.1/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] | |
| SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. | |
| SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] |
cinek810
/ setStripe.c
Created
November 13, 2020 12:46
[funinit.wordpress.com] setStripe.c library for core.py use
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #include <lustre/lustreapi.h> | |
| #include <stdio.h> | |
| //To build: gcc -llustreapi /path/to/setStripe.c -fPIC -shared -o /etc/irods/setStripe.so | |
| int setStripe(char *path, int offset) | |
| { | |
| int rc; | |
| char pool[100]; | |
| sprintf(pool, "pool%02d", offset); | |
| rc = llapi_file_open_pool(path, O_DIRECTORY, 0, 0, -1, -1, LOV_PATTERN_RAID0, pool); | |
| if (rc < 0) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| from ctypes import CDLL | |
| lustreFunc = CDLL("/etc/irods/setStripe.so") | |
| [...] | |
| def Pyrule_resource_mkdir_post(rule_args,callback,rei): | |
| dataObj = splitKV(rule_args[0]) | |
| physPath = str(dataObj['physical_path']) | |
| rc = lustreFunc.setStripe(physPath, pool) | |
| if rc: | |
| callback.writeLine("serverLog","setting pool failed with:'"+str(rc)+"'"); |
cinek810
/ slurm_commands.py
Last active
December 3, 2019 16:28
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import gdb | |
| #rewrite slurm_msg_type to array num->str | |
| slurm_msg_type = gdb.lookup_type("slurm_msg_type_t") | |
| rpc_num2str=dict() | |
| for name,field in slurm_msg_type.items(): | |
| rpc_num2str[field.enumval]=name |
cinek810
/ gathering_facts_profile.out
Created
July 6, 2019 12:30
Top 10 gathering facts results (funinit.wordpress.com)
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| $ python ./setup.py execute |grep 'facts/hardware' | head -10 | |
| 1 0.000 0.000 0.408 0.408 /home/cinek/.ansible/tmp/ansible-tmp-1562414269.69-180055591295251/debug_dir/ansible/module_utils/facts/hardware/base.py:56(collect) | |
| 1 0.000 0.000 0.408 0.408 /home/cinek/.ansible/tmp/ansible-tmp-1562414269.69-180055591295251/debug_dir/ansible/module_utils/facts/hardware/linux.py:79(populate) | |
| 1 0.027 0.027 0.264 0.264 /home/cinek/.ansible/tmp/ansible-tmp-1562414269.69-180055591295251/debug_dir/ansible/module_utils/facts/hardware/linux.py:506(get_device_facts) | |
| 1 0.000 0.000 0.143 0.143 /home/cinek/.ansible/tmp/ansible-tmp-1562414269.69-180055591295251/debug_dir/ansible/module_utils/facts/hardware/linux.py:418(get_mount_facts) | |
| 1 0.000 0.000 0.097 0.097 /home/cinek/.ansible/tmp/ansible-tmp-1562414269.69-180055591295251/debug_dir/ansible/module_utils/facts/hardware/linux.py:345(_lsblk_uuid) | |
| 1 0.000 0.000 0.096 |
cinek810
/ ansible_module_setup.py.patch
Created
May 25, 2019 09:54
Enable profiling for setup module
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| --- ansible_module_setup.py.bak 2019-05-25 09:51:33.891203596 +0000 | |
| +++ ansible_module_setup.py 2019-05-25 09:53:31.361990970 +0000 | |
| @@ -159,6 +159,11 @@ | |
| # rename namespace_name to root_key? | |
| namespace = PrefixFactNamespace(namespace_name='ansible', | |
| prefix='ansible_') | |
| + #Enable profiling of interesting code | |
| + import cProfile, pstats | |
| + prof=cProfile.Profile() | |
| + prof.enable() |
cinek810
/ failing_playbok.yml
Created
May 13, 2019 14:55
The root cause of strange ansible failure was in my playbook
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| - hosts: | |
| - | |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| @staticmethod | |
| def load(data, variable_manager=None, loader=None): | |
| print(data['hosts']) | |
| if ('name' not in data or data['name'] is None) and 'hosts' in data: | |
| if isinstance(data['hosts'], list): | |
| data['name'] = ','.join(data['hosts']) |
NewerOlder