Skip to content

Instantly share code, notes, and snippets.

@zz22394

zz22394/ln.command.sh

Created February 17, 2019 16:36
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save zz22394/6c004731423fb11095aa41ac19de3393 to your computer and use it in GitHub Desktop.
Save zz22394/6c004731423fb11095aa41ac19de3393 to your computer and use it in GitHub Desktop.
Download ZIP
Build symbolic link for atlas - sqoop hook lib
Raw
ln.command.sh
sudo ln -s /usr/hdp/current/atlas-server/hook/sqoop/atlas-sqoop-plugin-impl/atlas-client-common-1.1.0.3.1.0.0-78.jar /usr/hdp/current/sqoop-server/lib/zz-atlas-client-common-1.1.0.3.1.0.0-78.jar
sudo ln -s /usr/hdp/current/atlas-server/hook/sqoop/atlas-sqoop-plugin-impl/atlas-client-v1-1.1.0.3.1.0.0-78.jar /usr/hdp/current/sqoop-server/lib/zz-atlas-client-v1-1.1.0.3.1.0.0-78.jar
sudo ln -s /usr/hdp/current/atlas-server/hook/sqoop/atlas-sqoop-plugin-impl/atlas-client-v2-1.1.0.3.1.0.0-78.jar /usr/hdp/current/sqoop-server/lib/zz-atlas-client-v2-1.1.0.3.1.0.0-78.jar
sudo ln -s /usr/hdp/current/atlas-server/hook/sqoop/atlas-sqoop-plugin-impl/atlas-common-1.1.0.3.1.0.0-78.jar /usr/hdp/current/sqoop-server/lib/zz-atlas-common-1.1.0.3.1.0.0-78.jar
sudo ln -s /usr/hdp/current/atlas-server/hook/sqoop/atlas-sqoop-plugin-impl/atlas-intg-1.1.0.3.1.0.0-78.jar /usr/hdp/current/sqoop-server/lib/zz-atlas-intg-1.1.0.3.1.0.0-78.jar
sudo ln -s /usr/hdp/current/atlas-server/hook/sqoop/atlas-sqoop-plugin-impl/atlas-notification-1.1.0.3.1.0.0-78.jar /usr/hdp/current/sqoop-server/lib/zz-atlas-notification-1.1.0.3.1.0.0-78.jar
sudo ln -s /usr/hdp/current/atlas-server/hook/sqoop/atlas-sqoop-plugin-impl/hdfs-model-1.1.0.3.1.0.0-78.jar /usr/hdp/current/sqoop-server/lib/zz-hdfs-model-1.1.0.3.1.0.0-78.jar
sudo ln -s /usr/hdp/current/atlas-server/hook/sqoop/atlas-sqoop-plugin-impl/hive-bridge-1.1.0.3.1.0.0-78.jar /usr/hdp/current/sqoop-server/lib/zz-hive-bridge-1.1.0.3.1.0.0-78.jar
sudo ln -s /usr/hdp/current/atlas-server/hook/sqoop/atlas-sqoop-plugin-impl/jersey-json-1.19.jar /usr/hdp/current/sqoop-server/lib/zz-jersey-json-1.19.jar
sudo ln -s /usr/hdp/current/atlas-server/hook/sqoop/atlas-sqoop-plugin-impl/jsr311-api-1.1.jar /usr/hdp/current/sqoop-server/lib/zz-jsr311-api-1.1.jar
sudo ln -s /usr/hdp/current/atlas-server/hook/sqoop/atlas-sqoop-plugin-impl/kafka-clients-2.0.0.3.1.0.0-78.jar /usr/hdp/current/sqoop-server/lib/zz-kafka-clients-2.0.0.3.1.0.0-78.jar
sudo ln -s /usr/hdp/current/atlas-server/hook/sqoop/atlas-sqoop-plugin-impl/kafka_2.11-2.0.0.3.1.0.0-78.jar /usr/hdp/current/sqoop-server/lib/zz-kafka_2.11-2.0.0.3.1.0.0-78.jar
sudo ln -s /usr/hdp/current/atlas-server/hook/sqoop/atlas-sqoop-plugin-impl/sqoop-bridge-1.1.0.3.1.0.0-78.jar /usr/hdp/current/sqoop-server/lib/zz-sqoop-bridge-1.1.0.3.1.0.0-78.jar
sudo ln -s /usr/hdp/current/atlas-server/hook/sqoop/atlas-sqoop-plugin-impl/atlas-client-common-1.1.0.3.1.0.0-78.jar /usr/hdp/current/sqoop-client/lib/zz-atlas-client-common-1.1.0.3.1.0.0-78.jar
sudo ln -s /usr/hdp/current/atlas-server/hook/sqoop/atlas-sqoop-plugin-impl/atlas-client-v1-1.1.0.3.1.0.0-78.jar /usr/hdp/current/sqoop-client/lib/zz-atlas-client-v1-1.1.0.3.1.0.0-78.jar
sudo ln -s /usr/hdp/current/atlas-server/hook/sqoop/atlas-sqoop-plugin-impl/atlas-client-v2-1.1.0.3.1.0.0-78.jar /usr/hdp/current/sqoop-client/lib/zz-atlas-client-v2-1.1.0.3.1.0.0-78.jar
sudo ln -s /usr/hdp/current/atlas-server/hook/sqoop/atlas-sqoop-plugin-impl/atlas-common-1.1.0.3.1.0.0-78.jar /usr/hdp/current/sqoop-client/lib/zz-atlas-common-1.1.0.3.1.0.0-78.jar
sudo ln -s /usr/hdp/current/atlas-server/hook/sqoop/atlas-sqoop-plugin-impl/atlas-intg-1.1.0.3.1.0.0-78.jar /usr/hdp/current/sqoop-client/lib/zz-atlas-intg-1.1.0.3.1.0.0-78.jar
sudo ln -s /usr/hdp/current/atlas-server/hook/sqoop/atlas-sqoop-plugin-impl/atlas-notification-1.1.0.3.1.0.0-78.jar /usr/hdp/current/sqoop-client/lib/zz-atlas-notification-1.1.0.3.1.0.0-78.jar
sudo ln -s /usr/hdp/current/atlas-server/hook/sqoop/atlas-sqoop-plugin-impl/hdfs-model-1.1.0.3.1.0.0-78.jar /usr/hdp/current/sqoop-client/lib/zz-hdfs-model-1.1.0.3.1.0.0-78.jar
sudo ln -s /usr/hdp/current/atlas-server/hook/sqoop/atlas-sqoop-plugin-impl/hive-bridge-1.1.0.3.1.0.0-78.jar /usr/hdp/current/sqoop-client/lib/zz-hive-bridge-1.1.0.3.1.0.0-78.jar
sudo ln -s /usr/hdp/current/atlas-server/hook/sqoop/atlas-sqoop-plugin-impl/jersey-json-1.19.jar /usr/hdp/current/sqoop-client/lib/zz-jersey-json-1.19.jar
sudo ln -s /usr/hdp/current/atlas-server/hook/sqoop/atlas-sqoop-plugin-impl/jsr311-api-1.1.jar /usr/hdp/current/sqoop-client/lib/zz-jsr311-api-1.1.jar
sudo ln -s /usr/hdp/current/atlas-server/hook/sqoop/atlas-sqoop-plugin-impl/kafka-clients-2.0.0.3.1.0.0-78.jar /usr/hdp/current/sqoop-client/lib/zz-kafka-clients-2.0.0.3.1.0.0-78.jar
sudo ln -s /usr/hdp/current/atlas-server/hook/sqoop/atlas-sqoop-plugin-impl/kafka_2.11-2.0.0.3.1.0.0-78.jar /usr/hdp/current/sqoop-client/lib/zz-kafka_2.11-2.0.0.3.1.0.0-78.jar
sudo ln -s /usr/hdp/current/atlas-server/hook/sqoop/atlas-sqoop-plugin-impl/sqoop-bridge-1.1.0.3.1.0.0-78.jar /usr/hdp/current/sqoop-client/lib/zz-sqoop-bridge-1.1.0.3.1.0.0-78.jar
@zz22394
Copy link
Author

zz22394 commented Feb 17, 2019 

19/02/17 16:23:11 INFO tool.ImportTool: Publishing Hive/Hcat import job data to Listeners for table test_table_sqoop1
19/02/17 16:23:11 INFO atlas.ApplicationProperties: Looking for atlas-application.properties in classpath
19/02/17 16:23:11 INFO atlas.ApplicationProperties: Loading atlas-application.properties from file:/etc/sqoop/3.1.0.0-78/0/atlas-application.properties
19/02/17 16:23:11 INFO atlas.ApplicationProperties: No graphdb backend specified. Will use 'janus'
19/02/17 16:23:11 INFO atlas.ApplicationProperties: Using storage backend 'hbase2'
19/02/17 16:23:11 INFO atlas.ApplicationProperties: Using index backend 'solr'
19/02/17 16:23:11 INFO atlas.ApplicationProperties: Setting solr-wait-searcher property 'true'
19/02/17 16:23:11 INFO atlas.ApplicationProperties: Setting index.search.map-name property 'false'
19/02/17 16:23:12 INFO atlas.ApplicationProperties: Property (set to default) atlas.graph.cache.db-cache = true
19/02/17 16:23:12 INFO atlas.ApplicationProperties: Property (set to default) atlas.graph.cache.db-cache-clean-wait = 20
19/02/17 16:23:12 INFO atlas.ApplicationProperties: Property (set to default) atlas.graph.cache.db-cache-size = 0.5
19/02/17 16:23:12 INFO atlas.ApplicationProperties: Property (set to default) atlas.graph.cache.tx-cache-size = 15000
19/02/17 16:23:12 INFO atlas.ApplicationProperties: Property (set to default) atlas.graph.cache.tx-dirty-size = 120
19/02/17 16:23:12 ERROR security.InMemoryJAASConfiguration: Unable to add JAAS configuration for client [KafkaClient] as it is missing param [atlas.jaas.KafkaClient.loginModuleName]. Skipping JAAS config for [KafkaClient]
19/02/17 16:23:12 INFO kafka.KafkaNotification: ==> KafkaNotification()
19/02/17 16:23:12 INFO kafka.KafkaNotification: <== KafkaNotification()
19/02/17 16:23:12 INFO hook.AtlasHook: Created Atlas Hook
19/02/17 16:23:12 INFO mapreduce.PublishJobData: Published data is Operation=import, Url=jdbc:mysql://localhost/test?zeroDateTimeBehavior=convertToNull, User=centos, StoreType=mysql, StoreTable=test_table_sqoop1, StoreQuery=null, HiveDB=default, HiveTable=test_hive_table1, StartTime=1550420559485, EndTime=1550420591914, CmdLineArgs={reset.onemapper=false, codegen.output.delimiters.enclose=0, sqlconnection.metadata.transaction.isolation.level=2, codegen.input.delimiters.escape=0, codegen.auto.compile.dir=true, accumulo.batch.size=10240000, codegen.input.delimiters.field=0, accumulo.create.table=false, mainframe.input.dataset.type=p, enable.compression=false, skip.dist.cache=false, hive.compute.stats.table=false, hive.table.name=test_hive_table1, accumulo.max.latency=5000, db.username=root, sqoop.throwOnError=false, db.clear.staging.table=false, codegen.input.delimiters.enclose=0, hdfs.append.dir=false, import.direct.split.size=0, hcatalog.drop.and.create.table=false, codegen.output.delimiters.record=10, codegen.output.delimiters.field=1, hbase.bulk.load.enabled=false, mapreduce.num.mappers=1, export.new.update=UpdateOnly, db.require.password=true, hive.import=true, customtool.options.jsonmap={}, hdfs.delete-target.dir=false, codegen.output.delimiters.enclose.required=false, direct.import=false, codegen.output.dir=., hdfs.file.format=TextFile, hive.drop.delims=false, codegen.input.delimiters.record=0, db.batch=false, codegen.delete.compile.dir=false, split.limit=null, hcatalog.create.table=false, hive.fail.table.exists=false, hive.overwrite.table=false, incremental.mode=None, temporary.dirRoot=_sqoop, verbose=false, hbase.null.incremental.mode=Ignore, import.max.inline.lob.size=16777216, import.fetch.size=1, codegen.input.delimiters.enclose.required=false, relaxed.isolation=false, sqoop.oracle.escaping.disabled=true, db.table=test_table_sqoop1, hbase.create.table=false, codegen.compile.dir=/tmp/sqoop-centos/compile/3a0011318880684ddc36dfa828e3d30a, codegen.output.delimiters.escape=0, db.connect.string=jdbc:mysql://localhost/test?zeroDateTimeBehavior=convertToNull}
19/02/17 16:23:12 INFO hook.AtlasHook: ==> Shutdown of Atlas Hook
19/02/17 16:23:12 INFO kafka.KafkaNotification: ==> KafkaNotification.createProducer()
19/02/17 16:23:12 INFO producer.ProducerConfig: ProducerConfig values:
	compression.type = none
	metric.reporters = []
	metadata.max.age.ms = 300000
	metadata.fetch.timeout.ms = 60000
	acks = 1
	batch.size = 16384
	reconnect.backoff.ms = 10
	bootstrap.servers = [zzeng-hdp-3.field.hortonworks.com:6667, zzeng-hdp-1.field.hortonworks.com:6667, zzeng-hdp-2.field.hortonworks.com:6667]
	receive.buffer.bytes = 32768
	retry.backoff.ms = 100
	buffer.memory = 33554432
	timeout.ms = 30000
	key.serializer = class org.apache.kafka.common.serialization.StringSerializer
	retries = 0
	max.request.size = 1048576
	block.on.buffer.full = true
	value.serializer = class org.apache.kafka.common.serialization.StringSerializer
	metrics.sample.window.ms = 30000
	send.buffer.bytes = 131072
	max.in.flight.requests.per.connection = 5
	metrics.num.samples = 2
	linger.ms = 0
	client.id =

19/02/17 16:23:12 WARN producer.ProducerConfig: The configuration key.deserializer = null was supplied but isn't a known config.
19/02/17 16:23:12 WARN producer.ProducerConfig: The configuration value.deserializer = null was supplied but isn't a known config.
19/02/17 16:23:12 WARN producer.ProducerConfig: The configuration hook.group.id = null was supplied but isn't a known config.
19/02/17 16:23:12 WARN producer.ProducerConfig: The configuration zookeeper.connection.timeout.ms = null was supplied but isn't a known config.
19/02/17 16:23:12 WARN producer.ProducerConfig: The configuration zookeeper.session.timeout.ms = null was supplied but isn't a known config.
19/02/17 16:23:12 WARN producer.ProducerConfig: The configuration enable.auto.commit = null was supplied but isn't a known config.
19/02/17 16:23:12 WARN producer.ProducerConfig: The configuration zookeeper.connect = null was supplied but isn't a known config.
19/02/17 16:23:12 WARN producer.ProducerConfig: The configuration zookeeper.sync.time.ms = null was supplied but isn't a known config.
19/02/17 16:23:12 WARN producer.ProducerConfig: The configuration session.timeout.ms = null was supplied but isn't a known config.
19/02/17 16:23:12 WARN producer.ProducerConfig: The configuration auto.offset.reset = null was supplied but isn't a known config.
19/02/17 16:23:12 INFO kafka.KafkaNotification: <== KafkaNotification.createProducer()
19/02/17 16:23:12 INFO hook.AtlasHook: <== Shutdown of Atlas Hook

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment