Skip to content

Instantly share code, notes, and snippets.

@majidalfifi

majidalfifi/fix for spark lzo native dependency in CDH

Last active January 17, 2018 07:22
Show Gist options
  • Save majidalfifi/4063b62ce1a3ad0e6672bdb6df8281da to your computer and use it in GitHub Desktop.
Save majidalfifi/4063b62ce1a3ad0e6672bdb6df8281da to your computer and use it in GitHub Desktop.
Download ZIP
Raw
fix for spark lzo native dependency in CDH
I was getting the following error in the spark-shell when trying to load lzo files (CDH 5.4)
ERROR GPLNativeCodeLoader: Could not load native gpl library
java.lang.UnsatisfiedLinkError: no gplcompression in java.library.path
The fix invovles making spark-shell aware of this dependency by either of the follwoing
1. export LD_LIBRARY_PATH=/opt/cloudera/parcels/GPLEXTRAS/lib/hadoop/lib/native/
2. add saftey value in spark configuration in cloudera manager
Spark (Standalone) Service Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-env.sh
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/cloudera/parcels/GPLEXTRAS/lib/hadoop/lib/native/
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment