Skip to content

Instantly share code, notes, and snippets.

@romanegloo

romanegloo/installation.md

Created February 2, 2019 03:41
Show Gist options
  • Star 4 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save romanegloo/458a4d206e2fca8105348c332257ffe7 to your computer and use it in GitHub Desktop.
Save romanegloo/458a4d206e2fca8105348c332257ffe7 to your computer and use it in GitHub Desktop.
Download ZIP
PyLucene Installation
Raw
installation.md

On Mac OS

Java

Both of Oracle Java 1.8 and Apple's Java 1.6 are required. You can download Oracle Java 1.8 from here.

For Apple's Java 1.6, download the package from here and install it.

Set JAVA_HOME to make Java1.8 as default. Add this line to .baserc or .bash_profile,

export JAVA_HOME=`/usr/libexec/java_home`

and reload the configuration

source ~/.bashrc

JCC

JCC is a C++ code generator that produces a C++ object interface wrapping a Java library. Build JCC module. JCC requires JDK and C++ compiler.

First download the source file, and build it using python setuptools.

$ cd ~/Downloads
$ svn co http://svn.apache.org/repos/asf/lucene/pylucene/trunk/jcc jcc
$ cd jcc
$ python setup.py build
$ sudo python setup.py install

ANT

ANT is a Java library and command-line tool that help building software. Download a binary or source file from here.

$ wget http://mirror.olnevhost.net/pub/apache//ant/binaries/apache-ant-1.10.5-bin.zip
$ unzip apache-ant-1.10.5-bin.zip
$ sudo mv apache-ant-1.10.5-bin.zip /usr/local/

Add the bin path to the shell environment.

export PATH="/usr/local/apache-ant-1.10.5/bin:${PATH}"

PyLucene

Download the latest PyLucene and compile it.

$ cd ~/Downloads
$ wget http://apache.claz.org/lucene/pylucene/pylucene-7.6.0-src.tar.gz
$ tar zxvf pylucene-7.6.0-src.tar.gz
$ cd pylucene-7.6.0

Change the Makefile appropriately. I modified the settings as below:

PREFIX_PYTHON=/usr/local
ANT=/usr/local/apache-ant-1.10.5/bin/ant
PYTHON=$(PREFIX_PYTHON)/bin/python3
JCC=$(PYTHON) -m jcc --shared --arch x86_64
NUM_FILES=8

Run make

$ make
... It will take a while compiling files
$ make test
$ sudo make install

Check if it runs without a problem.

$ python3 -c "import lucene; lucene.initVM()"

On Linux

JAVA

Install Oracle JAVA 1.8

First, add the PPA, and update the package index

sudo add-apt-repository ppa:webupd8team/java
sudo apt update

Install the JAVA installer script

sudo apt install oracle-java8-installer

Check if JAVA 1.8 is installed correctly

javac -version

Set the JAVA environment variables

sudo apt install oracle-java8-set-default

JCC

JCC is a C++ code generator that produces a C++ object interface wrapping a Java library. Build JCC module. JCC requires JDK and C++ compiler.

First download the source file, and build it using python setuptools.

$ cd ~/Downloads
$ svn co http://svn.apache.org/repos/asf/lucene/pylucene/trunk/jcc jcc
$ cd jcc
$ python setup.py build
$ sudo python setup.py install

ANT

ANT is a Java library and command-line tool that help building software. Download a binary or source file from here.

$ wget http://mirror.olnevhost.net/pub/apache//ant/binaries/apache-ant-1.10.5-bin.zip
$ unzip apache-ant-1.10.5-bin.zip
$ sudo mv apache-ant-1.10.5-bin.zip /usr/local/share/

Add the bin path to the shell environment.

export PATH="/usr/local/share/apache-ant-1.10.5/bin:${PATH}"

PyLucene

Download the latest PyLucene and compile it.

$ cd ~/Downloads
$ wget http://apache.claz.org/lucene/pylucene/pylucene-7.6.0-src.tar.gz
$ tar zxvf pylucene-7.6.0-src.tar.gz
$ cd pylucene-7.6.0

Change the Makefile appropriately. I modified the settings as below:

PREFIX_PYTHON=/usr/local
ANT=/usr/local/apache-ant-1.10.5/bin/ant
PYTHON=$(PREFIX_PYTHON)/bin/python3
JCC=$(PYTHON) -m jcc --arch x86_64
NUM_FILES=8

Run make

$ make
... It will take a while compiling files
$ make test
$ sudo make install

Check if it runs without a problem.

$ python3 -c "import lucene; lucene.initVM()"
@gitclem
Copy link

gitclem commented Feb 9, 2021 •
edited

I'm having trouble building this. I suspect it's a problem that involved Python and the Mac and some other thing.

First, is it possible to download this as a binary? That would eliminate my problem (I hope)

Otherwise, sorry for the longish post that follows... any help would be appreciated.

I've encountered this problem while trying to do a build of another project, so I don't think it's pylucene's problem per se. I have tried to search through the source code but I can't find where arm64 is actually used.

My configuration:

  • Macbook Pro, Intel Core i7 (so it's bad that arm64 is passed to the compiler somehow)
  • OS 10.15.7
  • Xcode 12.4
 java -version
openjdk version "1.8.0_282"
OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_282-b08)
OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.282-b08, mixed mode)

With Python 3.8.2 when I build, I get an error msg:

python setup.py build
...
/opt/local/bin/gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -iwithsysroot/System/Library/Frameworks/System.framework/PrivateHeaders -iwithsysroot/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/Headers -arch arm64 -arch x86_64 -I/usr/local/opt/libomp/include -Xpreprocessor -fopenmp -dynamiclib -D_jcc_lib -DJCC_VER="3.8" -I/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/include -I/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/include/darwin -I_jcc3 -Ijcc3/sources -I/Users/cwang/3.7/include -I/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/include/python3.8 -c jcc3/sources/jcc.cpp -o build/temp.macosx-10.14.6-x86_64-3.8/jcc3/sources/jcc.o -DPYTHON -fno-strict-aliasing -Wno-write-strings -mmacosx-version-min=10.9 -std=c++11 -stdlib=libc++
gcc: error: this compiler does not support arm64
gcc: error: unrecognized command-line option '-iwithsysroot/System/Library/Frameworks/System.framework/PrivateHeaders'
gcc: error: unrecognized command-line option '-iwithsysroot/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/Headers'
gcc: error: unrecognized command-line option '-stdlib=libc++'

With python 2.7.16

python setup.py build
...
/opt/local/bin/gcc -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes -I/usr/local/opt/libomp/include -Xpreprocessor -fopenmp -dynamiclib -D_jcc_lib -DJCC_VER="3.8" -I/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/include -I/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/include/darwin -I_jcc2 -Ijcc2/sources -I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c jcc2/sources/jcc.cpp -o build/temp.macosx-10.15-x86_64-2.7/jcc2/sources/jcc.o -DPYTHON -fno-strict-aliasing -Wno-write-strings -mmacosx-version-min=10.9 -std=c++11 -stdlib=libc++
gcc: error: unrecognized command-line option '-stdlib=libc++'

I found some indication that perhaps there is an imcompability that is arising due to clang vs gcc

I also tried:
export CC=/usr/bin/clang
export CXX=/usr/bin/clang++

but that resulted in a bunch of different compile errors when using Python 3.8.2 :

python setup.py build
...
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types.h:55:9: error: unknown type name '__int64_t'
typedef __int64_t       __darwin_blkcnt_t;      /* total blocks */
        ^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types.h:56:9: error: unknown type name '__int32_t'; did you mean '__int128_t'?
typedef __int32_t       __darwin_blksize_t;     /* preferred block size */
        ^
note: '__int128_t' declared here
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types.h:57:9: error: unknown type name '__int32_t'; did you mean '__int128_t'?
typedef __int32_t       __darwin_dev_t;         /* dev_t */
        ^
note: '__int128_t' declared here
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types.h:60:9: error: unknown type name '__uint32_t'; did you mean '__uint128_t'?
typedef __uint32_t      __darwin_gid_t;         /* [???] process and group IDs */
        ^
note: '__uint128_t' declared here
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types.h:61:9: error: unknown type name '__uint32_t'; did you mean '__uint128_t'?
typedef __uint32_t      __darwin_id_t;          /* [XSI] pid_t, uid_t, or gid_t*/
[ETC]

I also tried:
export CC=/usr/bin/clang
export CXX=/usr/bin/clang++

but that resulted in a bunch of different compile errors when using Python 2.7.16 :

python setup.py build
...
ld: warning: directory not found for option '-L/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/lib/server'
ld: library not found for -ljava
clang: error: linker command failed with exit code 1 (use -v to see invocation)

The directory path exists: /Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/lib/
but server is missing (directory or file?)

@gpdfgpdf
Copy link

Hi, same struggle here. How did you fix it?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment