Skip to content

Instantly share code, notes, and snippets.

@ChadJPetersen
Last active October 24, 2023 21:39
Show Gist options
  • Save ChadJPetersen/e5215da9c00b20fabff05b4a0e43e385 to your computer and use it in GitHub Desktop.
Save ChadJPetersen/e5215da9c00b20fabff05b4a0e43e385 to your computer and use it in GitHub Desktop.
After hours of slaving I finally got Xapian working on windows with a ruby binding (I use it for Redmine). I know this is far from the right way to do it, and is rather hacky. But it works! Also I found little starting points online when trying to compile Xapian on windows. So hoping this helps someone else a bit!
#In Powershell.
# Install chocolatey https://chocolatey.org/install to be the software package manager. Right now the following line is the installer. But it would be good to go grab the new command from the site provided.
#Set-ExecutionPolicy Bypass -Scope Process -Force; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072; iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))
choco install git ghostscript xpdf-utils gnuwin 7zip cyg-get msys2 graphviz activeperl
choco install ruby --version=2.6.5.1
# This sets up ruby to be a dev kit
ridk install 1
cyg-get install catdoc unrtf unzip antiword xlsx2csv libvisio-tools
# !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
# SWITCH YOUR CONSOLE!!!!!!
# !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
# In MSYS2 console
pacman -Syu
# close and reopen if prompted
pacman -Su
# Close and reopen if prompted
# if getting errors on the below you can try the --force option.
pacman -Syuu
pacman -S base-devel --needed
pacman -S mingw-w64-x86_64-toolchain zlib file pcre mingw64/mingw-w64-x86_64-file mingw64/mingw-w64-x86_64-pcre mingw-w64-x86_64-tesseract-ocr mingw-w64-x86_64-tesseract-data-eng git doxygen tcl mingw-w64-x86_64-graphviz mingw-w64-x86_64-python-docutils python3-pip python3-pygments
pacman -R libtool
pacman -S mingw64/mingw-w64-x86_64-libtool mingw-w64-x86_64-cmake gcc mingw64/mingw-w64-x86_64-zlib msys/zlib-devel msys/libuuid-devel pcre-devel
pip3 install --upgrade pip
pip install sphinx docutils
pip3 install sphinx
# i686 versions of pacman packages are for a 32 bit build.
# pacman -S mingw-w64-i686-toolchain mingw64/mingw-w64-i686-libtool mingw32/mingw-w64-i686-file mingw32/mingw-w64-i686-pcre mingw-w64-i686-cmake mingw32/mingw-w64-i686-zlib
git clone https://github.com/xapian/xapian.git /C/Tools/Xapian/src
cd /C/Tools/Xapian/src
# !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
# SWITCH YOUR CONSOLE!!!!!!
# !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
# After installing things, switch from msys2 console to admin MINGW64
# !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
# Go through all xapian projects and change /dev/null to NUL in file types:
# *.cc, *.h, *.cpp, *.c
# !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
# BE SURE TO UPDATE PATH'S TO WHERE YOU WANT THEM!!!
# !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
./bootstrap
# make distclean # can be helpful if a previous build was here.
# You may also have to add the dot command to the path. This is part of graphviz and can be installed via chocolaty in powershell.
PATH=$PATH:/c/ProgramData/chocolatey/bin
# After running the next line you may get some issues about unexpected fi token. Open up the config file that was currently being configured and remove the empty else statement.
# -L/C/Tools/msys64/mingw32/lib
./configure --prefix=/C/Tools/Xapian --with-ruby RUBY="/C/Tools/ruby26/bin/ruby.exe" RDOC="/C/Tools/Ruby26/bin/rdoc.cmd" RUBY_INC="/C/Tools/Ruby26/include/ruby-2.6.0" RUBY_INC_ARCH="/C/Tools/Ruby26/include/ruby-2.6.0/x64-mingw32" CPPFLAGS="-D_FORTIFY_SOURCE=0"
# --enable-shared --disable-static
# LDFLAGS="-Wl,--no-undefined -Wl,-ldxerr9 " CFLAGS=-fPIC CXXFLAGS=-fPIC
# CC=x86_64-w64-mingw32-gcc CC_FOR_BUILD=x86_64-w64-mingw32-gcc CPP="x86_64-w64-mingw32-gcc -E" CXX=x86_64-w64-mingw32-g++ ac_ct_CC=x86_64-w64-mingw32-gcc ac_ct_CXX=x86_64-w64-mingw32-g++ CXXCPP="x86_64-w64-mingw32-g++ -E"
# You may need to add a file for mingw for the time strip function.
# you can do this by adding the line
# #define _XOPEN_SOURCE 700
# before the includes in scriptindex.cc
# You may also need to delete whitespace before some multi line commands if you get errors like 'recipe commences before first target. Stop.'
# However I would suggest running make first before just changing things willy nilly.
# to supress unused parameter errors use (void)(<unused parameter name>);
make dist
# This you may need to run as admin
make install
cd /C/Tools/Xapian
mkdir ./xapian-ruby
mkdir ./xapian-ruby/ext
mkdir ./xapian-ruby/lib
mkdir ./xapian-ruby/ext/lib
mkdir ./xapian-ruby/ext/_xapian
mkdir ./xapian-ruby/ext/_xapian/lib
mkdir ./xapian-ruby/ext/_xapian/xapian
echo "require 'rake/extensiontask'
spec = Gem::Specification.load('xapian.gemspec')
Rake::ExtensionTask.new do |ext|
ext.name = '_xapian' # indicate the name of the extension.
ext.ext_dir = 'ext/_xapian' # search for 'hello_world' inside it.
# ext.lib_dir = 'lib/my_lib' # put binaries into this folder.
# ext.config_script = 'extconf.rb' # use instead of the default 'extconf.rb'.
ext.tmp_dir = 'tmp' # temporary folder used during compilation.
ext.source_pattern = \"*.{c,cpp,cc}\" # monitor file changes to allow simple rebuild.
# ext.config_options << 'CPPFLAGS=\"-I/C/Tools/Xapian/include\"' # supply additional options to configure script.
ext.gem_spec = spec # optionally indicate which gem specification will be used.
end" > ./xapian-ruby/Rakefile
echo 'Gem::Specification.new do |s|
s.name = "_xapian"
s.version = "1.4.13"
s.summary = "Xapian for Ruby"
s.author = "Edited by Chad Petersen"
s.files = Dir.glob("ext/**/*.{ c,rb}") +
Dir.glob("lib/**/*.rb")
s.extensions << "ext/_xapian/extconf.rb"
s.add_development_dependency "rake-compiler"
end' > ./xapian-ruby/xapian.gemspec
# you will need to update your version number
cp ./lib/libxapian.a ./xapian-ruby/ext/lib/libxapian.a
cp ./lib/libxapian.a ./xapian-ruby/ext/_xapian/lib/libxapian.a
echo "require \"mkmf\"
LIBDIR = RbConfig::CONFIG['libdir']
INCLUDEDIR = RbConfig::CONFIG['includedir']
HEADER_DIRS = [INCLUDEDIR]
# setup constant that is equal to that of the file path that holds that static libraries that will need to be compiled against
LIB_DIRS = [LIBDIR, File.expand_path(File.join(File.dirname(__FILE__), \"lib\"))]
# array of all libraries that the C extension should be compiled against
libs = ['-lxapian','-lz','-lrpcrt4','-lws2_32','-luuid','-lstdc++'] #if xapian changes required libraries this line will have to change. Double check dependency_libs in C:\Tools\xapian-bindings-1.4.7\ruby\.libs\_xapian.la that the needed libraries are on this line.
dir_config('_xapian', HEADER_DIRS, LIB_DIRS)
# iterate though the libs array, and append them to the \$LOCAL_LIBS array used for the makefile creation
libs.each do |lib|
\$LOCAL_LIBS << \"#{lib} \"
end
create_makefile('_xapian/_xapian')
" > ./xapian-ruby/ext/_xapian/extconf.rb
#more version updating
cp ./src/xapian-bindings/ruby/xapian_wrap.cc ./xapian-ruby/ext/_xapian/_xapian.cc
cp ./src/xapian-bindings/ruby/xapian_wrap.h ./xapian-ruby/ext/_xapian/_xapian.h
sed -i -e 's/xapian_wrap/_xapian/g' ./xapian-ruby/ext/_xapian/_xapian.cc
#more version updating
cp ./lib/libxapian.a ./xapian-ruby/ext/lib/libxapian.a
cp -a ./include/. ./xapian-ruby/ext/_xapian/
cp ./src/xapian-bindings/ruby/xapian.rb ./xapian-ruby/lib/xapian.rb
# !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
# !!!!!!Change shell to powershell!!!!!!!!
# !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
cd C:/Tools/Xapian/xapian-ruby
gem install rake-compiler
rake compile
cp ./lib/_xapian.so C:/Tools/ruby26/lib/ruby/site_ruby/2.6.0/_xapian.so
cp ./lib/xapian.rb C:/Tools/ruby26/lib/ruby/site_ruby/2.6.0/xapian.rb
# !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
# !!!!!!Change shell to mingw64!!!!!!!!!!!
# !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
cd /C/Tools/ruby26/lib/ruby/site_ruby/2.6.0
echo "Make sure the following files exist in the ruby bin folder."
ldd *
@PolywickStudio
Copy link

Do you know how to compile the Windows DLL, or clues to do so?

@ChadJPetersen
Copy link
Author

ChadJPetersen commented Aug 26, 2020

I haven't gotten the DLL to build. I found a way to get the so file that the ruby gem wants and stopped there.
I've updated the script a bit to include install of some dependencies that have since come up.
If you do get the DLL building I'd love a link here to your script though!

@PolywickStudio
Copy link

PolywickStudio commented Aug 31, 2020

I found a way to get the DLL to build.
Its:

C:\tools\xapian-bindings\csharp>g++ -shared -o xapian.dll xapian_wrap.o C:\tools\xapian\lib\libxapian.a C:\tools\xapian\lib\libgetopt.a C:\msys64\mingw64\x86_64-w64-mingw32\lib\libwsock32.a C:\msys64\mingw64\x86_64-w64-mingw32\lib\libmincore.a C:\msys64\mingw64\lib\libz.a

@PolywickStudio
Copy link

PolywickStudio commented Aug 31, 2020

My bash script for Xapian-Core:
./configure --prefix=/C/Tools/Xapian --enable-shared --disable-static LDFLAGS="-Wl,--no-undefined -Wl,-ldxerr9 -L/C/Tools/msys64/mingw64/lib" CFLAGS=-fPIC CXXFLAGS=-fPIC

My bash script for Xapian-Language-Integration:

./configure --prefix=/C/Tools/Xapian --enable-shared --disable-static LDFLAGS="-Wl,--no-undefined -Wl,-ldxerr9 -L/C/Tools/msys64/mingw64/lib" CFLAGS=-fPIC CXXFLAGS=-fPIC XAPIAN_CONFIG=/c/tools/xapian/xapian-config --with-csharp CSC=/c/progra~2/micros~1/2019/Community/MsBuild/Current/bin/Roslyn/csc.exe GACUTIL=/c/progra~2/micros~1/windows/v10.0a/bin/NETFX4~1.8TO/gacutil.exe GACUTIL=/c/progra~2/micros~2/windows/v10.0a/bin/NETFX4~1.8TO/gacutil.exe SN=/c/progra~2/micros~2/windows/v10.0a/bin/NETFX4~1.8TO/sn.exe

I used Short-filenames to fix the space issue for directories.

@PolywickStudio
Copy link

I managed to get the integration working with Delphi from the C# Integration. I called the Xapian.DLL and called stdcall extern CSharp* functions

@alexsilva
Copy link

I was trying to compile xapian-core on msys2 and I always got "xapian configure: error: no 32 bit integer type found"
This part of the instructions solved my problem. But I'm not sure what the reason for the failure was.

# In MSYS2 console 
pacman -Syu
# close and reopen if prompted
pacman -Su
# Close and reopen if prompted
# if getting errors on the below you can try the --force option.
pacman -Syuu
pacman -S base-devel --needed
pacman -S mingw-w64-x86_64-toolchain zlib file pcre mingw64/mingw-w64-x86_64-file mingw64/mingw-w64-x86_64-pcre mingw-w64-x86_64-tesseract-ocr mingw-w64-x86_64-tesseract-data-eng git doxygen tcl mingw-w64-x86_64-graphviz mingw-w64-x86_64-python-docutils python3-pip python3-pygments
pacman -R libtool
pacman -S mingw64/mingw-w64-x86_64-libtool mingw-w64-x86_64-cmake gcc mingw64/mingw-w64-x86_64-zlib msys/zlib-devel msys/libuuid-devel pcre-devel

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment