dlamichhane/gist:5894774

## gistfile1.txt
Reference: http://rubyonwindowsguides.github.io/book/ch02-05.html

Installing Native Gems

Now we are fully equipped for Ruby development on Windows. Yet problem of installing native gems requires our attention and better explanation. DevKit’s smoke test passed without problem but does it mean we can install any gem written in C/C++? Unfortunately the answer is no.

Native gems might depend on other libraries which are hard, or sometimes, impossible to build on Windows. Example of such a gem is Nokogiri, HTML and XML parser for Ruby, built upon Libxml2 library. Libxml2 is XML parser and toolkit written in C. If you want to know how to deal, at least partially, with such a libraries you should continue reading this chapter. Otherwise you can freely skip to the section “Installing Ruby On Rails”.

Just for illustration try to install Nokogiri from sources. Installation will fail with quite a long message printed out in the Command Prompt. Leaving out irrelevant parts output looks similar to following:

c:\>gem install nokogiri --platform=ruby
Temporarily enhancing PATH to include DevKit...
Building native extensions.  This could take a while...
ERROR:  Error installing nokogiri:
        ERROR: Failed to build gem native extension.

C:/Ruby/192/bin/ruby.exe extconf.rb
checking for libxml/parser.h... no
-----
libxml2 is missing.  please visit http://nokogiri.org/tutorials/installing_nokogiri.html for help with
installing dependencies.
-----
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of
necessary libraries and/or headers.  Check the mkmf.log file for more
details.  You may need configuration options.
...
Our attempt to install Nokogiri did not succeed due to a missing header file from Libxml2 library. This will happen whenever you try to install native gems that do not have pre-built binaries for Windows system and libraries that these gems depend on are not available.

However this problem is not inherent to Windows and Ruby. In order to install C/C++ Ruby extensions on each system you must have build tools and all dependencies installed. This means if you do not have Libxml2 installed on the Linux or OSX you will face exactly the same problem. The fact is that most of such gems are developed primarily on Linux where developers, naturally, rely on existing libraries. On the other hand lot of these libraries are portable and can be used on Windows, but number of them are shipped in a binary format because building them on Windows is quite complicated.

But it is not all that bad as it looks on the first sight. The fact is that at a time of One-Click Ruby Installer which was built with Visual Studio it was very hard to build external libraries and gems. Moreover Visual Studio was commercial and not all of Ruby developers had it. RubyInstaller project has changed a lot in that field. MRI itself depends on some libraries like libYAML and LibFFI but both of them were built when we were installing Ruby from sources. Moving the whole Ruby build procedure to MSYS/MinGW tool chain was a big step which resulted in possibility to use external libraries built with same compiler tools. This lead to much bigger set of native gems that became available to Ruby developers on Windows.

We will walk through the complete procedure of installing one native gem altogether with its external dependency. The gem is Ruby interface for sqlite3 database. First we will try to install gem from sources as we did with Nokogiri.

C:\>gem install sqlite3 –platform=ruby
Temporarily enhancing PATH to include DevKit...
Building native extensions.  This could take a while...
ERROR:  Error installing sqlite3:
        ERROR: Failed to build gem native extension.

C:/Ruby/192/bin/ruby.exe extconf.rb
checking for sqlite3.h... no
sqlite3.h is missing. Try 'port install sqlite3 +universal'
or 'yum install sqlite3-devel' and check your shared library search path (the
location where your sqlite3 shared library is located).
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of
necessary libraries and/or headers.  Check the mkmf.log file for more
details.  You may need configuration options.

Provided configuration options:
        --with-opt-dir
        --without-opt-dir
        --with-opt-include
        --without-opt-include=${opt-dir}/include
        --with-opt-lib
        --without-opt-lib=${opt-dir}/lib
        --with-make-prog
        --without-make-prog
        --srcdir=.
        --curdir
        --ruby=C:/Ruby/192/bin/ruby
        --with-sqlite3-dir
        --without-sqlite3-dir
        --with-sqlite3-include
        --without-sqlite3-include=${sqlite3-dir}/include
        --with-sqlite3-lib
        --without-sqlite3-lib=${sqlite3-dir}/lib

Gem files will remain installed in C:/Ruby/192/lib/ruby/gems/1.9.1/gems/sqlite3-1.3.3 for inspection.
Results logged to C:/Ruby/192/lib/ruby/gems/1.9.1/gems/sqlite3-1.3.3/ext/sqlite3/gem_make.out
This time we are displaying complete output of gem install command because we will completely analyze it. We see that script extconf.rb is executed at the beginning of the installation procedure. This is simple program that checks for prerequisites needed for the extension to be built. If all goes well Makefile is created. But something went wrong – C header file sqlite3.h was not found, script has printed out information and exited. If you check content of extconf.rb you will easily find the line responsible for premature script exit.

asplode('sqlite3.h')  unless find_header  'sqlite3.h'
Function find_header, defined in Ruby’s mkmf module that creates Makefile, obviously returned false. Question is where it was looking for sqlite3.h file? The answer can be found in mkmf.log file which was created during this installation. Leaving out irrelevant parts here is the answer:

gcc -o conftest -IC:/Ruby/192/include/ruby-1.9.1/i386-mingw32 -IC:/Ruby/192/include/ruby-1.9.1/ruby/backward \
-IC:/Ruby/192/include/ruby-1.9.1 -I. -I/usr/local/include -I/opt/local/include \
-I/sw/local/include -I/usr/include
We can see several -I... sequences which tell GCC compiler where to search header files. First three directories are related to our Ruby (1.9.2), the fourth is current directory, followed by paths where header files are usually kept on Linux systems. But we are on Windows! Why does mkmf module uses these paths? Well, just one more proof that Ruby was in earlier days mostly oriented to Linux. But there is another, more logical, reason although it still favors Linux. Short explanation is that Linux File System Structure (FSS) defines structure of directories and where different type of files should be stored. Even though it is not obligatory for anyone, most of Linux versions today try to follow it, sometimes with neglecting modifications. According to FSS, header files should be in the /usr/local/include, /opt/local/include or /usr/include directories.

Windows operating system doesn’t have such specification which makes it impossible to use some predefined paths for searching header files. Besides, Windows doesn’t come with any development tool thus no header file exists on the system after installation. Despite FSS even on Linux some header files might be found in unusual places and you can pass additional paths that will be searched.

After this short digression let’s get back to the output we got when we tried to install sqlite3 gem. Script implied that we might need to use some configuration option and gave us a list of available options. End of list is obviously related to SQlite3 and if you look carefully you will find the one that might help us with missing header file, --with-sqlite3-include. If we pass a path where sqlite3 folder is with this option, configuration script will search it too and this might solve our problem.

Download tarball containing amalgamation for SQlite 3.7.4 with configure script and unpack it to the c:\projects directory where we keep all sources. After extracting, our sqlite3.h file will be in the sqlite-autoconf-3070400 directory and full path to it is exactly what we have to pass to the configuration script. Go to the folder where Rubygems have left sqlite3 gem’s files, c:\Ruby\192\lib\ruby\gems\1.9.1\gems\sqlite3-1.3.3 and execute following statement:

ruby setup.rb config -- --with-sqlite3-include=c:\projects\sqlite-autoconf-3070400
Notice two things. We passed config command to the setup.rb script, followed by two dashes surrounded by spaces and configuration option for adding new directory that will be searched for header files. Be careful. Without -- above command will not work. And result is – failure again:

checking for sqlite3.h... yes
checking for sqlite3_libversion_number() in -lsqlite3... no
....
Configuration script has found header file but now it looks for function libversion_number() in -lsqlite3. Does this ring a bell? And if you substitute -l with lib? You are right! Configuration script tries to find function in the library. Passing libraries which GCC linker should use when it links executable or shared library is done by prefixing library name by -l. Linker will, in our case, search for a libsqlite3 library. If you look in the sqlite-autoconf-3070400 folder you will see that there is no file with that name there. That’s because we haven’t built SQlite3. We have just downloaded and extracted sources.

Let’s do it now. In order to that we must, in the current Command Prompt, add DevKit to the path, then start MSYS shell, configure SQlite3 and finally build it. SQlite3 README file suggests we should define SQLITE_ENABLE_COLUMN_METADATA when we build SQlite3. Let’s do all this now:

c:\projects\sqlite-autoconf-3070400>c:\Ruby\DevKit\devkitvars.bat
Adding the DevKit to PATH...
c:\projects\sqlite-autoconf-3070400>sh
sh-3.1$./configure CFLAGS="-DSQLITE_ENABLE_COLUMN_METADATA"
...
c:\projects\sqlite-autoconf-3070400>make
When make command finishes check directory where SQlite3 sources were extracted. You will see new sub-directory .libs with several files in it. The most important to us are libsqlite3.dll.a (link library used for linking against SQlite3) and libsqlite3-0.dll (shared library used in run-time). Finally we have everything for building sqlite3 gem. Go to the sqlite3 gem folder and run following statements:

ruby setup.rb config -- --with-sqlite3-include=c:\projects\sqlite-autoconf-3070400
–with-sqlite3-lib=C:\projects\sqlite-autoconf-3070400\.libs
ruby setup.rb setup
As a result you will get C extension used by this gem. Let’s check list of installed gems:

C:\>gem list

*** LOCAL GEMS ***

minitest (1.6.0)
pik (0.2.8)
rake (0.8.7)
rdiscount (1.6.8)
rdoc (2.5.8)
Our sqlite3 isn’t here! Well what we really did is that we built C extension but there are two more steps to be done so Rubygems are aware of sqlite3 gem existence. We must extract gem specification from the downloaded gem and put it in the specifications directory. Go to Rubygems cache folder where all downloaded gems are saved c:\Ruby\192\lib\ruby\gems\1.9.1\cache and issue following command:

gem spec sqlite3-1.3.3.gem --ruby > ..\specifications\sqlite3-1.3.3.gemspec
gem list

*** LOCAL GEMS ***

minitest (1.6.0)
pik (0.2.8)
rake (0.8.7)
rdiscount (1.6.8)
rdoc (2.5.8)
sqlite3 (1.3.3)
Finally Rubygems see our newly built sqlite3 gem. But before you check whether gem is working or not, you must copy sqlite3_native.so shared library from ext\sqlite3 folder to lib\sqlite3 folder. Now start irb and try to load gem:

C:\>irb
irb(main):001:0> require 'sqlite3'
Another error! Message pops up with the message that libsqlite3-0.dll is missing. Our shared library sqlite3_native.so is linked against SQlite’s shared library, which means that later one will be loaded in run-time. The way Windows search for them is cause of this message. When shared library is needed Windows first checks current application’s directory, then system directories (Windows and Windows\System32) and finally all directories listed in the path. SQlite’s shared library is not in any of them. We have to put it somewhere in the path so gem’s library can find it when needed. Copy it to the the C:\tools folder and try to load gem in irb:

irb(main):002:0> require 'sqlite3'
=> true
irb(main):003:0>
	Reference: http://rubyonwindowsguides.github.io/book/ch02-05.html

	Installing Native Gems

	Now we are fully equipped for Ruby development on Windows. Yet problem of installing native gems requires our attention and better explanation. DevKit’s smoke test passed without problem but does it mean we can install any gem written in C/C++? Unfortunately the answer is no.

	Native gems might depend on other libraries which are hard, or sometimes, impossible to build on Windows. Example of such a gem is Nokogiri, HTML and XML parser for Ruby, built upon Libxml2 library. Libxml2 is XML parser and toolkit written in C. If you want to know how to deal, at least partially, with such a libraries you should continue reading this chapter. Otherwise you can freely skip to the section “Installing Ruby On Rails”.

	Just for illustration try to install Nokogiri from sources. Installation will fail with quite a long message printed out in the Command Prompt. Leaving out irrelevant parts output looks similar to following:

	c:\>gem install nokogiri --platform=ruby
	Temporarily enhancing PATH to include DevKit...
	Building native extensions. This could take a while...
	ERROR: Error installing nokogiri:
	ERROR: Failed to build gem native extension.

	C:/Ruby/192/bin/ruby.exe extconf.rb
	checking for libxml/parser.h... no
	-----
	libxml2 is missing. please visit http://nokogiri.org/tutorials/installing_nokogiri.html for help with
	installing dependencies.
	-----
	* extconf.rb failed *
	Could not create Makefile due to some reason, probably lack of
	necessary libraries and/or headers. Check the mkmf.log file for more
	details. You may need configuration options.
	...
	Our attempt to install Nokogiri did not succeed due to a missing header file from Libxml2 library. This will happen whenever you try to install native gems that do not have pre-built binaries for Windows system and libraries that these gems depend on are not available.

	However this problem is not inherent to Windows and Ruby. In order to install C/C++ Ruby extensions on each system you must have build tools and all dependencies installed. This means if you do not have Libxml2 installed on the Linux or OSX you will face exactly the same problem. The fact is that most of such gems are developed primarily on Linux where developers, naturally, rely on existing libraries. On the other hand lot of these libraries are portable and can be used on Windows, but number of them are shipped in a binary format because building them on Windows is quite complicated.

	But it is not all that bad as it looks on the first sight. The fact is that at a time of One-Click Ruby Installer which was built with Visual Studio it was very hard to build external libraries and gems. Moreover Visual Studio was commercial and not all of Ruby developers had it. RubyInstaller project has changed a lot in that field. MRI itself depends on some libraries like libYAML and LibFFI but both of them were built when we were installing Ruby from sources. Moving the whole Ruby build procedure to MSYS/MinGW tool chain was a big step which resulted in possibility to use external libraries built with same compiler tools. This lead to much bigger set of native gems that became available to Ruby developers on Windows.

	We will walk through the complete procedure of installing one native gem altogether with its external dependency. The gem is Ruby interface for sqlite3 database. First we will try to install gem from sources as we did with Nokogiri.

	C:\>gem install sqlite3 –platform=ruby
	Temporarily enhancing PATH to include DevKit...
	Building native extensions. This could take a while...
	ERROR: Error installing sqlite3:
	ERROR: Failed to build gem native extension.

	C:/Ruby/192/bin/ruby.exe extconf.rb
	checking for sqlite3.h... no
	sqlite3.h is missing. Try 'port install sqlite3 +universal'
	or 'yum install sqlite3-devel' and check your shared library search path (the
	location where your sqlite3 shared library is located).
	* extconf.rb failed *
	Could not create Makefile due to some reason, probably lack of
	necessary libraries and/or headers. Check the mkmf.log file for more
	details. You may need configuration options.

	Provided configuration options:
	--with-opt-dir
	--without-opt-dir
	--with-opt-include
	--without-opt-include=${opt-dir}/include
	--with-opt-lib
	--without-opt-lib=${opt-dir}/lib
	--with-make-prog
	--without-make-prog
	--srcdir=.
	--curdir
	--ruby=C:/Ruby/192/bin/ruby
	--with-sqlite3-dir
	--without-sqlite3-dir
	--with-sqlite3-include
	--without-sqlite3-include=${sqlite3-dir}/include
	--with-sqlite3-lib
	--without-sqlite3-lib=${sqlite3-dir}/lib

	Gem files will remain installed in C:/Ruby/192/lib/ruby/gems/1.9.1/gems/sqlite3-1.3.3 for inspection.
	Results logged to C:/Ruby/192/lib/ruby/gems/1.9.1/gems/sqlite3-1.3.3/ext/sqlite3/gem_make.out
	This time we are displaying complete output of gem install command because we will completely analyze it. We see that script extconf.rb is executed at the beginning of the installation procedure. This is simple program that checks for prerequisites needed for the extension to be built. If all goes well Makefile is created. But something went wrong – C header file sqlite3.h was not found, script has printed out information and exited. If you check content of extconf.rb you will easily find the line responsible for premature script exit.

	asplode('sqlite3.h') unless find_header 'sqlite3.h'
	Function find_header, defined in Ruby’s mkmf module that creates Makefile, obviously returned false. Question is where it was looking for sqlite3.h file? The answer can be found in mkmf.log file which was created during this installation. Leaving out irrelevant parts here is the answer:

	gcc -o conftest -IC:/Ruby/192/include/ruby-1.9.1/i386-mingw32 -IC:/Ruby/192/include/ruby-1.9.1/ruby/backward \
	-IC:/Ruby/192/include/ruby-1.9.1 -I. -I/usr/local/include -I/opt/local/include \
	-I/sw/local/include -I/usr/include
	We can see several -I... sequences which tell GCC compiler where to search header files. First three directories are related to our Ruby (1.9.2), the fourth is current directory, followed by paths where header files are usually kept on Linux systems. But we are on Windows! Why does mkmf module uses these paths? Well, just one more proof that Ruby was in earlier days mostly oriented to Linux. But there is another, more logical, reason although it still favors Linux. Short explanation is that Linux File System Structure (FSS) defines structure of directories and where different type of files should be stored. Even though it is not obligatory for anyone, most of Linux versions today try to follow it, sometimes with neglecting modifications. According to FSS, header files should be in the /usr/local/include, /opt/local/include or /usr/include directories.

	Windows operating system doesn’t have such specification which makes it impossible to use some predefined paths for searching header files. Besides, Windows doesn’t come with any development tool thus no header file exists on the system after installation. Despite FSS even on Linux some header files might be found in unusual places and you can pass additional paths that will be searched.

	After this short digression let’s get back to the output we got when we tried to install sqlite3 gem. Script implied that we might need to use some configuration option and gave us a list of available options. End of list is obviously related to SQlite3 and if you look carefully you will find the one that might help us with missing header file, --with-sqlite3-include. If we pass a path where sqlite3 folder is with this option, configuration script will search it too and this might solve our problem.

	Download tarball containing amalgamation for SQlite 3.7.4 with configure script and unpack it to the c:\projects directory where we keep all sources. After extracting, our sqlite3.h file will be in the sqlite-autoconf-3070400 directory and full path to it is exactly what we have to pass to the configuration script. Go to the folder where Rubygems have left sqlite3 gem’s files, c:\Ruby\192\lib\ruby\gems\1.9.1\gems\sqlite3-1.3.3 and execute following statement:

	ruby setup.rb config -- --with-sqlite3-include=c:\projects\sqlite-autoconf-3070400
	Notice two things. We passed config command to the setup.rb script, followed by two dashes surrounded by spaces and configuration option for adding new directory that will be searched for header files. Be careful. Without -- above command will not work. And result is – failure again:

	checking for sqlite3.h... yes
	checking for sqlite3_libversion_number() in -lsqlite3... no
	....
	Configuration script has found header file but now it looks for function libversion_number() in -lsqlite3. Does this ring a bell? And if you substitute -l with lib? You are right! Configuration script tries to find function in the library. Passing libraries which GCC linker should use when it links executable or shared library is done by prefixing library name by -l. Linker will, in our case, search for a libsqlite3 library. If you look in the sqlite-autoconf-3070400 folder you will see that there is no file with that name there. That’s because we haven’t built SQlite3. We have just downloaded and extracted sources.

	Let’s do it now. In order to that we must, in the current Command Prompt, add DevKit to the path, then start MSYS shell, configure SQlite3 and finally build it. SQlite3 README file suggests we should define SQLITE_ENABLE_COLUMN_METADATA when we build SQlite3. Let’s do all this now:

	c:\projects\sqlite-autoconf-3070400>c:\Ruby\DevKit\devkitvars.bat
	Adding the DevKit to PATH...
	c:\projects\sqlite-autoconf-3070400>sh
	sh-3.1$./configure CFLAGS="-DSQLITE_ENABLE_COLUMN_METADATA"
	...
	c:\projects\sqlite-autoconf-3070400>make
	When make command finishes check directory where SQlite3 sources were extracted. You will see new sub-directory .libs with several files in it. The most important to us are libsqlite3.dll.a (link library used for linking against SQlite3) and libsqlite3-0.dll (shared library used in run-time). Finally we have everything for building sqlite3 gem. Go to the sqlite3 gem folder and run following statements:

	ruby setup.rb config -- --with-sqlite3-include=c:\projects\sqlite-autoconf-3070400
	–with-sqlite3-lib=C:\projects\sqlite-autoconf-3070400\.libs
	ruby setup.rb setup
	As a result you will get C extension used by this gem. Let’s check list of installed gems:

	C:\>gem list

	* LOCAL GEMS *

	minitest (1.6.0)
	pik (0.2.8)
	rake (0.8.7)
	rdiscount (1.6.8)
	rdoc (2.5.8)
	Our sqlite3 isn’t here! Well what we really did is that we built C extension but there are two more steps to be done so Rubygems are aware of sqlite3 gem existence. We must extract gem specification from the downloaded gem and put it in the specifications directory. Go to Rubygems cache folder where all downloaded gems are saved c:\Ruby\192\lib\ruby\gems\1.9.1\cache and issue following command:

	gem spec sqlite3-1.3.3.gem --ruby > ..\specifications\sqlite3-1.3.3.gemspec
	gem list

	* LOCAL GEMS *

	minitest (1.6.0)
	pik (0.2.8)
	rake (0.8.7)
	rdiscount (1.6.8)
	rdoc (2.5.8)
	sqlite3 (1.3.3)
	Finally Rubygems see our newly built sqlite3 gem. But before you check whether gem is working or not, you must copy sqlite3_native.so shared library from ext\sqlite3 folder to lib\sqlite3 folder. Now start irb and try to load gem:

	C:\>irb
	irb(main):001:0> require 'sqlite3'
	Another error! Message pops up with the message that libsqlite3-0.dll is missing. Our shared library sqlite3_native.so is linked against SQlite’s shared library, which means that later one will be loaded in run-time. The way Windows search for them is cause of this message. When shared library is needed Windows first checks current application’s directory, then system directories (Windows and Windows\System32) and finally all directories listed in the path. SQlite’s shared library is not in any of them. We have to put it somewhere in the path so gem’s library can find it when needed. Copy it to the the C:\tools folder and try to load gem in irb:

	irb(main):002:0> require 'sqlite3'
	=> true
	irb(main):003:0>