Compiling libmagic and libmagic python bindings natively on OS X within your home directory

osx-libmagic-homedir.md

These instructions compile a native OS X version of libmagic and the libmagic python binding, which use the system magic database, that live in your home directory, and which cannot be moved around, but which should always be discoverable by applications.

Assumes OS X with Xcode or Xcode command-line tools, not macports or homebrew

To install Xcode command-line tools in recent versions of OS X, open Terminal, type xcode-select --install and click "Install"

Older versions of OS X may need to manually download disk images of older versions of Xcode and/or the Xcode command-line tools, as various certificates may have expired (e.g. 10.7.4 requires a manual install from the Xcode 4.6.2 command-line tools image)

Download Apple's latest modified file 5.04 (used in 10.7+) from http://opensource.apple.com/tarballs/file/file-47.tar.gz

Force libmagic to use the OS X system magic database:

$ cd ~/Downloads/
$ cd file-47/file/
$ mv configure configure.bak
$ sed "s/pkgdatadir='\$(datadir)\/misc'/pkgdatadir='\/usr\/share\/file'/" configure.bak > configure
$ chmod 755 configure

Compile file and libmagic as fat/universal binaries:

$ ./configure --prefix=$HOME LDFLAGS="-arch i386 -arch x86_64" CFLAGS="-arch i386 -arch x86_64" ARCHFLAGS="-arch i386 -arch x86_64"
$ make
$ make install
install: /usr/share/file/magic.mgc: Operation not permitted

It's okay that installing magic.mgc fails, because there's already the system one there.

Using an older version of OS X/Xcode and getting these warnings and errors after running make?

$ make
WARNING: `aclocal-1.10' is missing on your system.  You should only need it if
         you modified `acinclude.m4' or `configure.ac'.  You might want
         to install the `Automake' and `Perl' packages.  Grab them from
         any GNU archive site.
WARNING: `automake-1.10' is missing on your system.  You should only need it if
         you modified `Makefile.am', `acinclude.m4' or `configure.ac'.
         You might want to install the `Automake' and `Perl' packages.
         Grab them from any GNU archive site.
WARNING: `autoconf' is missing on your system.  You should only need it if
         you modified `configure.ac'.  You might want to install the
         `Autoconf' and `GNU m4' packages.  Grab them from any GNU
         archive site.
llvm-gcc-4.2: -E, -S, -save-temps and -M options are not allowed with multiple -arch flags

Just add --disable-dependency-tracking to your ./configure command like so:

$ make clean
$ ./configure --prefix=$HOME LDFLAGS="-arch i386 -arch x86_64" CFLAGS="-arch i386 -arch x86_64" ARCHFLAGS="-arch i386 -arch x86_64" --disable-dependency-tracking
$ make
$ make install

Edit the Python setup file to point to your newly compiled libmagic folders:

$ cd python/

Open setup.py in a plain text editor and change:

    library_dirs = ['./','../','../src','/usr/lib/'],
    include_dirs = ['./','../','../src','/usr/include/'],

to:

    library_dirs = ['./','../','../src','/usr/lib/','/Users/username/lib'],
    include_dirs = ['./','../','../src','/usr/include/','/Users/username/include'],

Compile the Python binding:

$ python setup.py build
$ python setup.py install --user

Now, your home directory has three new (probably) folders with file and libmagic in them:

$ cd ~
$ ls -al
drwxr-xr-x    3 username  1897370479    102 Aug 27 19:48 bin
drwxr-xr-x    3 username  1897370479    102 Aug 27 19:48 include
drwxr-xr-x    7 username  1897370479    238 Aug 27 19:48 lib

The ~/lib folder is in the default search path for dynamic libraries, so any software that uses libmagic should be able to automatically find it. This should always work without modification, even on OS X systems with System Integrity Protection (10.11 El Capitan and later), as long as they stay in ~/lib.

Your home directory's Python Library folder also has the magic binding:

$ cd ~/Library/Python/2.7/lib/python/site-packages/
$ ls -al
-rwxr-xr-x    1 milianov  1897370479  33360 Aug 27 19:24 magic.so

This is also in the default search path for Python modules, and should be available to any Python scripts that require it. (Note that some older versions of OS X had trouble with user directory modules, requiring a symlink to an alternate path, e.g. http://python.6.x6.nabble.com/Python-2-7-Lion-and-user-site-packages-folder-problem-td2154632.html.)