Skip to content

Instantly share code, notes, and snippets.

@dereckmezquita
Last active April 8, 2023 17:52
Show Gist options
  • Save dereckmezquita/ed860601138a46cf591a1bdcc95db0a2 to your computer and use it in GitHub Desktop.
Save dereckmezquita/ed860601138a46cf591a1bdcc95db0a2 to your computer and use it in GitHub Desktop.
Makevars and instructions for installing installing data.table on an M1 Mac; note config also works for installing slam/syuzhet.
# if you downloaded llvm manually above, replace with your chosen NEW_PATH/clang
LLVM_LOC = /opt/homebrew/opt/llvm
CC=$(LLVM_LOC)/bin/clang -fopenmp
CXX=$(LLVM_LOC)/bin/clang++ -fopenmp
# -O3 should be faster than -O2 (default) level optimisation ..
CFLAGS=-g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe
CXXFLAGS=-g -O3 -Wall -pedantic -std=c++11 -mtune=native -pipe
LDFLAGS=-L/opt/homebrew/opt/gettext/lib -L$(LLVM_LOC)/lib -Wl,-rpath,$(LLVM_LOC)/lib
CPPFLAGS=-I/opt/homebrew/opt/gettext/include -I$(LLVM_LOC)/include
@dereckmezquita
Copy link
Author

dereckmezquita commented Nov 17, 2021

Install multithreaded openMP data.table on an M1 Mac

Installing data.table on an M1 Mac with multithreading support can be frustrating. Here is the process I used to accomplish this.

If you've come across an error something like, cannot link file wrong architecture it is likely you've experienced the same erroneous set up I had:

  • ld: warning: ignoring file /Library/Frameworks/R.framework/R, building for macOS-arm64 but attempting to link with file built for macOS-x86_64

The solution, install an arm version of R (I had intel) the new version of RStudio which supports the arm version of R, and then setup your Makevars so everything can compile to arm. What was happening before is you had an intel version of R installed and wanted to compile for arm which is incompatible when linking.

Install arm R

Install R by using brew the package manager. The current version available is:

For more detailed information on installing data.table visit: data.table installation instructions

brew info R
Warning: Treating r as a formula. For the cask, use homebrew/cask/r
r: stable 4.1.2 (bottled)
Software environment for statistical computing
https://www.r-project.org/
/opt/homebrew/Cellar/r/4.1.2 (2,216 files, 64.6MB) *
  Poured from bottle on 2021-11-17 at 12:23:39
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/r.rb
License: GPL-2.0-or-later
==> Dependencies
Build: pkg-config ✔
Required: cairo ✔, gcc ✔, gettext ✔, jpeg ✔, libffi ✔, libpng ✔, openblas ✔, pcre2 ✔, readline ✔, tcl-tk ✔, xz ✔
==> Analytics
install: 15,733 (30 days), 35,010 (90 days), 137,251 (365 days)
install-on-request: 15,634 (30 days), 34,798 (90 days), 136,002 (365 days)
build-error: 25 (30 days)
brew install R

This will install in the following directory: /opt/homebrew/opt/r/

brew --prefix
/opt/homebrew

Note if you want to have a multi-user set up for your brew installations use this guide: github.com/dereckdemezquita/mac-setup

Install llvm

We need llvm to compile data.table:

brew install llvm

Make sure the path is exported in your .zshrc:

export PATH="/opt/homebrew/opt/llvm/bin:$PATH"

Makevars

Now that you have R installed (should be the Apple silicon version; brew installs this one for you). You can create the Makevars. This is simply a file with the name Makevars. This files holds the configurations necessary to compile packages on installation or with Rcpp. You don't have to have this configuration active at all times, just for compiling data.table.

The location of this file is: ~/.R/Makevars

Paste the content shown in the gist.

Install data.table

Install data.table through an external R console (terminal then type R). Note if R in the terminal window doesn't work this is because you haven't exported the path to R executable to your console.

install.packages("data.table", type = "source")
Installing package into ‘/opt/homebrew/lib/R/4.1/site-library’
(as ‘lib’ is unspecified)

trying URL 'https://cran.microsoft.com/src/contrib/data.table_1.14.2.tar.gz'
Content type 'application/octet-stream' length 5301817 bytes (5.1 MB)
==================================================
downloaded 5.1 MB

* installing *source* package ‘data.table’ ...
** package ‘data.table’ successfully unpacked and MD5 sums checked
** using staged installation
zlib 1.2.11 is available ok
R CMD SHLIB supports OpenMP without any extra hint
** libs
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c assign.c -o assign.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c between.c -o between.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c bmerge.c -o bmerge.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c chmatch.c -o chmatch.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c cj.c -o cj.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c coalesce.c -o coalesce.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c dogroups.c -o dogroups.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c fastmean.c -o fastmean.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c fcast.c -o fcast.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c fifelse.c -o fifelse.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c fmelt.c -o fmelt.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c forder.c -o forder.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c frank.c -o frank.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c fread.c -o fread.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c freadR.c -o freadR.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c froll.c -o froll.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c frollR.c -o frollR.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c frolladaptive.c -o frolladaptive.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c fsort.c -o fsort.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c fwrite.c -o fwrite.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c fwriteR.c -o fwriteR.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c gsumm.c -o gsumm.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c ijoin.c -o ijoin.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c init.c -o init.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c inrange.c -o inrange.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c nafill.c -o nafill.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c nqrecreateindices.c -o nqrecreateindices.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c openmp-utils.c -o openmp-utils.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c quickselect.c -o quickselect.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c rbindlist.c -o rbindlist.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c reorder.c -o reorder.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c shift.c -o shift.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c snprintf.c -o snprintf.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c subset.c -o subset.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c transpose.c -o transpose.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c types.c -o types.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c uniqlist.c -o uniqlist.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c utils.c -o utils.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c vecseq.c -o vecseq.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c wrappers.c -o wrappers.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -single_module -multiply_defined suppress -L/opt/homebrew/Cellar/r/4.1.2/lib/R/lib -L/opt/homebrew/opt/gettext/lib -L/opt/homebrew/opt/llvm/lib -Wl,-rpath,/opt/homebrew/opt/llvm/lib -o data.table.so assign.o between.o bmerge.o chmatch.o cj.o coalesce.o dogroups.o fastmean.o fcast.o fifelse.o fmelt.o forder.o frank.o fread.o freadR.o froll.o frollR.o frolladaptive.o fsort.o fwrite.o fwriteR.o gsumm.o ijoin.o init.o inrange.o nafill.o nqrecreateindices.o openmp-utils.o quickselect.o rbindlist.o reorder.o shift.o snprintf.o subset.o transpose.o types.o uniqlist.o utils.o vecseq.o wrappers.o -lz -L/opt/homebrew/Cellar/r/4.1.2/lib/R/lib -lR -lintl -Wl,-framework -Wl,CoreFoundation
PKG_CFLAGS =
PKG_LIBS = -lz
if [ "data.table.so" != "datatable.so" ]; then mv data.table.so datatable.so; fi
if [ "" != "Windows_NT" ] && [ `uname -s` = 'Darwin' ]; then install_name_tool -id datatable.so datatable.so; fi
installing to /opt/homebrew/lib/R/4.1/site-library/00LOCK-data.table/00new/data.table/libs
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded from temporary location
** checking absolute paths in shared objects and dynamic libraries
** testing if installed package can be loaded from final location
** testing if installed package keeps a record of temporary installation path
* DONE (data.table)

The downloaded source packages are in
	‘/private/var/folders/hv/3571hdb159zggq6whkpdnq400000gq/T/RtmpdJ6Pfu/downloaded_packages’
library("data.table")
data.table 1.14.2 using 4 threads (see ?getDTthreads).  Latest news: r-datatable.com

@dereckmezquita
Copy link
Author

dereckmezquita commented Nov 17, 2021

Versions

R --version
R version 4.1.2 (2021-11-01) -- "Bird Hippie"
Copyright (C) 2021 The R Foundation for Statistical Computing
Platform: aarch64-apple-darwin20.6.0 (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under the terms of the
GNU General Public License versions 2 or 3.
For more information about these matters see
https://www.gnu.org/licenses/.
brew --version
Homebrew 3.3.4
Homebrew/homebrew-core (git revision 9ba60a6af62; last commit 2021-11-17)
Homebrew/homebrew-cask (git revision 8f52edfbb5; last commit 2021-11-17)

RStudio

RStudio
2021.09.1 Build 372
© 2009-2021 RStudio, PBC

Screenshot 2021-11-17 at 14 02 05

@dereckmezquita
Copy link
Author

RStudio troubleshooting

If you have trouble running RStudio, installed by one user launched from another user on the same machine. Your permissions and ownership might be broken. Try first to re-install RStudio. If this doesn't work the run:

sudo chown -R root RStudio.app
sudo chmod -R 0755 RStudio.app

@gregmacfarlane
Copy link

As a note, this worked for me after being failed by the data.table instructions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment