bagong/SonicPi-Winbuild.md

## SonicPi-Winbuild.md

      
    Raw
  

              SonicPi-Winbuild.md
            
          
    Building Sonic Pi on Windows

Sonic Pi does an in-source build and on Windows uses two frameworks to assemble its dependencies: ruby/gem and vcpkg. Additionally three embedded applications, SuperCollider server & sc3-plugins, Erlang and Osmid need to be added to the package. Erlang and SuperCollider can be downloaded, osmid needs to be built using cmake. The Sonic Pi executable, using Qt, qscintilla and qwt for the editor, is built using qmake and nmake. Some of the deployment is handled by windeployqt, proper deployment and installer creation is not finalized yet. The build description assumes a 64-bit build using Visual Studio 2017, everything is done on the command line though. As to versioning, simply use the latest of everything (VS2017, ruby 2.41-2, cmake 3.8.2, erlang 20, supercollider 3.8) and the C/C++ libraries as provided in the vcpkg precombiled download (qt5.8, aubio 0.46, qscintilla 2.10, qwt 6.13, boost 1.64)
Prior knowledge required: basic command of git, basic familiarity with Windows command line (open command line, traverse file system, execute commands, copy and move files, create and run batch-file), basic understanding of running build tools, a basic understanding of the function of search paths.
Step by Step

1. Acquire Downloadables (see Links at the end)


Download and install build tools: git, cmake, 7zip, Visual Studio 2017 (Visual C/C++ component and Windows SDK). You also need an editor like Notepad++ or Atom that supports unix line endings


Clone the Sounic-pi Source: git clone https://github.com/samaaron/sonic-pi.git
Take a minute to get accustomed to the unusual file tree. The build is done inside this tree, the embedded applications are installed or copied into the tree as well. The two main folders are sonic-pi\app\server\native\win (placement of embedded apps) and sonic-pi\app\gui\qt (build folder)


Download Ruby (2.42 latest), SuperCollider 3.8 64-bit & sc3-plugins, Erlang


Clone and build osmid as described in the accompanying Readme.


Download the prebuilt vcpkg libraries. Contained are: qt5.8, qscintilla, qwt, aubio (using libsndfile and ffmpeg), boost 1.64. For SuperCollider there are also fftw3 and portaudio, but they are not used in the Sonic pi build. Qt5.8 comes with a lot of additional libraries (glib, zlib, icu and many more)


2. Place the embedded applications in the source tree

The folder in which everything is rooted is:
sonic-pi/app/server/native/win

The folder needs to be created. Copy the apps so that this file/folder structure results:
sonic-pi/app/server/native/win/ruby (all of ruby native goes here, the bin/include/lib folders go here)
sonic-pi/app/server/native/win/erlang (the bin/lib folders go here)
sonic-pi/app/server/native/win/osmid (place the two execs o2m and m2o here)
sonic-pi/app/server/native/win/scsynth.exe (the libs required by sc also go into the root folder)
sonic-pi/app/server/native/win/plugins (sc-core- and sc3-plugins)

From SuperCollder are required: scsynth.exe, fftw3f.dll, libsndfile.dll and the Windows runtime libraries (msvcp120.dll, msvcr120.dll, also contained in the SC application folder). All stock- and sc3-plugins go into the plugins folder.
If you build SuperCollider with VS2017 (undocumented, but possible), the list of files is: scsynth.exe, fftw3f.dll, libsndfile-1.dll, ogg.dll, flac.dll, vorbis.dll, vorbisenc.dll, concrt140.dll, msvcp140.dll.
3. Extract the VCPKG libraries

Chose a "safe" location, without potentially problematic elements like spaces and non-ascii characters in the path, e.g. C:\vcpkg. If you expand to the root folder of the drive (C:) and rename the top level foldername (vcpkg-export-2017....7z) to vcpkg the path statements given in the scripts below should work without adjustment.
If for some reason you want to build the vcpkg libraries yourself, note that building Qt5 takes quite some time (4-8 hours). Clone vcpkg from https://github.com/MicroSoft/vcpkg.git and follow the instructions to bootstrap the system (basically just run bootstrap-vcpkg.bat) Vcpkg controls build features like target architecture and library linkage (dynamic vs static) via so called triplets (little cmake files contained in vcpkg\triplets). By default vcpkg builds 32-bit dynamic libraries. The corresponding triplet is called x86-windows. As this build describes a 64-bit build, we need to append the triplet name x64-windows to each "install" command. For qt5 run:
vcpkg install qt5:x64-windows

The whole set of libraries required for SP would be:
vcpkg install qt5:x64-windows qwt:x64-windows aubio:x64-windows boost:x64-windows

Note that aubio will automatically pull in libsndfile and ffmpeg. Libsndfile will in turn pull in ogg, flac and vorbis.
Qscintilla was not included in above command, because it needs special treatment. The SP-build requires qscintilla to be provided as static build. You would expect that to be done by running vcpkg install qscintilla:x64-windows-static. But this will try to trigger a rebuild of all qscintilla dependencies (qt5 and its dependencies) as static libraries, which would not only take a long time, but would fail, because qt5 cannot be built as static libraries with vcpkg yet. The solution is to formulate an exception in the triplet-file used to define our build: vcpkg\triplets\x64-windows.cmake. Edit it and append the following lines:
if(PORT STREQUAL "qscintilla")
    set(VCPKG_LIBRARY_LINKAGE static)
endif()

After this modification qscintilla can be built the "normal" way:
vcpkg install qscintilla:x64-windows

This looks like the command to build a dynamic library, but because of the exception it will be built statically and nevertheless get installed to the same target folder as the other (dynamic) libs.
4. Prepare the environment (paths)

While this is not ideal, the Sonic Pi build depends on certain executables to be available via the search path. These are ruby, gem, qmake, lrelease, nmake, and the C/C++-toolchain executables. Also some libraries must either be found in the path or pointed to by an environment variable. Toolchain access is best handled by running a MS provided batch file that sets up all required paths (vcvarsallXXX.dll). The example below assumes using a 64-bit host (Win64) to build a 64-bit version of SP. There are other vcvarsall-batch files that support other combinations of build host and target architecture. For the SP-specific search locations we must prepend path statements to the search path generated by vcvarsall. This also makes sure that only and exactly the executables are found that should be used in the build, but not similar executables that may also be installed on your systen. This is best done by running a batch-file when opening the command-line. This batch file (for unixers: script) will call the above mentioned MS-batch-file that sets the VS-required paths, and then add the paths specific to the SP build. The example below assumes that vcpkg is extracted to C:\vcpkg, VS2017 is installed the default way, and the Sonic Pi source resides in C:\Projects\sp\sonic-pi. The batch-file also sets an environment-variable AUBIO_LIB that points to the dynamic library aubio.dll in C:\vcpkg\installed\x64-windows\bin. Adding the ruby binary folder to the path should make sure that this particular ruby install is used rather than any other that might be available (note though that file-associations can override search path order, so make sure to prepend the execution of each .rb-file by the command ruby. Thereby the system will search for ruby following the search-path order, and not look up a file assiciation):
echo off
set VCPKG_DIR=C:\vcpkg
set PROJECT_ROOT=C:\Projects\sp
echo Running vcvars64.bat to complete environment setup! Building 64-bit executable on 64-bit host.
call "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Auxiliary\Build\vcvars64.bat"
echo Using VS2017 64-bit compiler and default WinSDK.
echo Using libraries (including Qt) from vcpkg
echo Setting up search path...
set PATH=%VCPKG_DIR%\installed\x64-windows\bin;%VCPKG_DIR%\installed\x64-windows\tools\qt5;%PROJECT_ROOT%\sonic-pi\app\server\native\win\ruby\bin;%PATH%
set CMAKE_PREFIX_PATH=%VCPKG_DIR%\installed\x64-windows
set AUBIO_LIB=%VCPKG_DIR%\installed\x64-windows\bin\aubio.dll
echo Done!
cd /D %PROJECTS_DIR%\sp\sonic-pi\app\gui\qt

5. Install the Ruby gems

Note: if you have multiple rubies installed on your system, you have to make sure that the correct ruby-install is used in the following steps. If you only have one ruby install (and no other msys2 install), and the search path points to the bin folder of that ruby install, you can ignore this, you can run the command below from anywhere. If you have multiple installs, and especially if the file type .rb is registered to a different install from the one used in Sonic Pi, you have to take precautions. The ruby commands used are ruby, gem (for installing additional ruby libraries 'gems') and optionally ridk (for installing the development kit). All three commands call executables contained in the bin-folder of the ruby install:
<your prefix\>sonic-pi\app\server\native\win\ruby\bin

The safest is to prepend the commands with the absolute path:
<your prefix\>sonic-pi\app\server\native\win\ruby\bin\ruby
<your prefix\>sonic-pi\app\server\native\win\ruby\bin\gem
<your prefix\>sonic-pi\app\server\native\win\ruby\bin\ridk install

Another option is to use relative paths from your current location. If you are in the "build-folder" sonic-pi\app\gui\qt, then these path-expressions make sure that the correct executable is used:
..\..\server\native\win\ruby\bin\ruby
..\..\server\native\win\ruby\bin\gem
..\..\server\native\win\ruby\bin\ridk install


Install a fresh Ruby development-kit (msys2): if you installed ruby with the installer (making sure that it ends up in the correct "native\win\ruby" folder, you likely already installed the dk. If not, or if you just extracted the rubyinstaller archive, you need to run the script ridk install (remember the introductory note). Go through all three steps provided by the script one-by-one. (i.e. type 1 first and wait for the script to return, then 2 - return - 3 - done. It's quite a long process, a full C/C++ dev environment (msys2 and mingw-toolchain) is installed)


Install the gems listed below (see note below, copy paste command)
gem install win32-process windows-pr fast_osc rugged


Remove duplicate gems from the generic gems folder in sonic-pi\app\server\vendor

rugged
fast_osc
ffi
did_you_mean
minitest


Note: there used to be a longer list of gems to be installed. Further research is required if this is still necessary:
gem install win32-process windows-pr fast_osc ffi hamster wavefile rubame kramdown rugged multi_json ruby-beautify memoist
The minimum of gems that need to be removed from the vendor folder is:
- rugged
- fast_osc

You will get build errors if they are not removed. Note that in case of duplicate gems (in vendor and native) the generic one in the vendor folder will be used. The native gems are installed to
 sonic-pi\app\server\native\win\ruby\lib\ruby\gems\2.4.0\gems
6. Execute the Ruby part of the build


Change to build directory cd sonic-pi\app\gui\qt


Run i18n ruby ..\..\server\bin\i18n-tool.rb -t (remember the introductory remark. You might want to replace ruby by <your prefix\>sonic-pi\app\server\native\win\ruby\bin\ruby)


Copy ruby_help template to ruby_help.h copy ruby_help.tmpl ruby_help.h


Build docs ruby ..\..\server\bin\qt-doc.rb -o ruby_help.h
Check that folder info and files help_files.qrc and info_files.qrc have been created. You should not have gotten any warnings, otherwise something went wrong.


7. Execute the MSVC part of the build


Adjust win32-section in .\SonicPi.pro to current environment (not tested critically, adjust paths to your environment):

  # Windows only (adjusted to use with vcpkg)
  win32 {
    LIBS += -lqscintilla2 -lqwt
    QMAKE_CXXFLAGS += -IC:\vcpkg\installed\x64-windows\include -IC:\vcpkg\installed\x64-windows\include\qwt
    QMAKE_LFLAGS += /LIBPATH:C:\vcpkg\installed\x64-windows\lib
    DEFINES += _CRT_SECURE_NO_WARNINGS _WINSOCK_DEPRECATED_NO_WARNINGS
  }

Note: likely boost is not used properly in the build. Shared memory access seems not to be working

Copy folder sonic-pi\etc\doc\images\tutorial to .\images (the existing dummy file does not work as link)
Run (qt5 tools folder must be in path! (see above)):.

  lrelease SonicPi.pro
  qmake -o Makefile SonicPi.pro
  nmake
  nmake install
  cd release
  windeployqt .\sonic-pi.exe -printsupport      

Note: there is a batch-file win-build-app.bat which will automate the steps from running i18n to windeployment. It will run successfully if the tutorial folder was copied to the images folder, the file SinocPi.pro adjusted to this build, and the foldername windows replaced by win.
8. Prepare Sonic Pi runtime environment (no full install yet)


Continue using the search paths and AUBIO_LIB from the build environment (just continue using the batch file)
Configure erlang_bin_path in sonic-pi/app/server/sonicpi/lib/sonicpi/util.rb:329), e.g. (just to avoid an error, this does not work yet 😄):

  when :windows
    erlang_bin_path = File.join(native_path, "erlang", "bin", "erl.exe")
    "\"#{erlang_bin_path}\""

9. Run SonicPi in build target folder

cd sonic-pi\app\gui\qt\release
.\sonic-pi.exe

Note: at this stage you cannot run Sonic Pi from the file-explorer. The file explorer uses the default Windows search part which does not contain the path to the vcpkg bin-folder. SP has to be run from the command line. Making SP startable from the file-explorer requires a proper install that has not been defined yet.
Deployment/Install (not worked out)


copy qwt.dll to current folder (release)
copy Qt5OpenGL.dll and several dlls (zlib1, pcre, harfbuzz, and more (test)) to current folder
aubio.dll (and deps: libsndfile, ffmpeg, ogg, vorbis, flac - where does aubio.dll (and sibblings) go to work in the gem)
Windows runtime libraries
Qt platform plugins, dlls, plugins

Note: qscintilla2 is statically linked
Note: for now runtime access to libraries is handled via search path
Done!
Downloads

Installables

Build tools and resources


Visual Studio 2017
Windows SDK (corresponding, mind backwards compatibility, part of VS2017 install)
Gem & dev environment (MSYS2, atuomatically acquired by ruby 2.41)
VCPKG

Build helper


git
cmake
qmake, lrelease, windeployqt (comes with qt5)
namke (comes with VS)
7z

Embedded languages


Ruby
Erlang

Audio Tools


SuperCollider

Source links


sonic-pi: https://github.com/samaaron/sonic-pi.git
osmid: https://github.com/llloret/osmid.git

If built rather than downloaded:

supercollider: https://github.com/supercollider/supercollider.git
sc3-plugins: https://github.com/supercollider/sc3-plugins.git

Download links


Ruby: https://rubyinstaller.org/
Erlang: https://www.erlang.org/downloads
SuperCollider: http://supercollider.github.io/download
sc3-plugins: http://supercollider.github.io/download
vcpkg precompiled libs (temporarily, unofficial): https://github.com/bagong/dummy/releases
Visual Studio Community: https://www.visualstudio.com/downloads/
Cmake: https://cmake.org/download/
Git for Windows: https://git-scm.com/download/win

Forgettables


remove rugged-0.24.0 from sonic-pi/app/server/vendor
remove fast_osc from sonic-pi/app/server/vendor
do not remove "prerelease-aubio" from vendor folder, the more recent "native" version doesn't work yet
only when building supercollider: place ASIO-SDK next to portaudio folder
add special include for qwt in SonicPi.pro
don't forget libsndfile, flac, vorbis, vorbisenc, ogg ffmpeg when deploying aubio

Installer

?
TODO


FIX: WHY DOESN'T SOUND WORK YET? LIKELY SHARED MEMORY ACCESS DOESN'T WORK - boost - shared memory
Configure Erlang use properly
Follow up missing qtvirtualkeyboard plugin (required?)
Fix non-working enter in editor
Create installer

Considerations


use only native gems? (install all, or remove the ones from generic vendor which are installed natively?)
or use as many generic gems as possible and only install the ones required as native (windows specific, fast_osc, rugged), and more?
Erlang otp requires a ms runtime environment installed - that's a complication for the SP install...

Final request

Please give feedback if you've used this and got questions, problems, or solutions. I'll try to enhance this over time. Comment either below or enquire in the Gitter group sonic-pi.