est31/msvc-wine-rust.md Secret

## msvc-wine-rust.md

      
    Raw
  

              msvc-wine-rust.md
            
          
    Obtaining and using Microsoft's link.exe on Linux+Wine with Rust

I've got the initial idea for this project after watching a talk about someone in C++ a conference talk showing off the MSVC cpp compiler running on Wine on Linux (sorry, no link, as I can't find a link to the talk anymore, I guess it's buried now in the depths of youtube...). After seeing that talk I've always wondered, can this be done for Rust as well?
All the guides on the topic start off with something like "Step one: install Microsoft Visual Studio or the build tools on a Windows machine".
This is not good, because it means you need to go through all these arbitrary installer steps, only to obtain some files. Let's see whether this can be improved!
This post describes how I have figured out a way to obtain Microsoft's link.exe, the SDK and use both on Linux with Wine,  without requiring any execution of tools on Windows.
If you want to give it a try yourself, check out the script I've created for this purpose.
Why?

Why get the Microsoft provided linker, instead of the gcc provided one? You can cross compile to Windows with the right gcc toolchain just as well!
Generally the Microsoft linker is seen to be superior to the gcc one. I personally also liked the challenge of the task.
Obtaining the MSVC linker

The 2015 build tools are very easy to obtain. You can just download them directly.
They are being shipped in an even nicer format in the WDK
where you don't have to extract msi files.
However, we want to run the latest and coolest stuff. Who doesn't want to?
The official page for the C++ build tools says:

Looking for the Visual C++ Build Tools for VS 2017? They're integrated into the new, lightweight VS 2017 installer. Please see this post on the VC Blog for more information.

Web installers are stupid, especially if they are single platform and may not run on Wine. Let's do it without running that thing!
The Websites that Microsoft provides here are not really welcoming for those use cases. Even if you don't have internet access, the only option you seem to have is to download the files in advance, while specifying some ids, and then executing the installer offline. You are not given any URLs you can just fetch, it's all hidden inside some tool. As an icing on the cake, apparently the tool asks you for administrator privileges right at the start, even if you just want to download the files.
On my Windows VM I've downloaded the tool, started it, and then cancelled without confirming the installation of anything. And it still has "installed" a "Visual Studio 2017 installer" for me. That's really bad treatment of users!
Well, let's find out whether we can obtain the files without having to run that tool!
One approach is to execute the tool inside a VM and to monitor for HTTP requests, but I'm not familiar with the workflows here, e.g. how it works regarding HTTPS. What I did instead was to google for various URLs connected to the downloader, e.g. that get spit out because someone tried to run the tool offline and they so helpfully put that error log onto the internet, or e.g. part of some other tutorial.
This process led me to this URL (e.g. mentioned here and here):
https://aka.ms/vs/15/release/channel

If you know how rustup works, it has a channel.toml file, which for a given channel (nightly, beta, stable) contains all the info for the various rustc toolchains. Maybe this is similar to that?
Running curl -I https://aka.ms/vs/15/release/channel tells you that the URL is a redirect (expectable, as aka.ms is Microsoft's URL shortener service) to a file called visualstudio.15.release.chman with the long URL:
https://download.visualstudio.microsoft.com/download/pr/100158827/d1bbd8466ed033571fad6ec9d7e9e3c9/visualstudio.15.release.chman

Running curl -I on that URL gives you the content type application/octet-stream, but if you download the file you discover that it's in fact in the json format! Let's add a .json extension to it and look at the json using the built in viewer of Firefox:

Apparently the json file contains a list of product descriptions in various languages, with links to licenses, some version information, a reference to a "manifest" with an URL, and a signature (funnily the signature still appears to use sha1 but Microsoft is not alone with that so let's not judge them...).
Let's look at the manifest visualsutio.vsman! It's located at:
https://download.visualstudio.microsoft.com/download/pr/100158818/6996ff96818a4e1632a71a057b34a1ed/visualstudio.vsman

and it appears to be another json file, with 4 top level entries: info, packages, deprecate, and signature.
The packages entry contains 4857 sub-entries, all of them have an id field, while some have a dependencies field and others have a payload field with urls inside. Apparently this is a package dependency graph!
For the id's in the package table: we already had encountered id's previously! The page from above! Apparently that list is a rendered version of the json file, with the critical information, about where to download the binaries, omitted.
We can use those pages to obtain the ID of the component we are interested in: Microsoft.VisualStudio.Workload.VCTools for the "Visual C++ build tools".
The built in json viewer for Firefox is quite nice (I was positively surprised the first time I found out about it), but it meets its limits with files of this scale. Let's switch to better json tooling. Since I've discovered it, I'm a fan of jq. It allows you to view json files in a true UNIX like fashion, usable from your favourite shell.
Let's pass a request for the component we are interested in to jq and see whether we get any output:
$jq '.packages | .[] | select(.id=="Microsoft.VisualStudio.Workload.VCTools") ' visualstudio.vsman.json

We are getting output! It indeed gives us an entry for the "Visual C++ build tools". It contains descriptions in various languages, the english one is "Build classic Windows-based applications using the power of the Visual C++ toolset, ATL, and optional features like MFC and C++/CLI.". This is precisely what we want.
However, not a single URL. Probably the URL's are contained in the dependencies of the package. What comes now is a bit of digging, basically trial and error, downloading various components, checking whether they contain anything useful (the linker), etc. Microsoft.VisualStudio.Workload.VCTools leads us to Microsoft.VisualStudio.Component.VC.Tools.x86.x64 which leads us to Microsoft.VisualStudio.PackageGroup.VC.Tools.x86, which has this very nice entry:
{
  "id": "Microsoft.VisualStudio.PackageGroup.VC.Tools.x86",
  "version": "15.0.26823.1",
  "type": "Group",
  "dependencies": {
    "Microsoft.VisualCpp.CRT.Headers": "[14.0,16.0)",
    "Microsoft.VisualCpp.Redist.14": "[14.0,16.0)",
    "Microsoft.VisualCpp.RuntimeDebug.14": "[14.0,16.0)",
    "Microsoft.VisualCpp.CRT.Redist.x64": "[14.0,16.0)",
    "Microsoft.VisualCpp.CRT.Redist.x86": "[14.0,16.0)",
    "Microsoft.VisualCpp.CRT.source": "[14.0,16.0)",
    "Microsoft.VisualCpp.CRT.x64.Desktop": "[14.0,16.0)",
    "Microsoft.VisualCpp.CRT.x86.Desktop": "[14.0,16.0)",
    "Microsoft.VisualCpp.DIA.SDK": "[14.0,16.0)",
    "Microsoft.VisualCpp.Tools.Core.x86": "[14.0,16.0)",
    "Microsoft.VisualCpp.Tools.Core.Resources": "[14.0,16.0)",
    "Microsoft.VisualCpp.Tools.HostX86.TargetX86": "[14.0,16.0)",
    "Microsoft.VisualCpp.Tools.HostX86.TargetX86.Resources": "[14.0,16.0)",
    "Microsoft.VisualCpp.Tools.HostX86.TargetX64": "[14.0,16.0)",
    "Microsoft.VisualCpp.Tools.HostX86.TargetX64.Resources": "[14.0,16.0)",
    "Microsoft.VisualCpp.VCTip.HostX86.TargetX86": "[14.0,16.0)",
    "Microsoft.VisualCpp.VCTip.HostX86.TargetX64": "[14.0,16.0)"
  }
}
This gives us IDs like Microsoft.VisualCpp.Tools.HostX86.TargetX86 under which we then find URLs:
{
  "id": "Microsoft.VisualCpp.Tools.HostX86.TargetX86",
  "version": "14.10.25547.0",
  "type": "Vsix",
  "payloads": [
    {
      "fileName": "microsoft.visualcpp.tools.hostx86.targetx86.vsix",
      "sha256": "a40b52467d3359414947e5294b6385f6af95b49c2a765e0f7f4749ac826f3a93",
      "size": 11943108,
      "url": "https://download.visualstudio.microsoft.com/download/pr/11436915/aff3326c9d694e3f92617dcb3827e9f7/microsoft.visualcpp.tools.hostx86.targetx86.vsix"
    }
  ],
  "installSizes": {
    "targetDrive": 26047906
  },
  "dependencies": {
    "Microsoft.VisualCpp.Tools.HostX86.TargetX86.Resources": "14.10.25547.0"
  }
}

If you download those vsix files, you'll find out that they are basic .zip files that contain the link.exe that we wanted, under a path like Contents/VC/Tools/MSVC/14.11.25503/bin/Hostx64/x64/.
This is really awesome!! We downloaded the files, without having to run that weird Windows-only tool.
First try of integrating the linker

Let's now extract the microsoft.visualcpp.tools.hostx86.targetx86.vsix file and try out the linker with wine:
$ sudo apt install wine64-development
$ wine64 link.exe
[stuff omitted]
wine: Call from 0x7b44ed07 to unimplemented function api-ms-win-crt-conio-l1-1-0.dll.__conio_common_vcwprintf, aborting
err:seh:setup_exception stack overflow 544 bytes in thread 0009 eip 000000007bc94c26 esp 00000000001413e0 stack 0x140000-0x141000-0x240000

Noo! We ran into an unimplemented function error. And the name component vcwprintf makes it appear that it's some printing function. Not very good if that's broken in a terminal application!
After a bit of digging, I've found out that wine has implemented this function since, but the implementation is only available in Wine 2.1 and later. I am running Wine 2.0 (the one that ships with my distro Ubuntu 17.04 zesty). So I've ended up checking out Wine 2.19 using git and building it myself.
After that, we can produce the wanted output:
$ WINEDEBUG=fixme-all ~/path/to/wine64 link.exe /VERBOSE
Microsoft (R) Incremental Linker Version 14.11.25547.0
Copyright (C) Microsoft Corporation.  All rights reserved.


Starting pass 1
LINK : warning LNK4001: no object files specified; libraries used
LINK : warning LNK4068: /MACHINE not specified; defaulting to X64

Searching libraries

Finished searching libraries
LINK : fatal error LNK1561: entry point must be defined

Wine can apparently run the linker to output this message, very nice.
Now let's try to integrate it into some Rust project. Let's create a simple wrapper called linker.sh with the executable bit set (chmod +x linker.sh) and the following content:
#!/bin/bash
~/path/to/wine64 ~/path/to/link.exe $@
And let's add this to ~/.cargo/config:
[target.x86_64-pc-windows-msvc]
linker = "/path/to/linker.sh"
And finally, let's add the target:
rustup target add x86_64-pc-windows-msvc

Now let's try this with the glium examples:
$ git clone https://github.com/glium/glium
$ cd glium
$ cargo build --example triangle --target x86_64-pc-windows-msvc

This gives us a linker error, which is very verbose. Basically it contains errors of two kinds. Errors of the first kind look like:
LINK : warning LNK4044: unrecognized option '/path/to/glium/target/x86_64-pc-windows-msvc/debug/examples/triangle-b2eaa7d6b32b8280.triangle1.rust-cgu.o'; ignored
LINK : warning LNK4044: unrecognized option '/path/to/glium/target/x86_64-pc-windows-msvc/debug/deps/libglutin-b2d77681dfdf681d.rlib'; ignored

And the second kind looks like:
LINK : fatal error LNK1181: cannot open input file 'opengl32.lib'

The good news is that the linker actually works in the sense that wine can run it until that error gets spit out, and that it gets invoked by rustc. That's nice!
Now is the time where a friendly bunny arrives and gives you an explanation for the causes of the errors:

The missing lib file errors are created because the Windows SDK is not installed.
The unrecognized option errors are obtained because of CLI parameter semantics on Windows: a leading slash here is the equivalent of - on UNIX, basically starting a command line option.

To fix the first issue, we apparently need to obtain the Windows SDK somehow. Same rules apply as above, we don't really want to run any GUI installers, but download and extract the file. For the second issue, Wine ships with a tool called winepath that allows conversion between paths. We only need to apply it in our wrapper.
It is not that trivial though, as some options to the linker are intended to start with a forward slash, because they are traditional linker flags. E.g. /NOLOGO is being passed to the linker.
The bunny suggests to add some some smart logic to dispatch between "linker flag" and "path passed to the linker".
The following new content for linker.sh should do the job:
#!/bin/bash
wine_exec=/path/to/wine_executable
link_exec=/path/to/link.exe

args=""
for v in $@; do
	num_of_slash=`tr -dc '/' <<< "$v" | wc -c`
	num_of_colon=`tr -dc ':' <<< "$v" | wc -c`
	if [ "$num_of_slash" -gt "1" ] && [ "$num_of_colon" -eq "0" ]; then
		v=`$wine_exec winepath -w $v`
	fi
	args="$args $v"
done

$wine_exec $link_exec $args
Obtaining the Windows SDK

The Windows 10 SDK offers downloads in the form of an ISO and a web installer. The ISO contains a large number of files and installers. Far more than we need! Let's see whether the visualsutio.vsman we have contains something about the SDK...
It turns out yes! The Microsoft.VisualStudio.Workload.VCTools key contains a dependency on Microsoft.VisualStudio.Component.Windows10SDK.16299.Desktop, which depends on Win10SDK_10.0.16299.Desktop. That key has a large list of payloads, including "Windows SDK Desktop Libs x64-x86_en-us.msi" and "Windows SDK Desktop Libs x86-x86_en-us.msi".
However, if you download them you will notice that they don't actually contain any of the required data. Instead, the data is provided through cab files, which are both right above the two msi files in the list of payloads. It turns out that this is a common thing on Windows: often the files that a .msi installer installs  are not stored inside the installer itself. There is good tooling available to obtain the filenames of the required .cab files for a given msi file, but back then I didn't know it and just guessed my way through: I've extracted the msi file using 7z, and found out that the !_StringData text file contains mentions of .cab files.
The URLs you can obtain this way are for x64:
https://download.visualstudio.microsoft.com/download/pr/100107594/d176ecb4240a304d1a2af2391e8965d4/Windows%20SDK%20Desktop%20Libs%20x64-x86_en-us.msi
https://download.visualstudio.microsoft.com/download/pr/100107594/d176ecb4240a304d1a2af2391e8965d4/58314d0646d7e1a25e97c902166c3155.cab

And for x86:
https://download.visualstudio.microsoft.com/download/pr/100107594/d176ecb4240a304d1a2af2391e8965d4/Windows%20SDK%20Desktop%20Libs%20x86-x86_en-us.msi
https://download.visualstudio.microsoft.com/download/pr/100107594/d176ecb4240a304d1a2af2391e8965d4/53174a8154da07099db041b9caffeaee.cab

If you look at the cab files with your normal linux compressed file explorer (ark or 7z l or whatever you have), you see a large amount of cryptic file names like file5e28f00de9be3e5109ef75d06fd2719, but no cleartext file names. If you extract them, the file content seems untouched, though. So you only need the mapping from the hashes to the file names. The mapping seems to be stored in the msi file, in the !_StringData text file as well. However, restoring the file names manually would be too hard, we need a tool for this.
A search gives us two tools: The first tool is lessmsi, and the second is msiextract, which was a bit harder to find.
Let's try out msiextract:
$ sudo apt install msitools
$ msiextract Windows\ SDK\ Desktop\ Libs\ x86-x86_en-us.msi
/path/to/pwd: Error opening file /path/to/pwd: Is a directory

Hmm, it turns out that msitools has a bug here.
Let's try lessmsi now. I've downloaded version 1.6.1, and were able to use the command line tool with wine in a fashion like:
wine lessmsi/lessmsi.exe x Windows\ SDK\ Desktop\ Libs\ x86-x86_en-us.msi

You need to have the corresponding cab file in the same directory, otherwise it will complain.
Note that it was quite some challenge for me to get it running on Linux. Simply invoking mono didn't work because it tries to access a msi.dll file. Invoking it via the wine version I've built manually didn't work either, it exits without any output. Still no idea what the reason is, I'm guessing a wine bug or maybe missing 32 bit support or something. Invoking it via the distro provided wine-development (Wine 2.0) from the Ubuntu 17.04 package sources complains about not being able to find .Net.
However, I did get it working on the distro provided wine-development after I've installed mono manually (downloading the msi from here and then doing wine msiexec /i wine-mono-4.7.1.msi).
After lessmsi has extracted the .lib files of the Windows SDK, we need to tell the linker about it.
In order for the linker to find the SDK files, the path must either be mentioned in the /LIBPATH flag passed to the linker, or the LIB environment variable.
Let's add the following to our script before the for loop:
sdk_libs_path=/path/to/sdk/libs
sdk_libs_path=`realpath "$sdk_libs_path"`
export LIB=`$wine_exec winepath -w "$sdk_libs_path"`
And now let's try it.
LINK : fatal error LNK1181: cannot open input file 'user32.lib'

Seems it didn't work! However, note that the name of the .lib file is different. We seem to have made some kind of difference. Looking into the SDK libs directory it seems that user32.lib is indeed not present.
Getting more of the Windows SDK

We need to find out where user32.lib comes from. We remember that the SDK ISO file contains all required installers. Let's download it and mount it via sudo mount -o ro /path/to/iso /some/mount/path.
Lessmsi has a mode where it can list the files contained inside a given msi file.
With a little bit of bash scripting, we can convert this into a tool that searches all the msi files inside the installer for our required user32.lib:
$ for file in Installers/*.msi; do wine /path/to/lessmsi.exe l -t File "$file" | cut -d, -f3 | sed "s#^#$file: #"; done | grep -i "user32.lib" | sort | uniq
Installers/Windows SDK for Windows Store Apps Libs-x86_en-us.msi: User32.Lib

Seems we found something! Happens the file is also present inside visualsutio.vsman and has a downloadable URL. We obtain the associated cab files by doing:
lessmsi.exe l -t Media Windows\ SDK\ for\ Windows\ Store\ Apps\ Libs-x86_en-us.msi | cut -d, -f4

Cabinet

05047a45609f311645eebcac2739fc4c.cab
0b2a4987421d95d0cb37640889aa9e9b.cab
13d68b8a7b6678a368e2d13ff4027521.cab
463ad1b0783ebda908fd6c16a4abfe93.cab
5a22e5cde814b041749fb271547f4dd5.cab
ba60f891debd633ae9c26e1372703e3c.cab
e10768bb6e9d0ea730280336b697da66.cab
f9b24c8280986c0683fbceca5326d806.cab

With the file names known we can obtain the corresponding URLs via the visualsutio.vsman file again.
If we extract the file and tell the linker about it, we will see that it's still not happy with us, apparently there are still missing .lib files.
We repeat the above process and obtain a couple of more installers needed: Universal CRT Headers Libraries and Sources-x86_en-us.msi and microsoft.visualcpp.crt.x86.store.vsix (this one being obtained by guesswork from the required lib name msvcrt.lib).
Interlude: hacking together a Windows version of dpkg -S

Anyone who wanted to find out where a file comes from on a debian based distro knows about dpkg -S. It's a very useful command. Quoting the manpage:
-S, --search filename-search-pattern...
    Search for a filename from installed packages.

It would be cool if something like that were available on Windows as well. We are lucky, ther is a way.
The way Windows makes it possible that you can uninstall software is by having a hidden directory under the C:\Windows installation path called Installer.
It contains all the .msi files of programs that were installed on the machine, and enables the OS to uninstall them again.
I haven't known this fact before I started doing this. Cool, isn't it?
Well, as we know from above, we can ask lessmsi to list all files that an msi provides, and using powershell scripting we can obtain a script like:
Get-ChildItem "C:\Windows\Installer" -Filter *.msi | Foreach-Object {
 $K = $_.FullName
 .\lessmsi\lessmsi.exe l -t File $_.FullName | foreach {$_ + $K} | Select-String -Pattern "User32.lib"
}
This command searches for any msi files that contain User32.lib and is an alternative to the approach with the ISO outlined above.
The msi name is a bit obfuscated, but both lessmsi and the windows built in rightclick→properties menu on the msi file contain
useful information about what it exactly installed.
Running into limitations of Wine

If we now run the linker, it seems to not do anything. Have we done something wrong? It doesn't exit but it also doesn't seem to take up much CPU time. It's idling. It would be great if we could obtain the stdout output of the linker to see whether it reports anything suspicious, however rustc doesn't print any output eagerly,
only if there is an error, but apparently the code invoked by the linker script is in a state where something goes wrong but it doesn't abort.
Let's ask the linker script to pipe the output to a file by replacing the invocation with $wine_exec $link_exec $args >> /path/to/log/file 2>&1, and let's run tail -f on that file.
Seems we are getting a crash with a large scary output, but out of some reason the linker didn't exit! It seems that the cause of the crash is the following:
wine: Call from 0x7b44f727 to unimplemented function api-ms-win-crt-string-l1-1-0.dll._memicmp_l, aborting

If we look into the Wine source code, we can see that _memicmp_l is indeed missing. The function seems to be very simple,
it's only a string comparison function with a given locale. Let's implement it and see whether the crash is gone.
After implemeting the function locally, we see that the crash is gone indeed! And the linker produces working binaries! We did it!
Let's now file a bug over at wine to tell them about the missing function. The nice developers of Wine then provide a patch that implements the functionality in a clean fashion. As I'm writing this, some unit tests seem to fail for the patch, so it's not upstreamed yet. But it can be applied to your local clone of Wine 2.19 or 2.20. Edit: The _memicmp_l patch has been upstreamed into the 2.21 release of Wine! So if you have Wine 2.21 or newer, you won't be affected by the crash due to the missing function any more.
I've cleaned up everything, created an automated downloader, and put it onto Github. Enjoy!
Note: the triangle example of glium doesn't work but it doesn't work on the mingw version either, so idk... I have an application using glium which works fine. Also, I've tested some console only applications. They work fine as well.