Skip to content

Instantly share code, notes, and snippets.

@grenade
Last active July 8, 2023 12:11
Show Gist options
  • Star 5 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save grenade/1e63fcb938ff4fc22924da23127e1d81 to your computer and use it in GitHub Desktop.
Save grenade/1e63fcb938ff4fc22924da23127e1d81 to your computer and use it in GitHub Desktop.

How to build firefox on Windows

First some dependencies

If you don't want to understand the why's and hows of the dependency toolchain, skip this section and go straight to the Dependency Installation Guide.

Over the years, the list of things required to build Firefox has grown. Some of these things are really needed in order to do the actual work of compiling the browser. Some of them are needed by the things that are needed for compilation. Some of them are even further removed. Here we're going to attempt to document what does what.

Since I work in Release Engineering at Mozilla and do the majority of my work on the Windows build infrastructure, I'm going to document the toolchain that we use within releng to build Firefox on Windows. That toolchain evolves constantly so if you know better ways to achieve more, better, faster, Windows builds, get in touch.

Getting the code from source control

Mozilla's canonical repository is in Mercurial at: https://hg.mozilla.org/mozilla-central. So if you want to make life easy on yourself, in terms of accessing the latest source code frequently, install Mercurial 3.7.3 (or later). Since it's a large repository, do yourself a favour and make use of the clonebundle extension that is baked into version 3.7.3

Compilers and Linkers

Once you haz teh codez, you need a compiler. Most modern C compilers that run on Windows, can build Firefox. Some are faster than others and some are good for other reasons. We use the compilers and linkers bundled with Visual Studio 2015. Specifically, we use the cl and link executables (and supporting assemblies) found under the VC directory in a typical Visual Studio installation. This is usually %ProgramFiles(x86)%\Microsoft Visual Studio 14.0\VC. Which specific version of the compiler is used, depends on the platform architecture being targetted (32 bit or 64 bit).

  • cl.exe: Compilers produce Common Object File Format (COFF) object (.obj) files.
  • link.exe: Linkers produce executable (.exe) files or dynamic-link libraries (DLLs).

For more info, see: https://msdn.microsoft.com/en-us/library/9s7c9wdw.aspx

Build System

Since pointing a compiler at a codebase the size of Mozilla Central requires a lot of configuration, The next must-have-tool in the Mozilla-builders arsenal is a build system. If you come from the .Net world, this is the tool that does what MSBuild did in that world. At Mozilla we use good old-fashioned make and configure. Of course, being Mozilla, we wrap the calls to make with mozmake. We wrap the calls to mozmake, with pymake. We wrap the calls to pymake, with mach. We wrap the calls to mach, with mozconfig and we wrap the calls to mozconfig, with mozharness. Here's an ugly bullet diagram to help you wrap your head around the preceeding sentences:

  • mozharness
    • mozconfig
      • mach
        • pymake there is documentation which suggests this is deprecated but there are still references to pymake in build configurations
          • mozmake
            • make
              • cl/link (gcc for the 'nix minded)

mozharness is a python thingTM.

mozconfigs are bash shell scripts.

mach is a python thingTM too.

mozmake, make and cl/link are Windows executables (.exe)

Python and all the Python things

If you want to build from the mozharness or mach contexts (eg: without skipping blissfully ahead to make), then you need Python and a bunch of Python modules like pip, setuptools, distutils, wheel, virtualenv and many more. There are dozens of requirements.txt files in the mozilla-central repository each containing multiple python module requirements. Many of the modules required to build firefox are available as wheels (zip archives with a .whl extension) inside the repository. Some, required by mozharness and friends, are not.

Some of the modules you will need, have to be downloaded (with pip) as source and compiled. When you compile a Python module, you have to use the same compiler that the version of Python you are running, was built with. You can do hacky things to get around this, but if you want the-good-lifeTM, Use the same compiler that your version of Python was compiled with.

As mentioned above, we build Firefox with the compilers shipped with Visual Studio 2015 but many of the Python modules supporting the build toolchain were written and adopted before Visual Studio 2015 was a twinkle in Anders Hejlsberg's eye. So yes, you guessed it, you need more than one compiler. Since we use Python 2.7.11, we also install the VC++ for Python compilers.

bash

Since you've now seen the word bash in a document describing a Windows configuration, you know that we're going to need a bash prompt. Mozconfig relies completely on bash so if you're running mozconfigs, you have to have a bash.

make, automake, configure, autoconf, zip, unzip, tar, untar...

Like bash, these are GNU based software depended on by the build system. You need these and more.

Mozilla Build

Lucky for you, many of the components listed above, have been wrapped into a meta-package installer that has been maintained by Mozilla for years. More info at: https://wiki.mozilla.org/MozillaBuild. Pretty much everything except the compilers are included.

Dependency Installation Guide

Which Windows version is best for building Firefox?

Any recent Windows variant should be capable of building firefox. Some of the more recent variants, include tools by default that will help to install the many other tools needed by the Firefox build toolchain.

You need a 64 bit variant if you want to build 64 bit Firefox binaries. 32 bit binaries can be built on either 64 bit or 32 bit variants.

Buildbot uses Windows Server 2008 R2 to build Firefox.

TaskCluster uses Windows Server 2012 R2 to build Firefox.

Desired State Configuration (DSC) with Powershell

DSC (!!!insert version!!!) is bundled into Windows Server 2012 and Windows 10 as part of Powershell 4 (and 5), by default. DSC makes it easy to install our dependency toolchain from a simple json manifest. You don't need to use DSC to install your toolchain. You can also do it manually, it's just quicker. The steps for installing the build toolchain using DSC are documented here but it's pretty trivial to use the same instructions to install and set up your toolchain without DSC.

Install DSC

If you're using a version of Windows that does not include DSC, you can install DSC using the following powershell commands (at an elevated Powershell prompt (!!!insert link to running Powershell as an Admin!!!)):

# relax restrictions on running remote scripts for the current process
Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope Process -Force

# check if the install/upgrade is required
if ($PSVersionTable.PSVersion.Major -lt 4) {

   # start the windows update service if it isn't already running (required by Powershell upgrade)
   if ((Get-Service wuauserv).Status -ne "Running") {
     Start-Service wuauserv
   }
   
   # install or upgrade chocolatey
   Invoke-Expression ((New-Object Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))
   
   # install or upgrade Powershell
   & choco @('upgrade', 'powershell', '-y')
   
   # restart the host
   & shutdown @('-r', '-t', '0', '-c', 'Powershell upgraded', '-f', '-d', 'p:4:1')
}
Install your own modified toolchain using DSC (optional, for tinkerers)

If you need to make changes to the manifest before you install everything it contains (eg: because updates have become available since this guide was written, or you just have better ideas), copy these files to your own new gist:

Install the complete toolchain posted here using DSC

To run the DSC configuration using the manifest in this gist, execute the following powershell commands (at an elevated Powershell prompt). If You've forked or copied this gist and modified it, change the url in the command to point to your own gist.

Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope Process -Force
Invoke-Expression ((New-Object Net.WebClient).DownloadString('https://gist.githubusercontent.com/grenade/1e63fcb938ff4fc22924da23127e1d81/raw/Run-RemoteDesiredStateConfig.ps1'))

Environment Configuration Guide

Now that you have the dependency toolchain installed, you need to set some environment variables to tell the build system where to find everything.

PATH

Your command prompt uses the PATH environment variable to figure out what command you intend to run, when you don't provide the full path to the executable or batch script at each prompt. Some installers add their path to the system PATH as part of the installation. Some do if you tell them to, with an install argument and some, just don't bother. In any case, the following programs will need their paths in the PATH environment variable in order for the build scripts to function as intended:

  • python.exe: If you use the version of python bundled with mozilla-build, the path will usually be: %SystemDrive%\mozilla-build\python. If you installed python using the normal installer using default settings, the path is likely to be: %SystemDrive%\Python27.
  • python modules (pip.exe, virtualenv.exe, etc): These are usually in a scripts folder under the python path eg: %SystemDrive%\mozilla-build\python\Scripts.
  • zip.exe, unzip.exe: %SystemDrive%\mozilla-build\info-zip
  • msys GNU tools (bash.exe, sh.exe, etc): %SystemDrive%\mozilla-build\msys\bin
  • msys GNU tools (make.exe, autoconf, etc): %SystemDrive%\mozilla-build\msys\local\bin
  • nsis.exe: %SystemDrive%\mozilla-build\nsis-3.0b3
  • wget.exe: %SystemDrive%\mozilla-build\wget
  • yasm.exe: %SystemDrive%\mozilla-build\yasm

Microsoft Visual C++ Compiler for Python 2.7

In order for python and python modules to find and make use of the compiler, some environment variables need to be set. This is only important the first time a module which needs compilation is used. The installer does set these but only for the user that runs the installer. The variables needed are:

  • VCINSTALLDIR: should be set to the install path. Possibly: %SystemRoot%\SysWOW64\config\systemprofile\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC.
  • DISTUTILS_USE_SDK: set to 1
  • MSSdk: set to 1

tooltool

tooltool is an artifact repository (remote) and artifact cache (local). The build system uses tooltool to provide frequently updated binary artifacts required by the toolchain. tooltool is implemented as a python script name tooltool.py.

Notably, tooltool downloads and makes available the following binaries:

  • rustc: The rust compiler.
  • sccache
  • mozmake
  • vs2015u1: Visual Studio 2015

In order to use tooltool the environment variable TOOLTOOL_CACHE must be set to a local folder path. The releng build system uses %SystemDrive%\builds\tooltool-cache.

{
"Components": [
{
"ComponentName": "LogDirectory",
"ComponentType": "DirectoryCreate",
"Comment": "Required by OpenCloudConfig for DSC logging",
"Path": "C:\\log"
},
{
"ComponentName": "vcredist_vs2010_x86",
"ComponentType": "ExeInstall",
"Comment": "Required by yasm (c:/mozilla-build/yasm/yasm.exe)",
"Url": "http://download.microsoft.com/download/C/6/D/C6D0FD4E-9E53-4897-9B91-836EBA2AACD3/vcredist_x86.exe",
"Arguments": [
"/install",
"/passive",
"/norestart",
"/Log",
"C:\\log\\vcredist_vs2010_x86-install.log"
],
"Validate": {
"PathsExist": [
"C:\\Windows\\System32\\msvcr100.dll"
]
}
},
{
"ComponentName": "vcredist_vs2010_x64",
"ComponentType": "ExeInstall",
"Comment": "Required by yasm (c:/mozilla-build/yasm/yasm.exe)",
"Url": "http://download.microsoft.com/download/A/8/0/A80747C3-41BD-45DF-B505-E9710D2744E0/vcredist_x64.exe",
"Arguments": [
"/install",
"/passive",
"/norestart",
"/Log",
"C:\\log\\vcredist_vs2010_x64-install.log"
],
"Validate": {
"PathsExist": [
"C:\\Windows\\SysWOW64\\msvcr100.dll"
]
}
},
{
"ComponentName": "vcredist_vs2013_x86",
"ComponentType": "ExeInstall",
"Comment": "Required by rustc (tooltool artefact)",
"Url": "http://download.microsoft.com/download/2/E/6/2E61CFA4-993B-4DD4-91DA-3737CD5CD6E3/vcredist_x86.exe",
"Arguments": [
"/install",
"/passive",
"/norestart",
"/Log",
"C:\\log\\vcredist_vs2013_x86-install.log"
],
"Validate": {
"PathsExist": [
"C:\\Windows\\System32\\msvcr120.dll"
]
}
},
{
"ComponentName": "vcredist_vs2013_x64",
"ComponentType": "ExeInstall",
"Comment": "Required by rustc (tooltool artefact)",
"Url": "http://download.microsoft.com/download/2/E/6/2E61CFA4-993B-4DD4-91DA-3737CD5CD6E3/vcredist_x64.exe",
"Arguments": [
"/install",
"/passive",
"/norestart",
"/Log",
"C:\\log\\vcredist_vs2013_x64-install.log"
],
"Validate": {
"PathsExist": [
"C:\\Windows\\SysWOW64\\msvcr120.dll"
]
}
},
{
"ComponentName": "windowssdksetup",
"ComponentType": "ExeInstall",
"Comment": "Deprecated? Test building without and remove or justify",
"Url": "http://download.microsoft.com/download/B/0/C/B0C80BA3-8AD6-4958-810B-6882485230B5/standalonesdk/sdksetup.exe",
"Arguments": [
"/Quiet",
"/NoRestart",
"/Log",
"C:\\log\\windowssdksetup.log"
]
},
{
"ComponentName": "MozillaBuildSetup",
"ComponentType": "ExeInstall",
"Comment": "Base Firefox on Windows build requirement",
"Arguments": [
"/S",
"/D=C:\\mozilla-build"
],
"Url": "http://ftp.mozilla.org/pub/mozilla/libraries/win32/MozillaBuildSetup-2.2.0.exe",
"Validate": {
"PathsExist": [
"C:\\mozilla-build\\msys\\bin\\sh.exe",
"C:\\mozilla-build\\msys\\local\\bin\\make.exe",
"C:\\mozilla-build\\msys\\local\\bin\\autoconf-2.13",
"C:\\mozilla-build\\info-zip\\unzip.exe",
"C:\\mozilla-build\\info-zip\\zip.exe"
],
"FilesContain": [
{
"Path": "C:\\mozilla-build\\VERSION",
"Match": "2.2.0"
}
]
}
},
{
"ComponentName": "VCForPython",
"ComponentType": "MsiInstall",
"Comment": "Required by BuildBot VirtualEnv used by Mozharness (c:/mozilla-build/buildbotve)",
"Url": "https://download.microsoft.com/download/7/9/6/796EF2E4-801B-4FC4-AB28-B59FBF6D907B/VCForPython27.msi",
"Name": "Microsoft Visual C++ Compiler Package for Python 2.7",
"ProductId": "692514A8-5484-45FC-B0AE-BE2DF7A75891"
},
{
"ComponentName": "MercurialPythonPatch",
"ComponentType": "SymbolicLink",
"DependsOn": [
{
"ComponentType": "ExeInstall",
"ComponentName": "MozillaBuildSetup"
}
],
"Target": "C:\\mozilla-build\\python\\python27.dll",
"Link": "C:\\mozilla-build\\python\\Scripts\\python27.dll"
},
{
"ComponentName": "MercurialConfig",
"ComponentType": "FileDownload",
"DependsOn": [
{
"ComponentType": "ExeInstall",
"ComponentName": "MozillaBuildSetup"
}
],
"Source": "https://raw.githubusercontent.com/MozRelOps/OpenCloudConfig/master/userdata/Configuration/Mercurial/mercurial.ini",
"Target": "C:\\mozilla-build\\python\\Scripts\\mercurial.ini"
},
{
"ComponentName": "pip-upgrade-pip",
"ComponentType": "CommandRun",
"DependsOn": [
{
"ComponentType": "ExeInstall",
"ComponentName": "MozillaBuildSetup"
}
],
"Command": "C:\\mozilla-build\\python\\python.exe",
"Arguments": [
"-m",
"pip",
"install",
"--upgrade",
"pip==8.1.1"
],
"Validate": {
"CommandsReturn": [
{
"Command": "C:\\mozilla-build\\python\\python.exe",
"Arguments": [
"-m",
"pip",
"show",
"pip"
],
"Match": "Version: 8.1.1"
}
]
}
},
{
"ComponentName": "pip-upgrade-setuptools",
"ComponentType": "CommandRun",
"DependsOn": [
{
"ComponentType": "CommandRun",
"ComponentName": "pip-upgrade-pip"
}
],
"Command": "C:\\mozilla-build\\python\\python.exe",
"Arguments": [
"-m",
"pip",
"install",
"--upgrade",
"setuptools==20.7.0"
],
"Validate": {
"CommandsReturn": [
{
"Command": "C:\\mozilla-build\\python\\python.exe",
"Arguments": [
"-m",
"pip",
"show",
"setuptools"
],
"Match": "Version: 20.7.0"
}
]
}
},
{
"ComponentName": "pip-upgrade-virtualenv",
"ComponentType": "CommandRun",
"DependsOn": [
{
"ComponentType": "CommandRun",
"ComponentName": "pip-upgrade-pip"
}
],
"Command": "C:\\mozilla-build\\python\\python.exe",
"Arguments": [
"-m",
"pip",
"install",
"--upgrade",
"virtualenv==15.0.1"
],
"Validate": {
"CommandsReturn": [
{
"Command": "C:\\mozilla-build\\python\\python.exe",
"Arguments": [
"-m",
"pip",
"show",
"virtualenv"
],
"Match": "Version: 15.0.1"
}
]
}
},
{
"ComponentName": "pip-upgrade-wheel",
"ComponentType": "CommandRun",
"DependsOn": [
{
"ComponentType": "CommandRun",
"ComponentName": "pip-upgrade-pip"
}
],
"Command": "C:\\mozilla-build\\python\\python.exe",
"Arguments": [
"-m",
"pip",
"install",
"--upgrade",
"wheel==0.29.0"
],
"Validate": {
"CommandsReturn": [
{
"Command": "C:\\mozilla-build\\python\\python.exe",
"Arguments": [
"-m",
"pip",
"show",
"wheel"
],
"Match": "Version: 0.29.0"
}
]
}
},
{
"ComponentName": "pip-upgrade-pypiwin32",
"ComponentType": "CommandRun",
"DependsOn": [
{
"ComponentType": "CommandRun",
"ComponentName": "pip-upgrade-pip"
}
],
"Command": "C:\\mozilla-build\\python\\python.exe",
"Arguments": [
"-m",
"pip",
"install",
"--upgrade",
"pypiwin32==219"
],
"Validate": {
"CommandsReturn": [
{
"Command": "C:\\mozilla-build\\python\\python.exe",
"Arguments": [
"-m",
"pip",
"show",
"pypiwin32"
],
"Match": "Version: 219"
}
]
}
},
{
"ComponentName": "pip-upgrade-requests",
"ComponentType": "CommandRun",
"DependsOn": [
{
"ComponentType": "CommandRun",
"ComponentName": "pip-upgrade-pip"
}
],
"Command": "C:\\mozilla-build\\python\\python.exe",
"Arguments": [
"-m",
"pip",
"install",
"--upgrade",
"requests==2.9.1"
],
"Validate": {
"CommandsReturn": [
{
"Command": "C:\\mozilla-build\\python\\python.exe",
"Arguments": [
"-m",
"pip",
"show",
"requests"
],
"Match": "Version: 2.9.1"
}
]
}
},
{
"ComponentName": "pip-upgrade-psutil",
"ComponentType": "CommandRun",
"DependsOn": [
{
"ComponentType": "CommandRun",
"ComponentName": "pip-upgrade-pip"
}
],
"Command": "C:\\mozilla-build\\python\\python.exe",
"Arguments": [
"-m",
"pip",
"install",
"--upgrade",
"psutil==4.1.0"
],
"Validate": {
"CommandsReturn": [
{
"Command": "C:\\mozilla-build\\python\\python.exe",
"Arguments": [
"-m",
"pip",
"show",
"psutil"
],
"Match": "Version: 4.1.0"
}
]
}
},
{
"ComponentName": "ToolToolInstall",
"ComponentType": "FileDownload",
"DependsOn": [
{
"ComponentType": "ExeInstall",
"ComponentName": "MozillaBuildSetup"
}
],
"Source": "https://raw.githubusercontent.com/mozilla/build-tooltool/master/tooltool.py",
"Target": "C:\\mozilla-build\\tooltool.py"
},
{
"ComponentName": "BuildBotVirtualEnvCreate",
"ComponentType": "CommandRun",
"DependsOn": [
{
"ComponentType": "CommandRun",
"ComponentName": "pip-upgrade-virtualenv"
},
{
"ComponentType": "CommandRun",
"ComponentName": "pip-upgrade-setuptools"
},
{
"ComponentType": "CommandRun",
"ComponentName": "pip-upgrade-wheel"
}
],
"Command": "C:\\mozilla-build\\python\\python.exe",
"Arguments": [
"-m",
"virtualenv",
"C:\\mozilla-build\\buildbotve"
],
"Validate": {
"PathsExist": [
"C:\\mozilla-build\\buildbotve",
"C:\\mozilla-build\\buildbotve\\Scripts"
]
}
},
{
"ComponentName": "BuildBotVirtualEnvDotPyInstall",
"ComponentType": "FileDownload",
"DependsOn": [
{
"ComponentType": "CommandRun",
"ComponentName": "BuildBotVirtualEnvCreate"
}
],
"Source": "https://hg.mozilla.org/mozilla-central/raw-file/78babd21215d/python/virtualenv/virtualenv.py",
"Target": "C:\\mozilla-build\\buildbotve\\virtualenv.py"
},
{
"ComponentName": "BuildBotVirtualEnvSupportDirectory",
"ComponentType": "DirectoryCreate",
"Path": "C:\\mozilla-build\\buildbotve\\virtualenv_support",
"DependsOn": [
{
"ComponentType": "CommandRun",
"ComponentName": "BuildBotVirtualEnvCreate"
},
{
"ComponentType": "FileDownload",
"ComponentName": "BuildBotVirtualEnvDotPyInstall"
}
]
},
{
"ComponentName": "BuildBotVirtualEnvSupport-pip-8.1.1-py2.py3-none-any.whl",
"ComponentType": "FileDownload",
"DependsOn": [
{
"ComponentType": "DirectoryCreate",
"ComponentName": "BuildBotVirtualEnvSupportDirectory"
}
],
"Source": "https://pypi.python.org/packages/py2.py3/p/pip/pip-8.1.1-py2.py3-none-any.whl#md5=22db7b6a517a09c29d54a76650f170eb",
"Target": "C:\\mozilla-build\\buildbotve\\virtualenv_support\\pip-8.1.1-py2.py3-none-any.whl"
},
{
"ComponentName": "BuildBotVirtualEnvSupport-setuptools-20.7.0-py2.py3-none-any.whl",
"ComponentType": "FileDownload",
"DependsOn": [
{
"ComponentType": "DirectoryCreate",
"ComponentName": "BuildBotVirtualEnvSupportDirectory"
}
],
"Source": "https://pypi.python.org/packages/py2.py3/s/setuptools/setuptools-20.7.0-py2.py3-none-any.whl#md5=9e0248519bf3e19ca6b3c217dc633ba8",
"Target": "C:\\mozilla-build\\buildbotve\\virtualenv_support\\setuptools-20.7.0-py2.py3-none-any.whl"
},
{
"ComponentName": "BuildBotVirtualEnvSupport-virtualenv-15.0.1-py2.py3-none-any.whl",
"ComponentType": "FileDownload",
"DependsOn": [
{
"ComponentType": "DirectoryCreate",
"ComponentName": "BuildBotVirtualEnvSupportDirectory"
}
],
"Source": "https://pypi.python.org/packages/py2.py3/v/virtualenv/virtualenv-15.0.1-py2.py3-none-any.whl#md5=1fca0d03249563ccc475d39cfe71491e",
"Target": "C:\\mozilla-build\\buildbotve\\virtualenv_support\\virtualenv-15.0.1-py2.py3-none-any.whl"
},
{
"ComponentName": "BuildBotVirtualEnvSupport-wheel-0.29.0-py2.py3-none-any.whl",
"ComponentType": "FileDownload",
"DependsOn": [
{
"ComponentType": "DirectoryCreate",
"ComponentName": "BuildBotVirtualEnvSupportDirectory"
}
],
"Source": "https://pypi.python.org/packages/py2.py3/w/wheel/wheel-0.29.0-py2.py3-none-any.whl#md5=d7db45db5c131af262b8ffccde46a88a",
"Target": "C:\\mozilla-build\\buildbotve\\virtualenv_support\\wheel-0.29.0-py2.py3-none-any.whl"
},
{
"ComponentName": "BuildBotVirtualEnvSupport-pypiwin32-219-cp27-none-win_amd64.whl",
"ComponentType": "FileDownload",
"DependsOn": [
{
"ComponentType": "DirectoryCreate",
"ComponentName": "BuildBotVirtualEnvSupportDirectory"
}
],
"Source": "https://pypi.python.org/packages/cp27/p/pypiwin32/pypiwin32-219-cp27-none-win_amd64.whl#md5=d7bafcf3cce72c3ce9fdd633a262c335",
"Target": "C:\\mozilla-build\\buildbotve\\virtualenv_support\\pypiwin32-219-cp27-none-win_amd64.whl"
},
{
"ComponentName": "BuildBotVirtualEnvSupport-pypiwin32-219-cp27-none-win32.whl",
"ComponentType": "FileDownload",
"DependsOn": [
{
"ComponentType": "DirectoryCreate",
"ComponentName": "BuildBotVirtualEnvSupportDirectory"
}
],
"Source": "https://pypi.python.org/packages/cp27/p/pypiwin32/pypiwin32-219-cp27-none-win32.whl#md5=a8b0c1b608c1afeb18cd38d759ee5e29",
"Target": "C:\\mozilla-build\\buildbotve\\virtualenv_support\\pypiwin32-219-cp27-none-win32.whl"
},
{
"ComponentName": "BuildBotVirtualEnvSupport-requests-2.7.0-py2.py3-none-any.whl",
"ComponentType": "FileDownload",
"DependsOn": [
{
"ComponentType": "DirectoryCreate",
"ComponentName": "BuildBotVirtualEnvSupportDirectory"
}
],
"Source": "https://pypi.python.org/packages/2.7/r/requests/requests-2.7.0-py2.py3-none-any.whl#md5=564fb256f865a79f977e57b79d31659a",
"Target": "C:\\mozilla-build\\buildbotve\\virtualenv_support\\requests-2.7.0-py2.py3-none-any.whl"
},
{
"ComponentName": "BuildBotVirtualEnvSupport-requests-2.8.1-py2.py3-none-any.whl",
"ComponentType": "FileDownload",
"DependsOn": [
{
"ComponentType": "DirectoryCreate",
"ComponentName": "BuildBotVirtualEnvSupportDirectory"
}
],
"Source": "https://pypi.python.org/packages/2.7/r/requests/requests-2.8.1-py2.py3-none-any.whl#md5=46f1d621daa3ab38958a42f51478b1ee",
"Target": "C:\\mozilla-build\\buildbotve\\virtualenv_support\\requests-2.8.1-py2.py3-none-any.whl"
},
{
"ComponentName": "BuildBotVirtualEnvSupport-requests-2.9.1-py2.py3-none-any.whl",
"ComponentType": "FileDownload",
"DependsOn": [
{
"ComponentType": "DirectoryCreate",
"ComponentName": "BuildBotVirtualEnvSupportDirectory"
}
],
"Source": "https://pypi.python.org/packages/2.7/r/requests/requests-2.9.1-py2.py3-none-any.whl#md5=58a444aaa02780ad01983f5f540e67b2",
"Target": "C:\\mozilla-build\\buildbotve\\virtualenv_support\\requests-2.9.1-py2.py3-none-any.whl"
},
{
"ComponentName": "BuildBotVirtualEnvSupport-psutil-4.1.0-cp27-cp27m-win_amd64.whl",
"ComponentType": "FileDownload",
"DependsOn": [
{
"ComponentType": "DirectoryCreate",
"ComponentName": "BuildBotVirtualEnvSupportDirectory"
}
],
"Source": "https://pypi.python.org/packages/2.7/p/psutil/psutil-4.1.0-cp27-cp27m-win_amd64.whl#md5=ab1ce766f282ab4246fd00f78a6f72b4",
"Target": "C:\\mozilla-build\\buildbotve\\virtualenv_support\\psutil-4.1.0-cp27-cp27m-win_amd64.whl"
},
{
"ComponentName": "BuildBotVirtualEnvSupport-psutil-4.1.0-cp27-cp27m-win32.whl",
"ComponentType": "FileDownload",
"DependsOn": [
{
"ComponentType": "DirectoryCreate",
"ComponentName": "BuildBotVirtualEnvSupportDirectory"
}
],
"Source": "https://pypi.python.org/packages/2.7/p/psutil/psutil-4.1.0-cp27-cp27m-win32.whl#md5=4bc43677c085c31f6cd288c4d52422ec",
"Target": "C:\\mozilla-build\\buildbotve\\virtualenv_support\\psutil-4.1.0-cp27-cp27m-win32.whl"
},
{
"ComponentName": "env-PATH",
"ComponentType": "EnvironmentVariableUniqueAppend",
"DependsOn": [
{
"ComponentType": "ExeInstall",
"ComponentName": "MozillaBuildSetup"
}
],
"Name": "PATH",
"Values": [
"C:\\mozilla-build\\info-zip",
"C:\\mozilla-build\\msys\\bin",
"C:\\mozilla-build\\msys\\local\\bin",
"C:\\mozilla-build\\python",
"C:\\mozilla-build\\python\\Scripts",
"C:\\mozilla-build\\wget",
"C:\\mozilla-build\\yasm"
],
"Target": "Machine"
},
{
"ComponentName": "env-MOZBUILDDIR",
"ComponentType": "EnvironmentVariableSet",
"DependsOn": [
{
"ComponentType": "ExeInstall",
"ComponentName": "MozillaBuildSetup"
}
],
"Name": "MOZBUILDDIR",
"Value": "C:\\mozilla-build",
"Target": "Machine"
},
{
"ComponentName": "env-MOZILLABUILD",
"ComponentType": "EnvironmentVariableSet",
"DependsOn": [
{
"ComponentType": "ExeInstall",
"ComponentName": "MozillaBuildSetup"
}
],
"Name": "MOZILLABUILD",
"Value": "C:\\mozilla-build",
"Target": "Machine"
},
{
"ComponentName": "env-MOZ_TOOLS",
"ComponentType": "EnvironmentVariableSet",
"DependsOn": [
{
"ComponentType": "ExeInstall",
"ComponentName": "MozillaBuildSetup"
}
],
"Name": "MOZ_TOOLS",
"Value": "C:\\mozilla-build\\moztools-x64",
"Target": "Machine"
},
{
"ComponentName": "env-VCINSTALLDIR",
"ComponentType": "EnvironmentVariableSet",
"DependsOn": [
{
"ComponentType": "MsiInstall",
"ComponentName": "VCForPython"
}
],
"Name": "VCINSTALLDIR",
"Value": "C:\\Windows\\SysWOW64\\config\\systemprofile\\AppData\\Local\\Programs\\Common\\Microsoft\\Visual C++ for Python\\9.0\\VC",
"Target": "Machine"
}
]
}
<#
This Source Code Form is subject to the terms of the Mozilla Public
License, v. 2.0. If a copy of the MPL was not distributed with this
file, You can obtain one at http://mozilla.org/MPL/2.0/.
#>
Configuration DynamicConfig {
Import-DscResource -ModuleName PSDesiredStateConfiguration
$manifest = (Invoke-WebRequest -Uri 'https://gist.githubusercontent.com/grenade/1e63fcb938ff4fc22924da23127e1d81/raw/firefox-build-toolchain.json' -UseBasicParsing | ConvertFrom-Json)
# this hashtable maps json manifest component types to DSC component types for dependency mapping
$componentMap = @{
'DirectoryCreate' = 'File';
'DirectoryDelete' = 'Script';
'CommandRun' = 'Script';
'ExeInstall' = 'Script'
}
foreach ($item in $manifest.Components) {
switch ($item.ComponentType) {
'DirectoryCreate' {
File ('DirectoryCreate-{0}' -f $item.ComponentName) {
DependsOn = @($item.DependsOn | % ('[{0}]{1}-{2}' -f $componentMap.Get_Item($_.ComponentType), $_.ComponentType, $_.ComponentName))
Ensure = 'Present'
Type = 'Directory'
DestinationPath = $($item.Path)
}
Log ('Log-DirectoryCreate-{0}' -f $item.ComponentName) {
DependsOn = ('[File]DirectoryCreate-{0}' -f $item.ComponentName)
Message = ('{0}: {1}, completed' -f $item.ComponentType, $item.ComponentName)
}
}
'DirectoryDelete' {
Script ('DirectoryDelete-{0}' -f $item.ComponentName) {
DependsOn = @($item.DependsOn | % ('[{0}]{1}-{2}' -f $componentMap.Get_Item($_.ComponentType), $_.ComponentType, $_.ComponentName))
GetScript = "@{ DirectoryDelete = $($item.Path) }"
SetScript = {
try {
Remove-Item $($using:item.Path) -Confirm:$false -force
} catch {
Start-Process 'icacls' -ArgumentList @($($using:item.Path), '/grant', ('{0}:(OI)(CI)F' -f $env:Username), '/inheritance:r') -Wait -NoNewWindow -PassThru | Out-Null
Remove-Item $($using:item.Path) -Confirm:$false -force
# todo: another try catch block with move to recycle bin, empty recycle bin
}
}
TestScript = { (-not (Test-Path -Path $($using:item.Path) -ErrorAction SilentlyContinue)) }
}
Log ('Log-DirectoryDelete-{0}' -f $item.ComponentName) {
DependsOn = ('[Script]DirectoryDelete-{0}' -f $($item.Path).Replace(':', '').Replace('\', '_'))
Message = ('{0}: {1}, completed' -f $item.ComponentType, $item.ComponentName)
}
}
'CommandRun' {
Script ('CommandRun-{0}' -f $item.ComponentName) {
DependsOn = @($item.DependsOn | % ('[{0}]{1}-{2}' -f $componentMap.Get_Item($_.ComponentType), $_.ComponentType, $_.ComponentName))
GetScript = "@{ CommandRun = $item.ComponentName }"
SetScript = {
Start-Process $($using:item.Command) -ArgumentList @($using:item.InstallArguments | % { $($_) }) -Wait -NoNewWindow -PassThru -RedirectStandardOutput ('{0}\log\{1}-{2}-stdout.log' -f $env:SystemDrive, [DateTime]::Now.ToString("yyyyMMddHHmmss"), $using:item.ComponentName) -RedirectStandardError ('{0}\log\{1}-{2}-stderr.log' -f $env:SystemDrive, [DateTime]::Now.ToString("yyyyMMddHHmmss"), $using:item.ComponentName)
}
TestScript = { $false } # todo: implement
}
Log ('Log-CommandRun-{0}' -f $item.ComponentName) {
DependsOn = ('[Script]CommandRun-{0}' -f $item.ComponentName)
Message = ('{0}: {1}, completed' -f $item.ComponentType, $item.ComponentName)
}
}
'FileDownload' {
Script ('FileDownload-{0}' -f $item.ComponentName) {
DependsOn = @($item.DependsOn | % ('[{0}]{1}-{2}' -f $componentMap.Get_Item($_.ComponentType), $_.ComponentType, $_.ComponentName))
GetScript = "@{ FileDownload = $item.ComponentName }"
SetScript = {
try {
(New-Object Net.WebClient).DownloadFile($using:item.Source, $using:item.Target)
} catch {
# handle redirects (eg: sourceforge)
Invoke-WebRequest -Uri $using:item.Source -OutFile $using:item.Target -UserAgent [Microsoft.PowerShell.Commands.PSUserAgent]::FireFox
}
Unblock-File -Path $using:item.Target
}
TestScript = { return (Test-Path -Path $using:item.Target -ErrorAction SilentlyContinue) }
}
Log ('Log-FileDownload-{0}' -f $item.ComponentName) {
DependsOn = ('[Script]FileDownload-{0}' -f $item.ComponentName)
Message = ('{0}: {1}, completed' -f $item.ComponentType, $item.ComponentName)
}
}
'ExeInstall' {
Script ('Download-{0}' -f $item.ComponentName) {
DependsOn = @($item.DependsOn | % ('[{0}]{1}-{2}' -f $componentMap.Get_Item($_.ComponentType), $_.ComponentType, $_.ComponentName))
GetScript = "@{ ExeDownload = $item.ComponentName }"
SetScript = {
# todo: handle non-http fetches
try {
(New-Object Net.WebClient).DownloadFile($using:item.Url, ('{0}\Temp\{1}.exe' -f $env:SystemRoot, $using:item.ComponentName))
} catch {
# handle redirects (eg: sourceforge)
Invoke-WebRequest -Uri $using:item.Url -OutFile ('{0}\Temp\{1}.exe' -f $env:SystemRoot, $using:item.ComponentName) -UserAgent [Microsoft.PowerShell.Commands.PSUserAgent]::FireFox
}
Unblock-File -Path ('{0}\Temp\{1}.exe' -f $env:SystemRoot, $using:item.ComponentName)
}
TestScript = { return (Test-Path -Path ('{0}\Temp\{1}.exe' -f $env:SystemRoot, $using:item.ComponentName) -ErrorAction SilentlyContinue) }
}
Log ('Log-Download-{0}' -f $item.ComponentName) {
DependsOn = ('[Script]Download-{0}' -f $item.ComponentName)
Message = ('{0}: {1}, download completed' -f $item.ComponentType, $item.ComponentName)
}
Script ('ExeInstall-{0}' -f $item.ComponentName) {
DependsOn = ('[Script]Download-{0}' -f $item.ComponentName)
GetScript = "@{ ExeInstall = $item.ComponentName }"
SetScript = {
$exe = ('{0}\Temp\{1}' -f $env:SystemRoot, $using:item.ComponentName)
$process = Start-Process $exe -ArgumentList @('/Q') -Wait -NoNewWindow -PassThru -RedirectStandardOutput ('{0}\log\{1}-{2}-stdout.log' -f $env:SystemDrive, [DateTime]::Now.ToString("yyyyMMddHHmmss"), [IO.Path]::GetFileNameWithoutExtension($exe)) -RedirectStandardError ('{0}\log\{1}-{2}-stderr.log' -f $env:SystemDrive, [DateTime]::Now.ToString("yyyyMMddHHmmss"), [IO.Path]::GetFileNameWithoutExtension($exe))
if (-not (($process.ExitCode -eq 0) -or ($using:item.AllowedExitCodes -contains $process.ExitCode))) {
throw
}
}
TestScript = {
(
# if no validations are specified, this function will return $false and cause the exe package to be (re)installed.
(
(($using:item.Validate.PathsExist) -and ($using:item.Validate.Paths.Length -gt 0)) -or
(($using:item.Validate.CommandsReturn) -and ($using:item.Validate.CommandsReturn.Length -gt 0)) -or
(($using:item.Validate.FilesContain) -and ($using:item.Validate.FilesContain.Length -gt 0))
) -and (
# either no validation paths-exist are specified
(-not ($using:item.Validate.PathsExist)) -or (
# validation paths-exist are specified
(($using:item.Validate.PathsExist) -and ($using:item.Validate.PathsExist.Length -gt 0)) -and
# all validation paths-exist are satisfied (exist on the instance)
(-not (@($using:item.Validate.PathsExist | % {
(Test-Path -Path $_.Path -ErrorAction SilentlyContinue)
}) -contains $false))
)
) -and (
# either no validation commands-return are specified
(-not ($using:item.Validate.CommandsReturn)) -or (
# validation commands-return are specified
(($using:item.Validate.CommandsReturn) -and ($using:item.Validate.CommandsReturn.Length -gt 0)) -and
# all validation commands-return are satisfied
($false) # todo: implement
)
) -and (
# either no validation files-contain are specified
(-not ($using:item.Validate.FilesContain)) -or (
# validation files-contain are specified
(($using:item.Validate.FilesContain) -and ($using:item.Validate.FilesContain.Length -gt 0)) -and
# all validation files-contain are satisfied
(-not (@($using:item.Validate.FilesContain | % {
$fc = $_
(((Get-Content $fc.Path) | % {
$_ -match $fc.Match
}) -contains $true) # a line within the file contained a match
}) -contains $false)) # no files failed to contain a match (see '-not' above)
)
)
)
}
}
Log ('Log-ExeInstall-{0}' -f $item.ComponentName) {
DependsOn = ('[Script]ExeInstall-{0}' -f $item.ComponentName)
Message = ('{0}: {1}, completed' -f $item.ComponentType, $item.ComponentName)
}
}
}
}
}
  • what is overwriting .../workspace/build/buildprops.json

  • where should .../workspace/build/src/old-configure.vars be coming from or how is it generated (it never get's created in current build)?

  • what is the correct way to get python to acknowledge that VCForPython is installed? I currently have to run the following before calling mozharness:

      set VCINSTALLDIR=C:\Windows\SysWOW64\config\systemprofile\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC
      "%VCINSTALLDIR%\..\vcvarsall.bat"
    
function Run-RemoteDesiredStateConfig {
param (
[string] $url
)
$config = [IO.Path]::GetFileNameWithoutExtension($url)
$target = ('{0}\{1}.ps1' -f $env:Temp, $config)
(New-Object Net.WebClient).DownloadFile($url, $target)
Unblock-File -Path $target
. $target
$mof = ('{0}\{1}' -f $env:Temp, $config)
Invoke-Expression "$config -OutputPath $mof"
Start-DscConfiguration -Path "$mof" -Wait -Verbose -Force
}
New-Item -ItemType Directory -Force -Path ('{0}\log' -f $env:SystemDrive)
$logFile = ('{0}\log\{1}.dsc-run.log' -f $env:SystemDrive, [DateTime]::Now.ToString("yyyyMMddHHmmss"))
$manifestConfig = 'https://gist.githubusercontent.com/grenade/1e63fcb938ff4fc22924da23127e1d81/raw/JsonManifestConfig.ps1'
Start-Transcript -Path $logFile -Append
Run-RemoteDesiredStateConfig -url $manifestConfig
Stop-Transcript
::SET WORKSPACE=C:\home\worker\workspace
SET WORKSPACE=%cd%
cd %WORKSPACE%
SET GECKO_BASE_REPOSITORY=C:\builds\hg-shared\mozilla-central
SET GECKO_HEAD_REPOSITORY=https://hg.mozilla.org/try
SET GECKO_HEAD_REF=1ee783d30721cf277a61f7f95ba5633fbe5adadc
SET GECKO_HEAD_REV=1ee783d30721cf277a61f7f95ba5633fbe5adadc
C:\home\worker\workspace\checkout-sources.cmd
SET SRCSRV_ROOT=%GECKO_HEAD_REPOSITORY%
SET UPLOAD_PATH=%WORKSPACE%\public\build
mklink .\buildprops.json C:\home\worker\workspace\buildprops.json
%VCINSTALLDIR%\..\vcvarsall.bat
SET MOZHARNESS_SCRIPT=mozharness\scripts\fx_desktop_build.py
SET MOZHARNESS_CONFIG=builds\releng_base_windows_64_builds.py
%WORKSPACE%\build\src\testing\taskcluster\scripts\builder\build-windows.cmd
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment