The following is an internal memo I recently wrote to explain the structure of a project I have been working for the last 2 years. While there are a number of details that can only be understood if you have the mentioned repos in front of you, I figured this might give some interesting ideas to other devs.
about the project scaffolding structure used among other for BADASS and AVALON
Developing software is hard: a single piece of code can depend on multiple other projects, which each has its own dependencies. This is commonly referred to as dependency hell and remains a challenging problem for almost every project.
To solve the dependency hell, many languages rely on a single point of failure common project repository from where the required dependencies can be added to the project to build.
Node uses npm, Python uses anaconda, Ruby uses gem, Rust uses cargo. For C or C++ however, there aren't any real, multi-platform solutions. There's apt and yum on Linux for common software packages, as well as brew on macOS, and choco on Windows, but they don't permit to link a specific version of a 3rd party library with the project in development.
Conan is a step in the right direction, but the project being in its infancy, it's still miles away from being a commonly accepted solution.
Hence we have to roll our own solution.
To develop our C++ project, we require the following:
- 3rd party libraries, preferably in source code form, so we can debug-step through it if needed
- 3rd party binaries (tools) for code generation
- a way to generate projects for compilation through a multiplatform/multi-tool project generator
- a way to peg certain 3rd party libraries at a given version
- a way to keep certain 3rd party libraries at the latest version
- a way to easily manage those 3rd party dependencies
The purpose of the scaffolding is to fulfill these requirements as much as possible.
After trying a few more or less (mostly less) successful strategies and tools, we settled on using Google Depot Tools gclient to manage several git repositories into a single project structure.
gclient is used by the Chromium project, so, it figures what is good for browser development with a huge disparate team, cannot be too bad for our 1-man-project.
A core (or root or meta or workspace) repository is the root of our project. Coming from a Perforce background, I fancy the term workspace, as it reflects actually on its usage.
The workspace contains the main .gclient configuration file that sets up the references for the main repositories to be included. Those repositories are supposed to stay at HEAD.
Example from BADASS below:
solutions = [
{
"name" : "src/core",
"url" : "git@github.dena.jp:pfsys-client/badass-core-src.git",
"deps_file" : "DEPS",
"managed" : True,
"custom_deps" : {},
},
{
"name" : "src/bactra",
"url" : "git@github.dena.jp:pfsys-client/badass-bactra-src.git",
"deps_file" : "DEPS",
"managed" : True,
"custom_deps" : {},
},
{
"name" : "scaffolding",
"url" : "git@github.dena.jp:pfsys-client/generic-scaffolding.git",
"deps_file" : "badass.DEPS",
"managed" : True,
"custom_deps" : {},
},
]
cache_dir = NoneBesides a number of source repositories, it references a (finally) generic scaffolding repository. (Generic because it's aimed at being re-used by other projects).
When invoked, gclient will fetch those projects's HEAD in order, and then iterate over their respective _depsfile_s.
In the case of BADASS and AVALON, the workspace repo also contains a few scripts, a makefile (as command hub) and a gclient named script, that redirects to the installed gclient binary (actually, a python script).
Source repositories contain source code, hence their name.
The structure is up to each repository, but we settled on having a scripts folder for, well, (make, project) scripts.
Each source repository can contain further DEPS files, that when referenced by the main .gclient configuration, allow to include further pegged or unpegged repositories. (Technically as child folders below the source repository's folder, but gclient allows free placement).
The single and generic scaffolding repository contains several DEPS files to be used by one of all referencing workspaces.
Those DEPS files refer to:
- 3rd party libraries included under
scaffolding/thirdparty - 3rd party tools included under
scaffolding/tools - compiled binary tools kept in
scaffolding/toos/bin/<platform>(darwin, windows, linux) - 1 makefile used for project generation
- several script folders for make and project scripts
We're living in a git world. Also, Perforce has its own quirks that git addresses differently (some might say better).
While subversion allows to include sub-projects that are pegged or not to a given revision, the way of managing them is beyond painful. Also, same point as for Perforce.
Git submodules allow to recursively reference external repositories, and also to update them. Submodules are always pegged to a given revision, and updating them requires a commit in the containing repository. Also, changes/updates are harder to track than a simple textfile. And anyone that ever tried merging submodules can tell why they're a rather bad idea.
Git subtree is a (custom) extension to git that allows to add external repositories into a containing repo. While the idea is laudable, the implementation just adds the external source into the containing repo and changes its history. This makes submitting changes to upstream repositories a work of luck.
Repo is the tool used for Android development. It is quite similar to gclient, but has less features. For example, it does not allow to peg certain repositories at a given revision/tag. Incidentally, I used repo at the beginning of BADASS's development and later transited towards gclient.
There are many ways to create and maintain projects for different IDEs and source code build tools, but each has its own quirks that can be hard to deal with.
CMake is a widely spread meta-build system, that can generate project files for a multitude of IDEs.
It comes with a pretty weird syntax, but offers a lot of functionality.
Because of its design, that consists of generating project files that invoke cmake again for building, it does not allow to generate/re-generate build files for several systems in the same folder/at the same time.
Welcome to UNIX development.
Honestly, I have no idea how to use autoconf. I just barely manage to run some configure scripts that generate makefiles for some open source projects.
Perforce JAM and all its variations rely on a quite hard syntax structure.
I think this project has died by now.
gyp used to be the build system used by Chromium. It is still used by Node in its variant node-gyp.
gypfiles are basically JSON files that indicate the files to build, and their options.
gyp generates ninjafiles for the ninja build system.
gn is now the build system used by Chromium.
It's more flexible than gyp and faster. It generates ninjafiles (hence its name).
Premake is a meta-build system that is built on Lua and uses luafiles as scripts to declare the projects to build, along with their respective settings. Premake is currently stuck between version 4.4 beta and 5.0 alpha, as its sole author is trying to keep with life. As such, the project is making very little progress. Premake luafiles are very easy to write and to maintain. As a meta-build system, premake allows to generate projects for a multitude of platforms and IDEs (and at the same time), among which:
- Visual Studio (in several versions)
- Xcode
- make
I have personally used a customized version of premake 4.3 (+ patches) while working at Gameloft.
[GENie] is a fork of Premake 4.4 beta that has received a lot of improvements. It is also the tool we settled on for our scaffolding. Among its many stabilizations and improvements, genie now supports even more build environments:
- ninja
- latest VS, with Clang/C2 and Clang/LLVM support as well
- emscripten
It is used by its author for his own projects, and we borrowed a few of his project settings for scaffolding. Switching from premake to genie was very straightforward, and maintaining the project files was a bliss until now.
to be continued.