Generate a JSON Compilation Database from an Xcode project

generate-xcode-compilation-database.md

Introduction

A JSON compilation database is a very handy output format which is parsed and used by many development tools. Unfortunately for us Apple Developers, it is not straightforward to generate one from within Xcode, as it is (probably) not Apple's priority and therefore there is no toggle/switch/setting that can be easily enabled to get this information.

There is however a solution, thanks to Apple using Clang/LLVM as their main toolchain.

Implementation

The standard way to generate this with clang would be to use the -MJ flag and give it a file name that typically corresponds to the input file. Using this flag indirectly through Xcode is hard, given that we're not aware of all the other arguments when a compiler call is executed.

However, there is a second hidden/badly documented LLVM flag: -gen-cdb-fragment-path - it is implemented in terms of -MJ and has the same functionality, but it's argument in contrast is an output directory.

This allows us to collect all fragments from individual compiler executions in a central location, which greatly simplifies processing them.

xcconfig

OTHER_CFLAGS = $(inherited) -gen-cdb-fragment-path $(PROJECT_DIR)/CompilationDatabase

CMake

CMake has this functionality built in, represented by the variable CMAKE_EXPORT_COMPILE_COMMANDS, however this is only implemented for Makefile and Ninja generators and it is ignored for all others.

We can reuse this flag to set the required compiler flag for the generated Xcode project, which will in turn result in generated fragments. They will still need to be collected and processed separately (manually).

if(CMAKE_GENERATOR STREQUAL "Xcode")
	if(CMAKE_EXPORT_COMPILE_COMMANDS)
		set(CMAKE_XCODE_ATTRIBUTE_OTHER_CFLAGS  "$(inherited) -gen-cdb-fragment-path ${CMAKE_SOURCE_DIR}/CompilationDatabase")
	endif()
endif()

Processing

Once a build is executed with this flag, the output directory will contain a many JSON files, corresponding for each input source file that was compiled.

Each fragment on its own isn't valid JSON, but they can easily be combined into an array of objects which represents the actual compilation database.

Please check the script below for a reference implementation for combining the fragments into a compilation database.

References

1. CLion - Generating a Compilation Database
2. Clang - Generating a Compilation Database
3. How to generate a JSON Compilation Database?
4. JSON Compilation Database Format Specification
5. LLVM: Add a new option to emit a fragment of a compilation database for each compilation
6. LLVM: Add a new option to emit a fragment of a compilation database for each compilation 2

generate-xcode-compilation-database.sh

#!/usr/bin/env bash

# Global variables
readonly GXC_SCRIPT_PATH="$(test -L "${BASH_SOURCE[0]}" && readlink "${BASH_SOURCE[0]}" || echo "${BASH_SOURCE[0]}")"
readonly GXC_SCRIPT_DIR=$(cd "$(dirname "${GXC_SCRIPT_PATH}")"; pwd)

# brew install jq
: "${JQ:=$(command -v jq)}"
: "${OUTPUT_DIR:=$PWD}"

process_fragments() {
	local -a files=()
	for file in "$@"; do
		[[ -f "$file" ]] || continue

		# Fragments generated by clang have a comma before EOF
		# If a fragment is stil invalid after removing it, it should be skipped
		if sed -e '$s/,$//' "$file" | "$JQ" . > /dev/null; then
			echo "Processing: $file" >&2
			files+=("$file")
		else
			echo "Skipping: $file" >&2
		fi
	done

	if (( ${#files[@]} == 0 )); then
		echo "No input files found!" >&2
		return
	fi

	sed -e '1s/^/[\'$'\n''/' -e '$s/,$/\'$'\n'']/' "${files[@]}" > "${OUTPUT_DIR}/compile_commands.json"
}

generate_database() {
	echo "Running build command ..." >&2
	if grep -q 'OTHER_CFLAGS' <<< "$*"; then
		echo "OTHER_CLAGS detected in build command! This is unsupported as they will be overridden." >&2
		return 1
	fi

	# xcrun xcodebuild ...
	# https://reviews.llvm.org/D66555
	
	# Note: Some tools require extra cflags to properly parse the compilation database, e.g. Infer
	# OTHER_CFLAGS="\$(inherited) -DNS_FORMAT_ARGUMENT(A)= -D_Nullable_result=_Nullable -gen-cdb-fragment-path ${OUTPUT_DIR}/CompilationDatabase"
	"$@" COMPILER_INDEX_STORE_ENABLE=NO OTHER_CFLAGS="\$(inherited) -gen-cdb-fragment-path ${OUTPUT_DIR}/CompilationDatabase"
	process_fragments "${OUTPUT_DIR}/CompilationDatabase"/*.json
}

# main
if [[ "${BASH_SOURCE[0]}" != "$0" ]]; then
	echo "Script is being sourced: ${GXC_SCRIPT_PATH}"
else
	(( DEBUG == 0 )) || set -x
	set -euo pipefail

	if (( $# == 0 )); then
		cat <<-EOF
		Usage: $(basename "$0") <xcodebuild-command>

		Environment variables:
		JQ: Path to a jq binary
		OUTPUT_DIR: Writable directory for storing the result. Default: (\$PWD) - $PWD
		EOF

		exit 1
	fi

	generate_database "$@"
fi

# Example invocation
# generate-xcode-compilation-database.sh xcodebuild build -project TestProject.xcodeproj -target TestTarget -configuration Debug

Does the project need to be able to compile first?

@rockingdice Yes, generate_database needs to invoke your xcodebuild build ... command, so a compilable project is a prerequisite.

@T1T4N thank you for your info.

@rockingdice I tried but while running crypto I keep getting 'generate-xcode-compilation-database.sh: line 18: : command not found'. Can you help please

Also can This scan swift files as well?

@harshita29 There are two potential causes for your issue.

1. Your path is misconfigured and sed cannot be found (/usr/bin/sed).
2. You don't have jq installed: brew install jq

The script is language agnostic, it will dump the commands used to execute the compiler and it doesn't matter which one. Unfortunately, this is a clang feature so Swift is not supported.

@T1T4N does this still work for you on Xcode 14? I'm on macOS 13.2.1 so I can't downgrade to Xcode 13.4.1, and I was wondering if the Xcode version difference might be why it's not working for me.

Apart from using this script, I've also tried using -gen-cdb-fragment-path in both the CLI xcodebuild and in the Xcode's GUI settings, but neither produces a CompilationDatabase directory after building.

As a base test, I have tried using the -gen-cdb-fragment-path with clangd on a basic helloworld C file, and it does generate the json fragments as expected.

Hey @nguyenvukhang,
Thanks for your feedback. I just tested it on macOS 13.3.1 with Xcode 14.3 (14E222b) and it was working as expected.
The result that you're describing can happen if e.g. you have a pure, Swift only target, meaning no c/cpp/objc files as part of that target.
As the functionality in the script is passed along using OTHER_CFLAGS, no compilation database will be generated if clang wasn't ever executed.

Thank you very much. I used xcpretty to generate the compile_commands.json file, but the file had bugs. However, using your method, the file is correct.

For me, it doesn't seem to generate anything for CPP files, only C files.

I'm using CMake 3.26.1, macOS 13.4.1, Xcode 14.3.1.

My problem seems to be that CMAKE_XCODE_ATTRIBUTE_OTHER_CFLAGS does not propagate through to the target's C++ flags in the Xcode project, only to the C flags.

This seems to be because the generated Xcode project is missing "${inherited}" inside the C++ flags section of the target settings.

If I add it manually, then things work, but that's no use, as I need CMake to generate it properly.

@oviano try setting CMAKE_XCODE_ATTRIBUTE_OTHER_CPLUSPLUSFLAGS with the same values as well.

@oviano try setting CMAKE_XCODE_ATTRIBUTE_OTHER_CPLUSPLUSFLAGS with the same values as well.

I tried this and surprisingly it did not work either because it still does not set $(inherited) in the Other C++ Flags section of the Xcode properties, so while the above sets it for the project, the target does not inherit it. I presume it is a CMake bug.

However, I worked around it by manually adding $(inherited) to the CMAKE_CXX_FLAGS.

I tried in a XCode cocoapods workspace, it seems only generate json files with main project.