Skip to content

Instantly share code, notes, and snippets.

@T1T4N
Last active May 13, 2024 02:09
Show Gist options
  • Save T1T4N/f4d63a44476eb5c7046cc561cb8c7f77 to your computer and use it in GitHub Desktop.
Save T1T4N/f4d63a44476eb5c7046cc561cb8c7f77 to your computer and use it in GitHub Desktop.
Generate a JSON Compilation Database from an Xcode project

Introduction

A JSON compilation database is a very handy output format which is parsed and used by many development tools. Unfortunately for us Apple Developers, it is not straightforward to generate one from within Xcode, as it is (probably) not Apple's priority and therefore there is no toggle/switch/setting that can be easily enabled to get this information.

There is however a solution, thanks to Apple using Clang/LLVM as their main toolchain.

Implementation

The standard way to generate this with clang would be to use the -MJ flag and give it a file name that typically corresponds to the input file. Using this flag indirectly through Xcode is hard, given that we're not aware of all the other arguments when a compiler call is executed.

However, there is a second hidden/badly documented LLVM flag: -gen-cdb-fragment-path - it is implemented in terms of -MJ and has the same functionality, but it's argument in contrast is an output directory.

This allows us to collect all fragments from individual compiler executions in a central location, which greatly simplifies processing them.

xcconfig

OTHER_CFLAGS = $(inherited) -gen-cdb-fragment-path $(PROJECT_DIR)/CompilationDatabase

CMake

CMake has this functionality built in, represented by the variable CMAKE_EXPORT_COMPILE_COMMANDS, however this is only implemented for Makefile and Ninja generators and it is ignored for all others.

We can reuse this flag to set the required compiler flag for the generated Xcode project, which will in turn result in generated fragments. They will still need to be collected and processed separately (manually).

if(CMAKE_GENERATOR STREQUAL "Xcode")
	if(CMAKE_EXPORT_COMPILE_COMMANDS)
		set(CMAKE_XCODE_ATTRIBUTE_OTHER_CFLAGS  "$(inherited) -gen-cdb-fragment-path ${CMAKE_SOURCE_DIR}/CompilationDatabase")
	endif()
endif()

Processing

Once a build is executed with this flag, the output directory will contain a many JSON files, corresponding for each input source file that was compiled.

Each fragment on its own isn't valid JSON, but they can easily be combined into an array of objects which represents the actual compilation database.

Please check the script below for a reference implementation for combining the fragments into a compilation database.

References

  1. CLion - Generating a Compilation Database
  2. Clang - Generating a Compilation Database
  3. How to generate a JSON Compilation Database?
  4. JSON Compilation Database Format Specification
  5. LLVM: Add a new option to emit a fragment of a compilation database for each compilation
  6. LLVM: Add a new option to emit a fragment of a compilation database for each compilation 2
#!/usr/bin/env bash
# Global variables
readonly GXC_SCRIPT_PATH="$(test -L "${BASH_SOURCE[0]}" && readlink "${BASH_SOURCE[0]}" || echo "${BASH_SOURCE[0]}")"
readonly GXC_SCRIPT_DIR=$(cd "$(dirname "${GXC_SCRIPT_PATH}")"; pwd)
# brew install jq
: "${JQ:=$(command -v jq)}"
: "${OUTPUT_DIR:=$PWD}"
process_fragments() {
local -a files=()
for file in "$@"; do
[[ -f "$file" ]] || continue
# Fragments generated by clang have a comma before EOF
# If a fragment is stil invalid after removing it, it should be skipped
if sed -e '$s/,$//' "$file" | "$JQ" . > /dev/null; then
echo "Processing: $file" >&2
files+=("$file")
else
echo "Skipping: $file" >&2
fi
done
if (( ${#files[@]} == 0 )); then
echo "No input files found!" >&2
return
fi
sed -e '1s/^/[\'$'\n''/' -e '$s/,$/\'$'\n'']/' "${files[@]}" > "${OUTPUT_DIR}/compile_commands.json"
}
generate_database() {
echo "Running build command ..." >&2
if grep -q 'OTHER_CFLAGS' <<< "$*"; then
echo "OTHER_CLAGS detected in build command! This is unsupported as they will be overridden." >&2
return 1
fi
# xcrun xcodebuild ...
# https://reviews.llvm.org/D66555
# Note: Some tools require extra cflags to properly parse the compilation database, e.g. Infer
# OTHER_CFLAGS="\$(inherited) -DNS_FORMAT_ARGUMENT(A)= -D_Nullable_result=_Nullable -gen-cdb-fragment-path ${OUTPUT_DIR}/CompilationDatabase"
"$@" COMPILER_INDEX_STORE_ENABLE=NO OTHER_CFLAGS="\$(inherited) -gen-cdb-fragment-path ${OUTPUT_DIR}/CompilationDatabase"
process_fragments "${OUTPUT_DIR}/CompilationDatabase"/*.json
}
# main
if [[ "${BASH_SOURCE[0]}" != "$0" ]]; then
echo "Script is being sourced: ${GXC_SCRIPT_PATH}"
else
(( DEBUG == 0 )) || set -x
set -euo pipefail
if (( $# == 0 )); then
cat <<-EOF
Usage: $(basename "$0") <xcodebuild-command>
Environment variables:
JQ: Path to a jq binary
OUTPUT_DIR: Writable directory for storing the result. Default: (\$PWD) - $PWD
EOF
exit 1
fi
generate_database "$@"
fi
# Example invocation
# generate-xcode-compilation-database.sh xcodebuild build -project TestProject.xcodeproj -target TestTarget -configuration Debug
@rockingdice
Copy link

Does the project need to be able to compile first?

@T1T4N
Copy link
Author

T1T4N commented Sep 6, 2022

@rockingdice Yes, generate_database needs to invoke your xcodebuild build ... command, so a compilable project is a prerequisite.

@rockingdice
Copy link

@T1T4N thank you for your info.

@harshita29
Copy link

@rockingdice I tried but while running crypto I keep getting 'generate-xcode-compilation-database.sh: line 18: : command not found'. Can you help please

@harshita29
Copy link

Also can This scan swift files as well?

@T1T4N
Copy link
Author

T1T4N commented Mar 14, 2023

@harshita29 There are two potential causes for your issue.

  1. Your path is misconfigured and sed cannot be found (/usr/bin/sed).
  2. You don't have jq installed: brew install jq

The script is language agnostic, it will dump the commands used to execute the compiler and it doesn't matter which one. Unfortunately, this is a clang feature so Swift is not supported.

@nguyenvukhang
Copy link

@T1T4N does this still work for you on Xcode 14? I'm on macOS 13.2.1 so I can't downgrade to Xcode 13.4.1, and I was wondering if the Xcode version difference might be why it's not working for me.

Apart from using this script, I've also tried using -gen-cdb-fragment-path in both the CLI xcodebuild and in the Xcode's GUI settings, but neither produces a CompilationDatabase directory after building.

As a base test, I have tried using the -gen-cdb-fragment-path with clangd on a basic helloworld C file, and it does generate the json fragments as expected.

@T1T4N
Copy link
Author

T1T4N commented May 30, 2023

Hey @nguyenvukhang,
Thanks for your feedback. I just tested it on macOS 13.3.1 with Xcode 14.3 (14E222b) and it was working as expected.
The result that you're describing can happen if e.g. you have a pure, Swift only target, meaning no c/cpp/objc files as part of that target.
As the functionality in the script is passed along using OTHER_CFLAGS, no compilation database will be generated if clang wasn't ever executed.

@kxccc
Copy link

kxccc commented Jul 13, 2023

Thank you very much. I used xcpretty to generate the compile_commands.json file, but the file had bugs. However, using your method, the file is correct.

@oviano
Copy link

oviano commented Aug 27, 2023

For me, it doesn't seem to generate anything for CPP files, only C files.

I'm using CMake 3.26.1, macOS 13.4.1, Xcode 14.3.1.

@oviano
Copy link

oviano commented Aug 27, 2023

My problem seems to be that CMAKE_XCODE_ATTRIBUTE_OTHER_CFLAGS does not propagate through to the target's C++ flags in the Xcode project, only to the C flags.

This seems to be because the generated Xcode project is missing "${inherited}" inside the C++ flags section of the target settings.

If I add it manually, then things work, but that's no use, as I need CMake to generate it properly.

@T1T4N
Copy link
Author

T1T4N commented Aug 30, 2023

@oviano try setting CMAKE_XCODE_ATTRIBUTE_OTHER_CPLUSPLUSFLAGS with the same values as well.

@oviano
Copy link

oviano commented Aug 30, 2023

@oviano try setting CMAKE_XCODE_ATTRIBUTE_OTHER_CPLUSPLUSFLAGS with the same values as well.

I tried this and surprisingly it did not work either because it still does not set $(inherited) in the Other C++ Flags section of the Xcode properties, so while the above sets it for the project, the target does not inherit it. I presume it is a CMake bug.

However, I worked around it by manually adding $(inherited) to the CMAKE_CXX_FLAGS.

@zdl51go
Copy link

zdl51go commented Oct 25, 2023

I tried in a XCode cocoapods workspace, it seems only generate json files with main project.

@sideshowbarker
Copy link

sideshowbarker commented May 13, 2024

Another way to get the compile_commands.json file is to use https://github.com/xcpretty/xcpretty — like this:

xcodebuild -project path/to/.xcodeproj | \
    xcpretty -r json-compilation-database --output path/for/compile_commands.json

Or else, if you’re running a build from the Xcode GUI, you can press the Export… button in the Build tab to export the log, and then do:

cat Build\ Foo_202n-nn-nnTnn-nn-nn.txt | \
    xcpretty -r json-compilation-database --output path/for/compile_commands.json

I know somebody here mentioned that the “I used xcpretty to generate the compile_commands.json file, but the file had bugs” but I personally have never run into any bugs with it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment