Skip to content

Instantly share code, notes, and snippets.

@bsilver8192
Created March 17, 2016 16:12
Show Gist options
  • Save bsilver8192/0115ee5d040bb601e3b7 to your computer and use it in GitHub Desktop.
Save bsilver8192/0115ee5d040bb601e3b7 to your computer and use it in GitHub Desktop.
Basics of generating a compile_commands.json file with Bazel
py_binary(
name = 'generate_compile_command',
srcs = [
'generate_compile_command.py',
],
deps = [
'//third_party/bazel:extra_actions_proto_py',
],
)
action_listener(
name = 'generate_compile_commands_listener',
visibility = ['//visibility:public'],
mnemonics = [
'CppCompile',
],
extra_actions = [':generate_compile_commands_action'],
)
extra_action(
name = 'generate_compile_commands_action',
tools = [
':generate_compile_command',
],
out_templates = [
'$(ACTION_ID)_compile_command',
],
cmd = '$(location :generate_compile_command) $(EXTRA_ACTION_FILE)' +
' $(output $(ACTION_ID)_compile_command)',
)
# This is the implementation of a Bazel extra_action which genenerates
# _compile_command files for generate_compile_commands.py to consume.
import sys
import third_party.bazel.protos.extra_actions_base_pb2 as extra_actions_base_pb2
def _get_cpp_command(cpp_compile_info):
compiler = cpp_compile_info.tool
options = ' '.join(cpp_compile_info.compiler_option)
source = cpp_compile_info.source_file
output = cpp_compile_info.output_file
return '%s %s -c %s -o %s' % (compiler, options, source, output), source
def main(argv):
action = extra_actions_base_pb2.ExtraActionInfo()
with open(argv[1], 'rb') as f:
action.MergeFromString(f.read())
command, source_file = _get_cpp_command(
action.Extensions[extra_actions_base_pb2.CppCompileInfo.cpp_compile_info])
with open(argv[2], 'w') as f:
f.write(command)
f.write('\0')
f.write(source_file)
if __name__ == '__main__':
sys.exit(main(sys.argv))
#!/usr/bin/python3
# This reads the _compile_command files :generate_compile_commands_action
# generates a outputs a compile_commands.json file at the top of the source
# tree for things like clang-tidy to read.
# Overall usage directions: run bazel with
# --experimental_action_listener=//tools/actions:generate_compile_commands_listener
# for all the files you want to use clang-tidy with and then run this script.
# Afer that, `clang-tidy build_tests/gflags.cc` should work.
import sys
import pathlib
import os.path
import subprocess
'''
Args:
path: The pathlib.Path to _compile_command file.
command_directory: The directory commands are run from.
Returns a string to stick in compile_commands.json.
'''
def _get_command(path, command_directory):
with path.open('r') as f:
contents = f.read().split('\0')
if len(contents) != 2:
# Old/incomplete file or something; silently ignore it.
return None
return '''{
"directory": "%s",
"command": "%s",
"file": "%s",
},''' % (command_directory, contents[0].replace('"', '\\"'), contents[1])
'''
Args:
path: A directory pathlib.Path to look for _compile_command files under.
command_directory: The directory commands are run from.
Yields strings to stick in compile_commands.json.
'''
def _get_compile_commands(path, command_directory):
for f in path.iterdir():
if f.is_dir():
yield from _get_compile_commands(f, command_directory)
elif f.name.endswith('_compile_command'):
command = _get_command(f, command_directory)
if command:
yield command
def main(argv):
source_path = os.path.join(os.path.dirname(__file__), '../..')
action_outs = os.path.join(source_path,
'bazel-bin/../extra_actions',
'tools/actions/generate_compile_commands_action')
command_directory = subprocess.check_output(
('bazel', 'info', 'execution_root'),
cwd=source_path).decode('utf-8').rstrip()
commands = _get_compile_commands(pathlib.Path(action_outs), command_directory)
with open(os.path.join(source_path, 'compile_commands.json'), 'w') as f:
f.write('[')
for command in commands:
f.write(command)
f.write(']')
if __name__ == '__main__':
sys.exit(main(sys.argv))
@mmlac
Copy link

mmlac commented Dec 11, 2016

I had quite some trouble to understand where which file goes and what to do with them, so here is how I got it to work: (under Linux / Arch)
(and see the file-tree at the bottom to make it clearer where which file goes)

in your WORKSPACE root, create /tools/actions folder(s)
create all files from above in there.

download / create the file from (github bazel source) bazel/src/main/protobuf/extra_actions_base.proto (anywhere really, we don't need the proto file afterwards)
and run protoc extra_actions_base.proto --python_out=.
(you need to have protocol buffers installed on your system)

create a new folder (relative to thoe WORKSPACE root) /third_party/bazel/protos and move the Python file we just generated (extra_actions_base_pb2.py) there.
in /third_party/bazel (not /protos) we now create a BUILD file with the following content:

licenses(["notice"])

py_library(
    name = "extra_actions_proto_py",
    srcs = ["protos/extra_actions_base_pb2.py"],
    visibility = ["//visibility:public"],
)

now make sure your python has google.protobuf installed ( [sudo] pip install protobuf )

now if you run the bazel build command, i.e.
bazel build --experimental_action_listener=//tools/actions:generate_compile_commands_listener main:hello-world
it should work, i.e. not throw any errors.

Then do (from WORKSPACE root) cd tools/actions and from this folder you run the _json script:
python generate_compile_commands_json.py

Now there should be a compile_commands.json in your WORKSPACE root folder. Congratulations, it worked! :)

Below a file tree to make it easier to see where all the files are. The project is just the bazel C++ tutorial.

.
├── compile_commands.json
├── extra_actions_base.proto  <- This can be anywhere and can be deleted as soon as we have the .py file
├── gtest.BUILD
├── lib
│   ├── BUILD
│   ├── hello-greet.cc
│   └── hello-greet.h
├── main
│   ├── BUILD
│   ├── hello-time.cc
│   ├── hello-time.h
│   ├── hello-world.cc
├── test
│   ├── BUILD
│   └── hello-test.cc
├── third_party
│   └── bazel
│       ├── BUILD
│       └── protos
│           └── extra_actions_base_pb2.py
├── tools
│   └── actions
│       ├── BUILD
│       ├── generate_compile_command.py
│       └── generate_compile_commands_json.py
└── WORKSPACE

Hope it helps someone ¯\(ツ)

@amosbird
Copy link

@mmlac Thanks.

@abergmeier
Copy link

@vincent-picaud
Copy link

vincent-picaud commented Jun 26, 2017

@bsilver8192, @mmlac Many thanks for these explanations.
I embedded the codes in a bash script for ease of use. This is available in my GitHub repo.

@siddharthab
Copy link

Self-promoting plug here: I wrote Bazel rules to generate the compilation database. It is not perfect but will work for most cases, and requires much less setup, as simple as copying a file and running a script. And will work even if the code does not compile.

https://github.com/grailbio/bazel-compilation-database

@shifakhan4892
Copy link

I had quite some trouble to understand where which file goes and what to do with them, so here is how I got it to work: (under Linux / Arch)
(and see the file-tree at the bottom to make it clearer where which file goes)

in your WORKSPACE root, create /tools/actions folder(s)
create all files from above in there.

download / create the file from (github bazel source) bazel/src/main/protobuf/extra_actions_base.proto (anywhere really, we don't need the proto file afterwards)
and run protoc extra_actions_base.proto --python_out=.
(you need to have protocol buffers installed on your system)

create a new folder (relative to thoe WORKSPACE root) /third_party/bazel/protos and move the Python file we just generated (extra_actions_base_pb2.py) there.
in /third_party/bazel (not /protos) we now create a BUILD file with the following content:

licenses(["notice"])

py_library(
    name = "extra_actions_proto_py",
    srcs = ["protos/extra_actions_base_pb2.py"],
    visibility = ["//visibility:public"],
)

now make sure your python has google.protobuf installed ( [sudo] pip install protobuf )

now if you run the bazel build command, i.e.
bazel build --experimental_action_listener=//tools/actions:generate_compile_commands_listener main:hello-world
it should work, i.e. not throw any errors.

Then do (from WORKSPACE root) cd tools/actions and from this folder you run the _json script:
python generate_compile_commands_json.py

Now there should be a compile_commands.json in your WORKSPACE root folder. Congratulations, it worked! :)

Below a file tree to make it easier to see where all the files are. The project is just the bazel C++ tutorial.

.
├── compile_commands.json
├── extra_actions_base.proto  <- This can be anywhere and can be deleted as soon as we have the .py file
├── gtest.BUILD
├── lib
│   ├── BUILD
│   ├── hello-greet.cc
│   └── hello-greet.h
├── main
│   ├── BUILD
│   ├── hello-time.cc
│   ├── hello-time.h
│   ├── hello-world.cc
├── test
│   ├── BUILD
│   └── hello-test.cc
├── third_party
│   └── bazel
│       ├── BUILD
│       └── protos
│           └── extra_actions_base_pb2.py
├── tools
│   └── actions
│       ├── BUILD
│       ├── generate_compile_command.py
│       └── generate_compile_commands_json.py
└── WORKSPACE

Hope it helps someone ¯_(ツ)_/¯

Hi @mmlac !
I am trying to follow this. My system uses Python 2.7 version and hence does not support "yield from" command in generate_compile_commands.json .
Is there an alternate to that line of code which will be compatible on python 2.7.

Thank you ! Would really appreciate your help.

@kdungs
Copy link

kdungs commented Dec 2, 2020

Unfortunately, the example code produces compile commands that confuse (modern?) compilers. Specifically line 13 in generate_compile_command.py results in duplicate "-o" and "-c" options which both gcc (version 10.2.0) and clang (version 11.0.0) complain about.

Also, the result is not valid JSON but that doesn't seem to bother ccls or clangd 😄

Here's the code I'm using right now. I might refactor this into a dedicated repository at some point 😄
https://github.com/kdungs/adventofcode/tree/main/2020/tools/actions


My system uses Python 2.7 version and hence does not support "yield from"

IMHO you should use Python 3 🙈 But until then,

for foo in bar:
    yield foo

should do the trick.

@bsilver8192
Copy link
Author

For anybody else looking at this: I (the author of the gist) have since moved to https://github.com/grailbio/bazel-compilation-database. The aspect-based approach is definitively better IMHO with the improved CC Starlark API since I threw this together. It has better fidelity and it's less brittle. You can see a little fix from me at grailbio/bazel-compilation-database@9682280, and then it works great on my codebases.

@kdungs
Copy link

kdungs commented Dec 3, 2020

Thanks a lot for the clarification 😄

It's good to have this here for posterity since this gist is referenced in a lot of places and one of the first results in any search related to Bazel and compile commands.

Happy holidays!

@cpsauer
Copy link

cpsauer commented Nov 19, 2021

For anyone landing here, we'd highly recommend using https://github.com/hedronvision/bazel-compile-commands-extractor, just released.

[It's fast and less brittle thanks to an aquery-based approach that directly asks Bazel what build commands it would run. The grail bio/bazel-compilation-database creator had some nice things to say about it and passing the torch. (See here for more)]

@zeroxia
Copy link

zeroxia commented Mar 11, 2022

This works great with bazel 0.5.3 on Ubuntu 18.04.

Yesterday I tried Ubuntu 16.04, to install the Python 2.7's package protobuf, you need to do it as follows:

sudo apt install python-pip
sudo pip install protobuf=3.17.3

Otherwise the default installed protobuf with version 3.19+ is not working.

@tsr-boxbot
Copy link

@bsilver8192 could you add a license to this gist? (preferably MIT)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment