black analysis.md

May 9, 2019

@sangaline commented early in April about the affect of black on our codebase.

Here's the current state of master as of 5/9/19:

$ cloc ${code_directories[*]}
     768 text files.
     767 unique files.
     121 files ignored.

1 error:
Unable to read:  xceligent-schema/

github.com/AlDanial/cloc v 1.82  T=1.77 s (392.2 files/s, 83216.7 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
JSON                            24              2              0          64047
Python                         442           9791           8249          47935
HTML                           167           1218             62          11325
YAML                            38            155             26           2907
SQL                             10             85            176            624
Markdown                         7            122              0            372
XML                              1              0              0            148
INI                              3              9              0             92
Bourne Shell                     3             20             12             72
-------------------------------------------------------------------------------
SUM:                           695          11402           8525         127522
-------------------------------------------------------------------------------

Black does not permit much customization, but we can specify reasonable defaults in pyproject.toml:

[tool.black]
target-version = ['py27']
skip-string-normalization = true
line-length = 100

Running black in "diff" mode against our entire codebase from root (tenantbase/) we get the following results:

$ echo Removed lines: $(black --diff --target-version py27 -S -l 100 . 2>/dev/null | grep '^-' | wc -l)
Removed lines: 11110
$ echo Added lines: $(black --diff --target-version py27 -S -l 100 . 2>/dev/null | grep '^+' | wc -l)
Added lines: 12480

These numbers are considerably less than Evan reported, likely because of the -l 100 option.

Running black in "run" mode against our entire codebase yields:

319 files reformatted, 153 files left unchanged, 13 files failed to reformat.

All thirteen failures were files in scripts app.

Nine failures occurred in date-stamped, one-off scripts. All these failures returned:

failed to parse source file with Python 3.6's builtin AST.

and involved used of print statement.

Four failures of regular utility scripts complained about stop using deprecated Python 2 syntax.

Special Formatting

Evan mentioned carefully formatted code which should not be disturbed for some reason. In these situations you can add # fmt: on and # fmt: off comment options:

[Black] doesn't reformat blocks that start with # fmt: off and end with # fmt: on.

Supported Dev Editors/IDEs

• PyCharm

• SublimeText 3

• Atom

• VIM

• VS Code

See Editor Integration for more details.

Version Control Integration

See Version Control Integration for details.

Implementation

Wait until we move to Python 3.x!

Ask devs to protect specially formatted code first with # fmt: off & # fmt: on.

Fix scripts files, typically usage of print "somthing" without parentheses. Change 13 files.

Set pyproject.toml file:

[tool.black]
target-version = ['py36']
skip-string-normalization = true
line-length = 100

Ask developers to add pre-commit hooks and/or IDE integration.

Run black .

Examine the results, merge with master.

Consider whether we need to use git-filter-branch.

Consider the benefit of black AST guarantee (--safe mode).