Reproduce make-format errors and issues due to typed-ast-1.4.3 for python 3.9.10 Darwin on feast v0.17.0-13-gb4d12bd0

repro.md

Expected Behavior

make format and make lint should work and is able to format newly contributed code, not reformat the current code base.

Current Behavior

make format fails with ImportError undefined symbol _PyUnicode_DecodeUnicodeEscape due to issue with typed-ast 1.4.3 on python3.10 (python/typed_ast#169 , https://stackoverflow.com/questions/69912264/python-3-9-8-fails-using-black-and-importing-typed-ast-ast3).

(venv) ➜  feast git:(b4d12bd0) make format
# Sort
cd /Users/aht/src/feast/sdk/python; python -m isort feast/ tests/
Skipped 1 files
# Format
cd /Users/aht/src/feast/sdk/python; python -m black --target-version py37 feast tests
Traceback (most recent call last):
  File "/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/aht/src/feast/venv/lib/python3.9/site-packages/black.py", line 45, in <module>
    from typed_ast import ast3, ast27
  File "/Users/aht/src/feast/venv/lib/python3.9/site-packages/typed_ast/ast3.py", line 40, in <module>
    from typed_ast import _ast3
ImportError: dlopen(/Users/aht/src/feast/venv/lib/python3.9/site-packages/typed_ast/_ast3.cpython-39-darwin.so, 0x0002): symbol not found in flat namespace '__PyUnicode_DecodeUnicodeEscape'
make: *** [format-python] Error 1

Trying to work around by upgrading black cause reformatting 50 files.

Steps to reproduce

1. Starts with Python 3.9.10 (I'm on Mac OS and installed python via homebrew)
2. Clone feast, checkout feast git hash b4d12bd0444706566c2103c7f53f9efdbec32e2a
3. Follow Environment setup https://github.com/aht/feast/blob/master/CONTRIBUTING.md#environment-setup, i.e.

python -m venv venv/
source venv/bin/activate
pip install -e "sdk/python[dev]"

This ends up with typed-ast-1.4.3.

3. Run make format
4. Tried to work around by upgrading black to 21.10b0 per Stackoverflow comment. That is pip install "black==21.10b0"
5. Run make format again and got 50 files reformatted, 80 files left unchanged:

cd /Users/aht/src/feast/sdk/python; python -m black --target-version py37 feast tests
reformatted feast/feature.py
reformatted feast/diff/FcoDiff.py
reformatted feast/inference.py
reformatted feast/driver_test_data.py
reformatted feast/feature_view.py
reformatted feast/infra/offline_stores/bigquery_source.py
reformatted feast/data_source.py
reformatted feast/infra/offline_stores/offline_store.py
reformatted feast/infra/aws.py
reformatted feast/cli.py
reformatted feast/infra/offline_stores/file.py
reformatted feast/infra/offline_stores/bigquery.py
reformatted feast/infra/offline_stores/redshift_source.py
reformatted feast/infra/passthrough_provider.py
reformatted feast/infra/online_stores/sqlite.py
reformatted feast/infra/offline_stores/redshift.py
reformatted feast/infra/online_stores/dynamodb.py
reformatted feast/infra/online_stores/datastore.py
reformatted feast/request_feature_view.py
reformatted feast/infra/provider.py
reformatted feast/templates/aws/test.py
reformatted feast/infra/utils/aws_utils.py
reformatted feast/templates/gcp/test.py
reformatted feast/templates/local/example.py
reformatted feast/on_demand_feature_view.py
reformatted feast/utils.py
reformatted feast/repo_config.py
reformatted tests/benchmarks/test_benchmark_universal_online_retrieval.py
reformatted tests/example_repos/example_feature_repo_2.py
reformatted tests/example_repos/example_feature_repo_1.py
reformatted tests/doctest/test_all.py
reformatted tests/example_repos/example_feature_repo_with_inference.py
reformatted tests/example_repos/example_feature_repo_with_missing_bq_source.py
reformatted tests/conftest.py
reformatted feast/repo_operations.py
reformatted tests/foo_provider.py
reformatted feast/feature_store.py
reformatted tests/integration/e2e/test_usage_e2e.py
reformatted feast/usage.py
reformatted tests/integration/feature_repos/universal/data_sources/file.py
reformatted tests/integration/online_store/test_e2e_local.py
reformatted feast/type_map.py
reformatted tests/integration/registration/test_inference.py
reformatted tests/utils/data_source_utils.py
reformatted tests/unit/diff/test_fco_diff.py
reformatted tests/utils/online_read_write_test.py
reformatted tests/integration/registration/test_universal_types.py
reformatted tests/integration/registration/test_feature_store.py
reformatted tests/integration/registration/test_registry.py
reformatted tests/integration/offline_store/test_universal_historical_retrieval.py
All done! ✨ 🍰 ✨
50 files reformatted, 80 files left unchanged.

7. Pulling from master (9221624b2386ba0927a5ce78fe29dbb067752980) and then reinstall dependencies (pip install -e "sdk/python[dev]") does not solve the problem as it keeps typed-ast 1.4.3.

8. However, pulling from master (9221624b2386ba0927a5ce78fe29dbb067752980) and install deps from a clean venv will install typed-ast 1.5.2 and solve the issue.

Specifications

• Version: Happens for v0.17.0-13-gb4d12bd0 (output of git describe --tags)
• Platform: Darwin
• Subsystem: Python 3.9.10 (main, Jan 15 2022, 11:48:04) [Clang 13.0.0 (clang-1300.0.29.3)] on darwin

Possible Solution

Pin typed-ast to >=1.5.0.