This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
15:47:14.67 [INFO] Completed: Building pylint_runner.pex | |
15:47:16.82 [ERROR] Completed: Lint using Pylint - pylint failed (exit code 16). | |
************* Module flows.example.tasks.dummy | |
flows/example/tasks/dummy.py:5:0: C9002: The task name is not suffixed by `_task` (invalid-task-definition-name) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
from prefect import task | |
@task(name="Dummy task") | |
def dummy(items: list[str]) -> list[str]: | |
... |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
[tool.pylint.'MASTER'] | |
load-plugins = [ | |
"tasks", | |
] |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
[GLOBAL] | |
pants_version = "2.13.0" | |
backend_packages = [ | |
"pants.backend.python.lint.pylint", | |
] | |
[python] | |
resolves = { python-default = "pants.lock", pylint-plugins = "pylint_plugins/pants-pylint-plugins.lock" } | |
enable_resolves = true |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
[GLOBAL] | |
pants_version = "2.13.0" | |
backend_packages = [ | |
"pants.backend.python.lint.pylint", | |
] | |
[python] | |
resolves = { pylint-plugins = "pylint_plugins/pants-pylint-plugins.lock" } | |
enable_resolves = true |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
python_sources( | |
name="pylint-plugins-sources", | |
sources=["**/*.py", "!**/test_*.py"], | |
resolve="pylint-plugins", | |
) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
class FlowObjectNameChecker(checkers.BaseChecker): | |
__implements__ = interfaces.IAstroidChecker | |
name = "invalid-flow-object-definition-name" | |
msgs = { | |
"C9002": ( | |
"The task name is not suffixed by `_task`", | |
"invalid-task-definition-name", | |
"Emitted when a flow function definition is not suffixed by `_task` as per the naming convention requires", |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
zip_obj = s3_resource.Object("path/to/key.zip") | |
buffer = BytesIO(zip_obj.get()["Body"].read()) | |
with ZipFile(buffer) as zipfile: | |
with TransferManager( | |
config=transfer_config, | |
client=boto3.client("s3"), | |
) as transfer_manager: | |
def _upload(file_name: str): | |
dest_key = build_destination_key(file_name) | |
transfer_manager.upload( |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
transfer_config = TransferConfig( | |
max_submission_concurrency=threads_number, | |
) | |
transfer_manager = TransferManager( | |
config=transfer_config, | |
client=boto3.client("s3"), | |
) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
zip_obj = s3_resource.Object("path/to/key.zip") | |
buffer = BytesIO(zip_obj.get()["Body"].read()) | |
with ZipFile(buffer) as zipfile: | |
for file_name in zipfile.namelist(): | |
s3_client.upload_fileobj( | |
fileobj=zipfile.open(file_name), | |
Bucket="my_bucket", | |
Key=f"some_prefix/{file_name}", | |
) |
NewerOlder