Skip to content

Instantly share code, notes, and snippets.

View anteverse's full-sized avatar
🐍

Aurelien Didier anteverse

🐍
View GitHub Profile
@anteverse
anteverse / lint-console.log
Created November 3, 2022 14:31
Log after lint execution
15:47:14.67 [INFO] Completed: Building pylint_runner.pex
15:47:16.82 [ERROR] Completed: Lint using Pylint - pylint failed (exit code 16).
************* Module flows.example.tasks.dummy
flows/example/tasks/dummy.py:5:0: C9002: The task name is not suffixed by `_task` (invalid-task-definition-name)
@anteverse
anteverse / task.py
Created November 3, 2022 13:57
Example of task declaration that does not match our conventions
from prefect import task
@task(name="Dummy task")
def dummy(items: list[str]) -> list[str]:
...
@anteverse
anteverse / pyproject.toml
Created November 3, 2022 13:53
Example of pyproject.toml configuration
[tool.pylint.'MASTER']
load-plugins = [
"tasks",
]
@anteverse
anteverse / pants.toml
Last active December 1, 2022 15:01
Example of pants.toml configuration for a project of plugins
[GLOBAL]
pants_version = "2.13.0"
backend_packages = [
"pants.backend.python.lint.pylint",
]
[python]
resolves = { python-default = "pants.lock", pylint-plugins = "pylint_plugins/pants-pylint-plugins.lock" }
enable_resolves = true
@anteverse
anteverse / pants.toml
Created November 3, 2022 13:45
Example of pants.toml configuration for a project of plugins
[GLOBAL]
pants_version = "2.13.0"
backend_packages = [
"pants.backend.python.lint.pylint",
]
[python]
resolves = { pylint-plugins = "pylint_plugins/pants-pylint-plugins.lock" }
enable_resolves = true
@anteverse
anteverse / BUILD
Created November 3, 2022 13:42
Example of BUILD file for a pylint plugins project
python_sources(
name="pylint-plugins-sources",
sources=["**/*.py", "!**/test_*.py"],
resolve="pylint-plugins",
)
@anteverse
anteverse / task_name_checker.py
Created November 3, 2022 13:39
Example of Pylint custom checker that verifies tasks definition names
class FlowObjectNameChecker(checkers.BaseChecker):
__implements__ = interfaces.IAstroidChecker
name = "invalid-flow-object-definition-name"
msgs = {
"C9002": (
"The task name is not suffixed by `_task`",
"invalid-task-definition-name",
"Emitted when a flow function definition is not suffixed by `_task` as per the naming convention requires",
@anteverse
anteverse / uploads_in_threads.py
Last active October 31, 2021 09:39
Put uploads in threads
zip_obj = s3_resource.Object("path/to/key.zip")
buffer = BytesIO(zip_obj.get()["Body"].read())
with ZipFile(buffer) as zipfile:
with TransferManager(
config=transfer_config,
client=boto3.client("s3"),
) as transfer_manager:
def _upload(file_name: str):
dest_key = build_destination_key(file_name)
transfer_manager.upload(
@anteverse
anteverse / tm_config.py
Last active October 31, 2021 09:14
S3 TransferManager and TransferConfig
transfer_config = TransferConfig(
max_submission_concurrency=threads_number,
)
transfer_manager = TransferManager(
config=transfer_config,
client=boto3.client("s3"),
)
@anteverse
anteverse / dl_and_up_with_buffer.py
Last active October 30, 2021 16:51
Remove the disk usage
zip_obj = s3_resource.Object("path/to/key.zip")
buffer = BytesIO(zip_obj.get()["Body"].read())
with ZipFile(buffer) as zipfile:
for file_name in zipfile.namelist():
s3_client.upload_fileobj(
fileobj=zipfile.open(file_name),
Bucket="my_bucket",
Key=f"some_prefix/{file_name}",
)