This Python script is designed to perform several operations related to text and data processing using Minio for object storage and the Transformers library for text tokenization. The script is organized into various functions, each responsible for a specific task.
-
init_logger(log_level: str)
: Initializes the logging mechanism with a given log level. -
check_env()
: Checks for the existence of required environment variables for Minio client setup.