Skip to content

Instantly share code, notes, and snippets.

View pspeter's full-sized avatar
🖊️
Blogging

pyter pspeter

🖊️
Blogging
View GitHub Profile
@pspeter
pspeter / migrate_airflow_logs_v2_3.py
Created May 16, 2022 11:57
Migrate airflow logs from v2.2.x to v2.3.x folder structure
import pandas as pd
from pathlib import Path
from airflow.settings import Session
log_dir = Path("/opt/airflow/logs")
session = Session()
tis_iter = session.execute("SELECT * FROM task_instance;")
tis = pd.DataFrame(tis_iter, columns=["task_id", "dag_id", "date1", "date2", "num0", "status", "num1", "hash", "user", "num2", "pool", "str1", "num3", "operator", "date3", "num4", "num5", "mem_addr", "num6", "num7", "longid", "None1", "None2", "None3", "None4", "scheduled_task_id", "num8"])