Skip to content

Instantly share code, notes, and snippets.

@cupdike
Created June 13, 2018 16:39
Show Gist options
  • Star 4 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save cupdike/935cec4d82db45715876e3d4b8b2f758 to your computer and use it in GitHub Desktop.
Save cupdike/935cec4d82db45715876e3d4b8b2f758 to your computer and use it in GitHub Desktop.
Airflow Beeline Connection Using Kerberos via CLI
### There aren't many good examples of how to do this when also using kerberos
(venv) [airflow@cray01 dags]$ airflow connections --add \
--conn_id beeline_hive \
--conn_type 'beeline' \
--conn_host 'myserver.mydomain.com' \
--conn_port 10000 \
--conn_extra '{"use_beeline": true, "auth":"kerberos;principal=mysvcname/myservicehost@MYDOMAIN.COM;"}'
### Then, a sample DAG to use it
import time
import airflow
from airflow.operators.hive_operator import HiveOperator
from airflow.models import DAG
args = {
'owner': 'airflow',
'start_date': airflow.utils.dates.days_ago(2)
}
dag = DAG(
dag_id='HelloHiveDag',
default_args=args, # supplied to all operators, accessible directly as properties on the task instance
schedule_interval=None) # only run via cli
t1 = HiveOperator(
hive_cli_conn_id='beeline_hive_bd',
task_id='HelloHiveTask',
hql= """
USE DEFAULT;
CREATE TABLE IF NOT EXISTS employee (eid int, name String, salary String, destination String)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '\t'
LINES TERMINATED BY '\n'
STORED AS TEXTFILE;
""",
dag=dag)
@nicks85
Copy link

nicks85 commented Aug 17, 2021

Just wanted to ask you, which plugin you downloaded to get the --conn_type 'beeline' appear in the drop down list, I am facing the error while using the almost similar code with Beeline connection

@cupdike
Copy link
Author

cupdike commented Aug 17, 2021

My recollection is that you add the connection via the CLI and then it can be accessed via the UI.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment