Skip to content

Instantly share code, notes, and snippets.

@pdxjohnny
Last active June 16, 2020 17:12
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save pdxjohnny/5f3e0e86dd37b4b3e71a002c5b3a58e8 to your computer and use it in GitHub Desktop.
Save pdxjohnny/5f3e0e86dd37b4b3e71a002c5b3a58e8 to your computer and use it in GitHub Desktop.
DFFML sharedconfig
dffml service dev export -configloader yaml test:dataflow
configs:
db_lookup:
database:
config:
filename: test.db
plugin: sqlite
db_query:
database:
config:
filename: test.db
plugin: sqlite
definitions:
get_single_output:
name: get_single_output
primitive: map
get_single_spec:
name: get_single_spec
primitive: array
query_cols:
name: query_cols
primitive: Dict[str, str]
query_conditions:
name: query_conditions
primitive: Conditions
query_lookups:
name: query_lookups
primitive: Dict[str, Any]
query_table:
name: query_table
primitive: str
flow:
db_lookup:
inputs:
cols:
- seed
conditions:
- seed
table_name:
- seed
db_query:
inputs:
cols:
- seed
table_name:
- seed
get_single:
inputs:
spec:
- seed
linked: true
operations:
db_lookup:
inputs:
cols: query_cols
conditions: query_conditions
table_name: query_table
name: db_query_lookup
outputs:
lookups: query_lookups
stage: processing
db_query:
inputs:
cols: query_cols
table_name: query_table
name: db_query_create_table
outputs: {}
stage: processing
get_single:
inputs:
spec: get_single_spec
name: get_single
outputs:
output: get_single_output
stage: output
from dffml import *
sdb = SqliteDatabase(SqliteDatabaseConfig(filename="test.db"))
dataflow = DataFlow(
operations={
"db_query": db_query_create_table.op,
"db_lookup": db_query_lookup.op,
"get_single": GetSingle.imp.op,
},
configs={"db_query": "db", "db_lookup": "db",},
sharedconfigs={"db": DatabaseQueryConfig(database=sdb)},
)
from dffml import *
sdb = SqliteDatabase(SqliteDatabaseConfig(filename="test.db"))
dataflow = DataFlow(
operations={
"db_query": db_query_create_table.op,
"db_lookup": db_query_lookup.op,
"get_single": GetSingle.imp.op,
},
configs={"db_query": "db", "db_lookup": "db",},
sharedconfigs={"db": DatabaseQueryConfig(database=sdb)},
)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment