slopp

A modification of the LangChain SQL Q&A tutorial https://python.langchain.com/docs/tutorials/sql_qa/.

The changes are:

• uses Pydantic to type the state and inputs/outputs
• uses duckDB on the palmerpenguin dataset
• uses a Nvidia NIM for the LLM
• instead of a sequence write_query -> run_query -> gen_answer, this graph adds a LLM that checks the write_query output for validity and to see if it answers the question, leading to a more dynamic graph that looks like this:

slopp

See https://loppsided.blog/posts/2024-12-27-a-basic-tool-calling-agent/

slopp

See https://loppsided.blog/posts/2024-12-26-a-rag-to-chat-with-your-favorite-blogger/

slopp

This simple streamlit app uses the Google Maps and Places API, along with a hosted Nvidia NIM wrapper of the Llama model, to help you find coffee shops near an address.

To run

1. Install dependencies

slopp

Torch Experiment

Goal

• First attempt at using Torch for some type of "deep" learning
• Take advantage of modal to access serverless Python compute, including GPUs

Approach

slopp

This example shows a skeleton for how to build a Dagster project that extracts tables from SQL Server, stores the extract as a CSV in GCS, and then uploads the GCS extract to BigQuery.

The actual extract and load logic is omitted. But the purpose of this project is to show how such a pipeline can be represented in Dagster assets.

First, a single pipeline for one table is created. This is demonstrated in the file dagster_mock_one_table.py. To run this example:

1. Create a Python virtual environment and then run:

pip install dagster dagster-webserver

slopp

slopp

from typing import Any
from dagster import ConfigurableResource, ConfigurableIOManager, InputContext, OutputContext, asset, Definitions, ResourceDependency, EnvVar
from pydantic import Field

# https://docs.dagster.io/concepts/resources#resources-that-depend-on-other-resources

class myResource(ConfigurableResource):
    username: str = Field(description="the username")
    password: str = Field(description="the password")

slopp

import os

from dagster import define_asset_job, load_assets_from_package_module, repository, with_resources, op, job, ScheduleDefinition
from my_dagster_project import assets
from datadog_api_client import ApiClient, Configuration
from datadog_api_client.v2.api.metrics_api import MetricsApi
from datadog_api_client.v2.model.metric_intake_type import MetricIntakeType
from datadog_api_client.v2.model.metric_payload import MetricPayload
from datadog_api_client.v2.model.metric_point import MetricPoint
from datetime import datetime

slopp

Dynamic Pipeline

This example shows the psuedo-code for a Dagster pipeline that:

1. Accepts the path to a raw dataset as a string
2. Runs a step to break the raw dataset into partitions
3. For each partition, the pipeline runs a series of two processing steps. Each processing step is a call out to a Docker container to run supplying the partition key as an input argument. The partitions are run together in parallel before being collected in a final processing step that operates on all the partitions.

To run the pipeline: