Skip to content

Instantly share code, notes, and snippets.

@lemonteaa
Created June 24, 2023 09:33
Show Gist options
  • Save lemonteaa/a2b29fa6383ebc27e82eea9a2067fdbf to your computer and use it in GitHub Desktop.
Save lemonteaa/a2b29fa6383ebc27e82eea9a2067fdbf to your computer and use it in GitHub Desktop.

Prompt IDE Idea

Goal and Non-goal

Not to become an enterprisy tool - so probably not fancy collaboration or auto-eval by LLM itself. Also focus on locally hosted open source LLM models and use cases like tool-using (retrieval augmented chatbot), document ingest, agent.

Single Prompt Editing

Basic Features

  • Automatic output parsing and logging
  • Prompt version control
    • Variations - try concurrently
  • Experiment - models and params... dataset?
    • Allows rerun/sample multiple outputs to ensure robustness/reliability
  • Show rendered prompt
  • Set default choice for inputs

How about these (in other products):

  • Compare results side by side
  • Prompt Version diff
  • Generate dataset from chat (interactive discovery)

Template Management

  • Input values
    • Rich input data structure with editor UI
  • Components/inheritance
    • Version pinning and or specify multiple version to instantiate experiment
  • Quick extract
  • Guidance
    • Continuation

Experiment

  • Union of sets
  • Each set, each type of thing
    • Can be "Try all", "Constant", or "Try subset"

Workflow

  • Draft initial prompt
    • Can extract logical units
    • Structured inputs
  • First manual run
  • Try variations
    • Branch
    • Then "Run Experiment"

Later during review:

  • Can view past experiments
    • Experiment pinned to specific version of prompt
    • Can choose a "good" result and resume from there (load all states)
  • Can modify based on experiment result
    • eg token limit reached -> ask to extend answer (i.e. relax token limit)
    • Can manually intervene/edit
    • Quick extract for continuation
  • Can save a dataset for fine-tuning

Value added features

  • Prompt suggestion ability
    • Search on prompthub
    • AI suggestion based on magic prompt
  • Insert prompt snippet
  • Data auto-generation
    • Let LLM do it (!)
  • Data import

Publish and export

Once we're satisfied with the result.

  • Publish results in a fixed prompt + dataset.
    • Have a separate UI (playground) that let other users try it out quickly.
  • Export to download a file containing all relevant data
    • So that it can be used elsewhere, e.g. in an LLM application development framework.

Advanced feature: Human Eval

When running larger experiments, may want to out source evaluation to a team of human raters. Would like to have a separate part to manage this.

(Integration with Scale AI?)

Advanced feature: Finetuning

Single Prompt Editor UI Design

Load Prompt

  • Separate page
  • Category and keyword
  • Display versions + branches
  • List tagged versions
  • New Prompt
    • Basic option (empty, from template)
    • Wizard to guide beginner (choose base template + content from AI or prompthub)

Main UI

  • Two Column Panel
  • Left: Edit prompts or data (more later)
  • Right: Experiment Results
    • Filtered list of experiments (can change filter criteria)
    • Accordian to expand individual experiments
  • Alternative: Competition view of experiemnt
    • Current accepted choice on top
    • Candidates at bottom
  • Has other panels
  • Run Experiment
    • Using default
    • Use last config
    • New Experiment (full config)

Option panel

  • Set Models and Params here
  • Can have multiple instances (named)
  • Must have one default model and default param (used for quick run)

Prompt panel

  • Root Prompt and subprompts with tree view
  • List view of variations
  • Preview of rendered?

Data/Input panel

  • List view of all input variables
  • Natural text or structured data
    • Advanced JSON editor for structured data
  • Dataset (with a default) (also list view)

Experiment config UI

  • Big table
  • Each row for an adjustable input type (prompt variants, dataset, model, param)
  • First column: Run default, run specifics (ad hoc value?)
  • Second column: Filterable dropdown to choose (can also enter text + autocomplete)

Playground (For published prompts)

  • Show the compiled prompt template
    • Realtime update to show actual rendered final prompt when user input values to the variables?
  • Simplified input
    • Each variable has a textfield/JSON editor
    • Dropbox to choose example value

Prompt Chain Visual Editor

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment