aggregatorIdea.md

The flow below illustrates a workflow where all social accounts are polled, then fed into a system that helps surface the best 5% of things to publish in the main feed. We then take a portion of that, remove anything linking to Twitter, give Videos around two minutes long more weight (pulling in more if needed), and publish to Mastodon, just as an example of how we might use multiple outputs.

Here's the flow:

flowchart TD
    sTwitter[Twitter] --> cli{Fetcher};
    sBlog[Blog] --> cli{Fetcher};
    sYouTube[YouTube] --> cli{Fetcher};
    sDevTo[DevTo] --> cli{Fetcher};
    sReddit[Reddit] --> cli{Fetcher};
    sHashNode[HashNode] --> cli{Fetcher};
    sInstagram[Instagram] --> cli{Fetcher};
    sTikTok[TikTok] --> cli{Fetcher};
    cli{Fetcher} ----> cli_m{Smart Selection};
    cli_m{Smart Selection} ----> cli{Fetcher};
    cli_u{CLI Utility} ----> cli_m{Smart Selection};
    cli_m{Smart Selection} ----> cli_u{CLI Utility};
    cli_b{Bias Handler} ----> cli_m{Smart Selection};
    cli_m{Smart Selection} ----> cli_b{Bias Handler};
    cli_u{CLI Utility} ---> cli_d{Draft Generator}
    cli_d{Draft Generator} --> cli_u{CLI Utility};
    cli_d{Draft Generator} --> cli_p{Publisher};
    cli_p{Publisher} ----> pub_a[Main Feed];
    cli_p{Publisher} ----> pub_b[Mastodon W/O Twitter & More YouTube];
    cli_p{Publisher} ----> pub_c[Optimized For Newsletter];

Diagram, Explained:

Each part of this could conceivably be a separate hosted function, micro service or even just library component.

Fetcher

The fetcher is responsible for pulling a feed (via API, RSS, whatever) of source accounts and saving each individual post in a uniform object format with any associated metadata.

Smart Selection

Over time, learn what gets pulled to go into the feed and suggest items accordingly. Also maintain a comprehensive tone analysis of what's been in the feed. This learns from what the editor chooses, as well as engagement metadata from both the source and the main feed that we publish.

CLI Utility

A tool to manage everything. Commands might include fetch, refresh, create-drafts, publish and more. The CLI is more for automating things than making decisions.

Bias Handler

A serverless (or local) function that comports a publication feed to conform to a certain bias (e.g. don't show Twitter links in this output, and give short YT videos extra preference, or similar). These will be defined in a configuration file. The handler also helps us keep track of tone as we publish, so we don't over-index on topics.

Publisher

This produces MD/MDX that can be checked in, or prepares copy-pasteable snippets for publishing to the other outputs. Eventually all of this should be automated so that no copypasta is needed.