Packaging docs for tea

tea-packaging.md

tea pantry overview

What is a Pantry?

Pantries provide consistent metadata about open source packages. This metadata shouldn’t require manual collection, but at this current state in open source it does.

It is collected and duplicated thousands of times. A huge waste of effort.

tea aims to eradicate this wasted effort, though unfortunately, the journey there will require—to some extent—doing that duplication one more time.

Doing it a Little Better This Time

Our format is YAML, which is at least non-proprietary and could be used by other tools without an independent parser. And we’re pulling in data from other sources as much as possible, eg. versions are taken from the “source” whenever possible.

Structure of a pantry

A pantry requires a projects folder with packages inside. Each package name should be a unique identifier. One package directory may contain multiple packages. This is common for toolchains for C and Rust, which is nested in gnu.org and crates.io, respectively, as well as packages without a defined home which are publish under github.com.

At the core of each package is a package.yml file, that contains metadata and build instructions for the package.

root/
  projects/
  |--- pkg-a.com/
       |--- package.yml
  |--- pkg-b.com/
       |--- pkg-d/
            |--- package.yml
  |--- pkg-c.com
       ...

Decentralized pantries

The repository pantry.zero is a stub that will only ever contain a README file. The packages contained in forks of this repo will be searchable when using tea/cli itself. Thus decentralizing the upkeep of the open source package graph.

Forks of specific pantries will also be able to inherit functionality from those pantries.

⚠️ This is not built yet, but it’s a near term goal. ⚠️

We love GitHub, but it’s centralized. Open source is too important to have a single gatekeeper.

Current implementation

We’re launching with two pantries:

• pantry.core
• pantry.extra

pantry.core is essential libraries and utilities and we intend for its contents to always be up-to-date, well-tested and available for all our supported platforms.

pantry.extra is everything else.

New package submissions should go to pantry.extra. Once they are stable and if otherwise deemed appropriate we will move them to pantry.core.

Creating packages

There are 2 ways of creating packages. You can create your own pantry, or you can contribute a package.

1. Create your own pantry
2. Contribute a package (recommended)

1. Create your own pantry

⚠️ This is not built yet, as mentioned in the section, Decentralized pantries.

• Fork pantry.zero.
• Rename your fork.
• Complete step-by-step section.

The packages contained in your fork will be searchable when using tea/cli itself. Thus decentralizing the upkeep of the open source package graph.

Forks of specific pantries will also be able to inherit functionality from those pantries.

This is not built yet, but it’s a near term goal.

2. Contribute a package (recommended)

• Fork pantry.extra.
• Create a branch for your new package (git checkout -b example.com).
• Complete step-by-step section.
• Submit a pull request for your package. Once they are stable and if otherwise deemed appropriate we will move them to pantry.core.

Step-by-step guide to create a package

The following is a step-by-step guide to creating a package. Please don't just blindly copy commands into your terminal.

Setup local dev env

Install tea/cli

sh <(curl https://tea.xyz)

Clone the following repositories in the same directory

# create a new package development directory and step into it
mkdir pkgdev
cd pkgdev

# clone tea/cli repository
git clone https://github.com/teaxyz/cli
# clone pantry.core repository
git clone https://github.com/teaxyz/pantry.core
# clone your fork of pantry.extra repository
git clone https://github.com/<USERNAME>/pantry.extra # replace with your github username

If you cloned the repositories correctly, you should find all three repositories in the directory pkgdev:

$ ls
cli
pantry.core
pantry.extra # your fork

It's important that you keep the repository tea/cli and the tea installation up-to-date.

sh <(curl https://tea.xyz) # updates the latest version of `tea`

cd cli 
git pull origin main # pulls the latest code from the repository

Create a new package

⚠️ Before you create a new package,

• double-check the pantry if the package already exists with open ~/.tea/tea.xyz/var/pantry/projects or
• if the package is being worked on in pantry.extra pull requests.

Checkout a new git branch

cd pantry.extra
git checkout -b example.org

Create a new folder inside pantry.extra

mkdir example.org # this is your unique package name
touch example.org/package.yml

Edit the package.yml

The package.yml file is the core of building new software. They can range from the exceedingly simple (just.systems/package.yml), to the very, very complicated (python.org/package.yml), and everything in between. Generally, they are very simple, as package maintainers do a lot internally to simplify building for the benefit of their users.

The best way to get started with editing the package.yml is to look at existing pantry packages that share a similar tech stack to the package you want to create.

Please check out the package.yml documentation for a comprehensive explanation of each key and subkey.

Here's the simplest version of package.yml:

# package.yml

distributable:
  url: https://example.org/download/{{version}}/src.tar.gz
  strip-components: 1

versions:
  - 1.0.0

build:
  script: |
    touch "{{prefix}}"/example

test:
  script: |
    ls -l
  working-directory: build

The build --> script subkey has to build something to {{ prefix }}, which in the example above is $TEA_PREFIX/example.org/v1.0.0.

The tea build system requires a test --> script key to be present. See Test a package section for more

Build a package

Before you submit your new package, make sure that package builds are actually running successfully.

Requirements

• macOS requires Xcode Command Line Tools: sudo xcode-select --install
• Linux requires libc-dev

Package building also require

1. a permissionless GitHub personal access token (PAT) to fetch the lastest version of the source code from GitHub and
2. the path to the pantry.extra fork, so the build tools know which pantry you are editing.

export GITHUB_TOKEN=gph_secret

export TEA_PANTRY_PATH="$PWD/pantry.extra" # path to fork of pantry.extra

Build instructions

Finally, run the build script with your package to test if everything builds as expected

# make sure you are in pkgdev/ directory
pantry.core/scripts/build.ts example.org

Test a package

Packages require a test YAML node. This script should thoroughly verify all the functionality of the package is working. We are not testing the source, we're testing the implementation. You want to write a test that uses the tool.

You can run the test with:

pantry.core/scripts/test.ts example.org

tea requires all packages be relocatable and cross platform. Our CI will verify this for you. You can check locally by moving the installation from ~/.tea to another tea installation (eg. ~/scratch/tea†) and running the test again.

† TEA_PREFIX="~/scratch/tea sh <(curl tea.xyz)

Contribute your package

Now make a pull request! We’ll test on all platforms we support in the PR. If it passes both CI and review: we’ll merge!

After your contribution

We build “bottles” (tar’d binaries) and upload them to both our centralized bottle storage and decentralized IPFS.

tea automatically builds new releases of packages as soon as they are released (usually starting the builds within seconds). There is no need to submit PRs for updates.