Skip to content

Instantly share code, notes, and snippets.

@wmanley
Last active December 29, 2016 01:09
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save wmanley/ea88e4ffc67eda8fef9c6ddd9e901276 to your computer and use it in GitHub Desktop.
Save wmanley/ea88e4ffc67eda8fef9c6ddd9e901276 to your computer and use it in GitHub Desktop.
Incremental ostree deploy notes

How does ostree admin deploy work?

src/ostree/ot-admin-builtin-deploy.c

ostree_sysroot_deploy_tree (sysroot,
                            opt_osname, revision, origin,
                            merge_deployment, kargs_strv,
                            &new_deployment,
                            cancellable, error)

src/libostree/ostree-sysroot-deploy.c

/**
 * ostree_sysroot_deploy_tree:
 * @self: Sysroot
 * @osname: (allow-none): osname to use for merge deployment
 * @revision: Checksum to add
 * @origin: (allow-none): Origin to use for upgrades
 * @provided_merge_deployment: (allow-none): Use this deployment for merge path
 * @override_kernel_argv: (allow-none) (array zero-terminated=1) (element-type utf8): Use these as kernel arguments; if %NULL, inherit options from provided_merge_deployment
 * @out_new_deployment: (out): The new deployment path
 * @cancellable: Cancellable
 * @error: Error
 *
 * Check out deployment tree with revision @revision, performing a 3
 * way merge with @provided_merge_deployment for configuration.
 */
gboolean
ostree_sysroot_deploy_tree (OstreeSysroot     *self,
                            const char        *osname,
                            const char        *revision,
                            GKeyFile          *origin,
                            OstreeDeployment  *provided_merge_deployment,
                            char             **override_kernel_argv,
                            OstreeDeployment **out_new_deployment,
                            GCancellable      *cancellable,
                            GError           **error)
  • What is the merge_deployment?

The actual checking-out seems to be done by checkout_deployment_tree:

checkout_deployment_tree (self, repo, new_deployment, &deployment_dfd,
                          cancellable, error)
/**
 * checkout_deployment_tree:
 *
 * Look up @revision in the repository, and check it out in
 * /ostree/deploy/OS/deploy/${treecsum}.${deployserial}.
 */
static gboolean
checkout_deployment_tree (OstreeSysroot     *sysroot,
                          OstreeRepo        *repo,
                          OstreeDeployment  *deployment,
                          int               *out_deployment_dfd,
                          GCancellable      *cancellable,
                          GError           **error)

So if we want to overwrite an existing tree this function would have to be replaced.

This function blows away any existing deployment at that directory and then checks a tree out from fresh. It delegates the actual checking out to ostree_repo_checkout_at:

ostree_repo_checkout_at (repo, &checkout_opts, osdeploy_dfd,
                                checkout_target_name, csum,
                                cancellable, error)

Perhaps instead of blowing away the existing we can just modify it in place (assuming that it hasn't been corrupted in some way).

But first we need a way of transitioning one tree to another. Lets look at ostree_repo_checkout_at:

src/libostree/ostree-repo-checkout.c:


/**
 * ostree_repo_checkout_at:
 * @self: Repo
 * @options: (allow-none): Options
 * @destination_dfd: Directory FD for destination
 * @destination_path: Directory for destination
 * @commit: Checksum for commit
 * @cancellable: Cancellable
 * @error: Error
 *
 * Similar to ostree_repo_checkout_tree(), but uses directory-relative
 * paths for the destination, uses a new `OstreeRepoCheckoutAtOptions`,
 * and takes a commit checksum and optional subpath pair, rather than
 * requiring use of `GFile` APIs for the caller.
 *
 * It also replaces ostree_repo_checkout_at() which was not safe to
 * use with GObject introspection.
 *
 * Note in addition that unlike ostree_repo_checkout_tree(), the
 * default is not to use the repository-internal uncompressed objects
 * cache.
 */
gboolean
ostree_repo_checkout_at (OstreeRepo                        *self,
                         OstreeRepoCheckoutAtOptions       *options,
                         int                                destination_dfd,
                         const char                        *destination_path,
                         const char                        *commit,
                         GCancellable                      *cancellable,
                         GError                           **error)

Delegates to static function checkout_tree_at:

checkout_tree_at (self, options,
                  destination_dfd,
                  destination_path,
                  (OstreeRepoFile*)target_dir, target_info,
                  cancellable, error)
/*
 * checkout_tree_at:
 * @self: Repo
 * @mode: Options controlling all files
 * @overwrite_mode: Whether or not to overwrite files
 * @destination_parent_fd: Place tree here
 * @destination_name: Use this name for tree
 * @source: Source tree
 * @source_info: Source info
 * @cancellable: Cancellable
 * @error: Error
 *
 * Like ostree_repo_checkout_tree(), but check out @source into the
 * relative @destination_name, located by @destination_parent_fd.
 */
static gboolean
checkout_tree_at (OstreeRepo                        *self,
                  OstreeRepoCheckoutAtOptions       *options,
                  int                                destination_parent_fd,
                  const char                        *destination_name,
                  OstreeRepoFile                    *source,
                  GFileInfo                         *source_info,
                  GCancellable                      *cancellable,
                  GError                           **error)

Finally this is a function that acutally does some checking out. It recurses through a directory tree checking out each directory with checkout_tree_at (itself) and each file with checkout_one_file_at. It's a little complicated as it has to deal with all the different options that could be passed in including --user-mode and --union.

There is a conflation of iterating though the tree and actually performing actions in this function.

How can we make updating a tree work?

For replacing one tree with another we will need to iterate though the differences between two trees creating, deleting and replacing files as necessary. ostree already provides a method of listing differences between two trees: ostree_diff_dirs:

/**
 * ostree_diff_dirs:
 * @flags: Flags
 * @a: First directory path, or %NULL
 * @b: First directory path
 * @modified: (element-type OstreeDiffItem): Modified files
 * @removed: (element-type Gio.File): Removed files
 * @added: (element-type Gio.File): Added files
 * @cancellable: Cancellable
 * @error: Error
 *
 * Compute the difference between directory @a and @b as 3 separate
 * sets of #OstreeDiffItem in @modified, @removed, and @added.
 */
gboolean
ostree_diff_dirs (OstreeDiffFlags flags,
                  GFile          *a,
                  GFile          *b,
                  GPtrArray      *modified,
                  GPtrArray      *removed,
                  GPtrArray      *added,
                  GCancellable   *cancellable,
                  GError        **error)

This function returns a list of modified, removed and added files. It doesn't expose an iterator interface. This has the disadvantage that it may use up a bunch of RAM when the diff is large. OTOH it makes usage simple. ostree_diff_dirs works by recursing through the two trees appending to the lists of modified, removed and added files.

Much like checkout_tree_at ostree_diff_dirs iterates through a directory tree with recursion, although is more general as it can iterate through two trees at the same time.

Seperation of concerns

So theres an oppertunity to refactor to share the tree iteration between the two functions. Then we will be able to enhance checkout_tree_at to deal with a diff between trees.

So ostree_diff_dirs_iterate will implement iteration as an internal iterator which ostree_diff_dirs and checkout_tree_at can use. This is a seperation of concerns allowing reuse.

typedef enum {
  OSTREE_DIFF_ITER_DIR_PRE,
  OSTREE_DIFF_ITER_FILE,
  OSTREE_DIFF_ITER_DIR_POST
} OstreeDiffIterOrder;

typedef gboolean (*OstreeInternalIteratorCallback)(
    gpointer, OstreeDiffItem *, OstreeDiffIterOrder, GCancellable *, GError **);


/**
 * ostree_diff_dirs_iterate:
 * @flags: Flags
 * @a: First directory path, or %NULL
 * @b: First directory path
 * @callback: Callback called for each modified file
 * @data: Data passed to each invocation of the callback
 * @cancellable: Cancellable
 * @error: Error
 *
 * Compute the difference between directory @a and @b.  Calls the provided
 * callback for each file that is added, removed or modified.  This function
 * provides internal iteration of the diff which should be faster and use less
 * memory than using ostree_diff_dirs, particularly for diffs involving many
 * files.
 *
 * The callback will be called for each file that has been added or modified
 * between @a and @b.  It will be called twice for each modified directory, once
 * before listing the contents and once afterwards.  The order parameter is used
 * to distinguish these cases.
 *
 * The callback should have the signature:
 *
 *     gboolean callback(
 *        gpointer data,
 *        OstreeDiffItem *diffitem,
 *        OstreeDiffIterOrder order,
 *        GCancellable *cancellable,
 *        GError **error);
 *
 * * data is the parameters passed into `ostree_diff_dirs_iterate`.  This allows
 *   you to pass context to the callback function.
 * * diffitem describes the diff of the file.  The callback is called twice for
 *   directories and this member can be used to distinguish between the two
 *   calls.  For directories this will be either OSTREE_DIFF_ITER_DIR_PRE or
 *   OSTREE_DIFF_ITER_DIR_POST and for all other files it will be
 *   OSTREE_DIFF_ITER_FILE.
 * * cancellable is the parameter passed into `ostree_diff_dirs_iterate`.  You
 *   can perform blocking work in the callback sharing the cancellable with the
 *   blocking work performed to iterate.
 * * error allows you to propgate errors back to the caller of
 *   `ostree_diff_dirs_iterate`.
 *
 * The callback can return FALSE to stop iterating.  In that case
 * `ostree_diff_dirs_iterate` will also return FALSE.  Otherwise
 * `ostree_diff_dirs_iterate` will return TRUE.
 */
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment