Skip to content

Instantly share code, notes, and snippets.

@cdrw-nutyx
Forked from ruario/README.md
Created January 29, 2019 15:03
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save cdrw-nutyx/d3adaeaa54d2e0216881cc31c9044b2b to your computer and use it in GitHub Desktop.
Save cdrw-nutyx/d3adaeaa54d2e0216881cc31c9044b2b to your computer and use it in GitHub Desktop.
This script provides a method for creating generic (cross-distro), installable, binary packages. The created packages include a log that makes uninstall easy.

Createpkg

Intro

This script provides a simple method for tracking software that you have built from source, or binaries that you want to repackage from another distribution. It also simplifies transferring the software to other machines for install or backup. It works as a nice secondary package management tool, without the need to work with complex, distro specific, build tools.

Concepts have been borrowed from Arch Linux's makepkg, CRUX's pkgmk, porg (formely paco) and other Linux package managers. Like makepkg/pkgmk it works by reading a meta file in the current working directory (in this case, named "BUILD"). This file specifies details about where to find the software, what it is called and how to build it. This information is combined with commands within the createpkg script itself, allowing for automation of source download, extraction, compilation and creation of installable “packages”. As with porg, these packages include a log of all the files present (including the log itself). This log can be used to precisely remove the files that form the package contents, even if they are scattered across the file system.

Unlike more comprehensive package management tools (including those from which it borrows ideas), this script has no major dependencies outside of a typical Linux build environment. This means that whilst its abilities are quite limited, it is very easy to setup (create a BUILD file and run the script). It also automatically handles many of the tedious, repeative steps such as source fetching/extraction, setting (root) file ownership, compressing man pages, stripping binaries and tracking of installed files.

Usage

In an empty directory create a BUILD file that sets basic variables describing the package (usually only “name”, “version” and an array called “source” [that links to source files] are required). Add a “build” function that includes the build and installation steps. The familiar ./configure; make install combo is often enough, with some method to redirect the output to the packaging/staging directory (e.g. “DESTDIR=”, “--prefix”, etc.).

BUILD

The easist way to explain how BUILD files work is with examples, so here are a selection.

name=dos2unix
version=7.4.0
source=("https://waterlan.home.xs4all.nl/$name/$name-$version.tar.gz")

build() {
  make prefix="$pkg/usr/local" install
}
name=lz4
version=1.8.3

build() {
  git clone -b v$version --single-branch --depth 1 https://github.com/$name/$name.git
  cd $name
  make install DESTDIR="$pkg"  
}
name=slack-desktop
version=3.3.3
arch=x86_64 # Slack only offers x64 packages
options=(!man !strip)
source=("https://downloads.slack-edge.com/linux_releases/$name-$version-amd64.deb")

build() {
  cp -R usr "$pkg"
  rm -fr "$pkg/usr/share/lintian"
}
name=createpkg
_name=11246070 # Non-standard variable
version=master
arch=noarch
options=(clearsrc !man !strip)
source=("https://gist.github.com/ruario/${_name}/archive/$version.zip")

build() {
  cd ${_name}-$version
  prefix="$pkg/usr/local"
  bindir="$prefix/bin"
  docdir="$prefix/share/doc"
  for script in $name.sh gen-BUILD.sh; do
    install -Dm755 $script "$bindir/${script%.sh}"
  done 
  for doc in README.md template-BUILD; do
    install -Dm644 $doc "$docdir/$name/$doc"
  done
}

Options

The "options" array is used to change the default actions:

  • clearsrc: Re-downloaded source packages on every run (Default = Off)
  • man: Automatic compression of man pages (Default = On)
  • root: Set all files as being owned by root:root (Default = On)
  • strip: Strip symbols from binaries (Default = On)

To switch an option On, write its name in the array. To switch it Off, write it prefaced with “!”. Anything missing from the array will use the defaults (see above).

For example, “options=(!man !strip)” would disable man page compression and binary stripping, but ownership would still be set to “root:root” (within the package) and source packages that are already locally present would not be downloaded again.

Creation, Installation & Removal

Once a BUILD file has been created, all that is needed is to run the createpkg from within the directory where BUILD is housed and an installable package should be created. Using the first BUILD file above, would result in an installable package called “dos2unix-7.4.0-x86_64-1.pkg.tar.gz” being created (on a 64-Bit machine). A nice advantage of a well made BUILD file is that typically you need only adjust the version variable when a new release comes out.

Installation of these packages simply requires extracting their contents directly into the top level of the file system.

sudo tar Cfx / dos2unix-7.4.0-x86_64-1.pkg.tar.gz

It is then possible to view a listing of the contents of the installed package at any time by reading the appropriate log file.

tr \\0 \\n < /var/lib/createpkg/dos2unix-7.4.0-x86_64-1

Should you ever need to remove the package contents from your system, it can be done by issuing a command like the following:

sudo xargs -0 rm < /var/lib/createpkg/dos2unix-7.4.0-x86_64-1

Tip: To do all creation steps except making a package, use the switch “--no-package” (or simply “-n”). An alternative installation command will be echoed back, at the end of the creation process (uninstall works the same way).

Empty directories

All typical filetypes (including symlinks) that were included in the installed package can be removed in this way but not directories. They are intentionally omitted from the package log, since they may have been shared with other software previously present on your system. For the most part empty directories cause no problems and generally have negligible space requirements.

However, if it ever bothers you, construct a find command to track down old empty directories that you might want to consider removing. For example:

find /usr/local -type d -empty
⋮
/usr/local/share/doc/dos2unix-7.4.0/sv
/usr/local/share/doc/dos2unix-7.4.0/zh_CN
/usr/local/share/doc/dos2unix-7.4.0/es
/usr/local/share/doc/dos2unix-7.4.0/de
/usr/local/share/doc/dos2unix-7.4.0/fr
/usr/local/share/doc/dos2unix-7.4.0/uk
/usr/local/share/doc/dos2unix-7.4.0/pt_BR
/usr/local/share/doc/dos2unix-7.4.0/nl
/usr/local/share/doc/dos2unix-7.4.0/pl
⋮

You can remove a tree of nested empty folders, such as “/usr/local/share/doc/dos2unix-7.4.0”, quite easily.

sudo find /usr/local/share/doc/dos2unix-7.4.0 -depth -type d -empty -exec rmdir -v {} \;

Tip: The Filesystem Hierarchy Standard (3.0) documents the default directories you can expect to find on a Linux system.

Extras

A sample (template) BUILD file and an automatic BUILD file generator script are also included.


See also: pkgtrack

#!/usr/bin/env bash
#
# Version: 3.5.1
#
# This script provides a method for creating packages to track software that
# where no native package is available
#
# Copyright 2018 Ruari Oedegaard, Olso, Norway
# All rights reserved.
#
# Redistribution and use of this script, with or without modification, is
# permitted provided that the following conditions are met:
#
# 1. Redistributions of this script must retain the above copyright
# notice, this list of conditions and the following disclaimer.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
# ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER BE LIABLE FOR ANY
# DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
# (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
# LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
# THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
startdir="$PWD"
arch="${arch:-$(uname -m)}"
src="$startdir/src"
pkg="$startdir/pkg"
createpkglogs="/var/lib/createpkg"
buildnr=1
doman=y
dostrip=y
doroot=y
doclearsrc=n
options=''
available() {
command -v $1 >/dev/null 2>&1
}
if available lbzip2; then
compressor=lbzip2
compext=bz2
elif available pigz; then
compressor=pigz
compext=gz
else
compressor='gzip -1'
compext=gz
fi
if available wget; then
downloader=wget
elif available curl; then
downloader="curl -LO"
else
echo "Neither Wget nor cURL were found, downloading of source(s) may fail." 1>&2
downloader=echo
fi
if [ ! -r "BUILD" ]; then
echo 'No BUILD file found. You must create one first!' 1>&2
exit 1
fi
. BUILD
for opts in "${options[@]}"; do
case $opts in
clearsrc) doclearsrc=y ;;
!man) doman=n ;;
!strip) dostrip=n ;;
!root) doroot=n ;;
esac
done
rm -fr "$src" "$pkg"
mkdir "$src" "$pkg"
for files in "${source[@]}"; do
if [ "$doclearsrc" = "y" -a -e "${files##*/}" ]; then
rm "${files##*/}"
fi
if [ ! -e "${files##*/}" ] && echo "$files" | grep -q '^\(http\|ftp\)'; then
$downloader "$files"
fi
if [ ! -e "${files##*/}" ]; then
echo "${files##*/} not found!" >&2
exit 1
fi
(cd "$src"; ln -s "$startdir/${files##*/}" .)
case "${files##*/}" in
*.tar.bz2) tar -xf "${files##*/}" -C "$src" ;;
*.tar.[glx]z) tar -xf "${files##*/}" -C "$src" ;;
*.tar.lzma) tar -xf "${files##*/}" -C "$src" ;;
*.t[bglx]z) tar -xf "${files##*/}" -C "$src" ;;
*.cpio.bz2) bzcat "${files##*/}" | (cd "$src"; cpio --quiet -id) ;;
*.cpio.gz) zcat "${files##*/}" | (cd "$src"; cpio --quiet -id) ;;
*.cpio.lz) lzip -cd "${files##*/}" | (cd "$src"; cpio --quiet -id) ;;
*.cpio.lzma) xzcat "${files##*/}" | (cd "$src"; cpio --quiet -id) ;;
*.cpio.xz) xzcat "${files##*/}" | (cd "$src"; cpio --quiet -id) ;;
*.zip) unzip -qd "$src" "${files##*/}" ;;
*.rpm) mkdir -p "$src/$name-$version"; rpm2cpio "${files##*/}" | (cd "$src/$name-$version"; cpio --quiet -id) ;;
*.deb) mkdir -p "$src/$name-$version"; (cd "$src"; ar x "${files##*/}"; tar xf data.tar.* -C "$name-$version") ;;
esac
done
if [ -d "$src/$name-$version" ]; then
echo "Switching directory: $src/$name-$version"
cd "$src/$name-$version"
else
cd "$src"
fi
set -eu
build
pkgname="$name-$version-$arch-$buildnr"
if [ "$doman" = "y" ]; then
for f in $(find "$pkg" -regextype posix-egrep -regex ".*/man/([[:alnum:]\._-]*/)?man[[:alnum:]]*/.*" -type f | grep -Ev '\.(bz2|[glx]z|lzma)$'); do
gzip -9 "$f"
done
for l in $(find "$pkg" -regextype posix-egrep -regex ".*/man/([[:alnum:]\._-]*/)?man[[:alnum:]]*/.*" -type l | grep -Ev '\.(bz2|[glx]z|lzma)$'); do
cd "${l%/*}"
ln -s "$(readlink "${l##*/}").gz" "${l##*/}.gz"
rm "${l##*/}"
cd - >/dev/null
done
fi
if [ "$dostrip" = "y" ]; then
for bin in $(find "$pkg" -type f -exec file {} \; | sed -rn 's/(.*): (setuid )*ELF.*(executable|object).*not stripped/\1/p'); do
strip --strip-unneeded "$bin"
done
fi
if [ "$doroot" = "y" ]; then
taropts="--owner 0 --group 0 "
cpioopts="-R 0:0 "
else
taropts=''
cpioopts=''
fi
cd "$pkg"
find * ! -type d -printf '/%p\0' > .footprint
printf "$createpkglogs/$pkgname\0" >> .footprint
install -Dm644 .footprint ".$createpkglogs/$pkgname"
find * ! -type d -print0 > .footprint
if [ "x${1:-}" = "x-n" -o "x${1:-}" = "x--no-package" ]; then
printf '\n'
xargs -0 ls -l < .footprint
cat <<END
Package creation was prevented because "${1:-}" was issued.
To install $name-$version, switch the the "${pkg##*/}" directory and issue the
following (as root or prefaced with 'sudo'):
cpio $cpioopts-p0d / < .footprint
To create a package (form the "${pkg##*/}" directory):
tar -cvvf- $taropts--null -T .footprint | $compressor > ../$pkgname.pkg.tar.$compext
END
else
echo -e "\nCreating: $pkgname.pkg.tar.$compext\n"
tar -cvvf- $taropts --null -T .footprint | $compressor > "$startdir/$pkgname.pkg.tar.$compext"
echo -e "\nTo install (as root or prefaced with 'sudo'):\n\n tar Cfx / $pkgname.pkg.tar.$compext\n"
fi
echo -e "\nTo uninstall (as root or prefaced with 'sudo'):\n\n xargs -0 rm < $createpkglogs/$pkgname\n\n"
#!/bin/sh
# Auto-generate a BUILD file based on URL to source package
# Usage: ./gen-BUILD.sh [URL to source package]
if [ -n "$1" ]; then
package=$(echo "${1##*/}" | rev | sed -nr 's#((zg|zl|amzl|zx|2zb)\.(rat|cpio)|piz)((\.crs|\.giro)){,1}\.([0-9a-z.]+)[_-](.*)#\6/\7#p' | rev)
if [ -z "$package" ]; then
echo "Could not work out package name and version!"
exit 1
fi
name="${package%/*}"
version="${package#*/}"
source="$(echo "$1" | sed "s/$name/\${name}/g;s/$version/\${version}/g;")"
sed 's/^ //' <<BUILD
name=$name
version=$version
#arch=noarch
#buildnr=2
#options=(!root !man !strip)
source=("$source")
build() {
./configure
make install DESTDIR="\$pkg"
}
BUILD
else
echo 'You must provide the URL to the source package' >&2
exit 1
fi
# To use this createpkg BUILD file template
#
# • Save it to an empty directory
# • Rename it to 'BUILD'
# • Adust as needed
# • Run createpkg (from the directory containing 'BUILD')
# Set the application's name, e.g. name=example
name=
# Set the application's own version number, e.g. version=1.0
version=
# If the package will be architecture independent (e.g. software based on
# Perl, Python, Ruby, etc.), uncomment the following line.
#arch=noarch
# This is packaging number. You can uncomment and increase it if you make
# improved build steps but the version number of the software has not
# increased. It is preset to 1 if you do not define it.
#buildnr=2
# The "options" array is used to change the default actions:
#
# • clearsrc: Re-downloaded source packages on every run (Default = Off)
# • man: Automatic compression of man pages (Default = On)
# • root: Set all files as being owned by root:root (Default = On)
# • strip: Strip symbols from binaries (Default = On)
#
# To switch an option "on", write its name in the array. To switch it "off",
# write it prefaced with '!'.
#options=(clearsrc !man !root !strip)
# This is an array that lists the the source package(s). Where possible you
# should use full URL(s). This will allow for automatic downloading of source
# if it is not present locally. Using variables in the URL(s) and package
# name means that you will likely only need to update the version number when
# a new source package is released. You can also remove this line if you
# don't want sources automatically fetched/extracted and would prefer handle
# these steps in the build function.
source=("https://example.com/downloads/$name-$version.tar.gz")
# The following are typical build steps for software that builds via GNU
# Autotools. Uncomment and adjust the examples as needed.
build() {
# Extraction of tar, cpio, zip, rpm and deb source files packages is
# automatic. For anything else you will need to provide your own extraction
# command(s), before proceeding.
#extraction_command foo
# if a "${name}-$version" directory was created during automatic
# extraction you will start inside this directory, otherwise you will start
# in the directory alongside your source package(s). Switch directory as
# needed.
#cd bar
# Configure the software. Adding any switches to suite your own
# requirements.
#./configure
# Make and install the software into $pkg. Tip: If $DESTDIR is not
# supported you may be able to use --prefix= in the previous step).
#make install DESTDIR="$pkg"
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment