Skip to content

Instantly share code, notes, and snippets.

Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save sbleon/3982291 to your computer and use it in GitHub Desktop.
Save sbleon/3982291 to your computer and use it in GitHub Desktop.
Keeping server software up to date

Keeping Server Software Up To Date

Goals (Why do we upgrade things?)

Primary goal: Improve Security

Secondary goal: More consistency across servers

Which things do we upgrade?

The Stack

  1. OS
  2. Managed Packages
  3. Unmanaged Packages
  4. Application Frameworks
  5. Application Libraries
  6. Application Code

From the OS up to app code, each layer that is out of date makes each higher layer harder to update. For example, if the OS is badly out of date, the package manager's repositories may not contain the latest packages. If the OS has reached EOL, the repositories may no longer be online. We encountered this recently when centos 4 reached EOL.

Upgrade Decisions

For the highest quality semantically versioned projects, with versions greater than or equal to 1.0.0, upgrade decisions are easy.

  1. patch-level - apply immediately
    • semver: only backwards compatible bug fixes
    • these usually contain critical security fixes
  2. minor-version - apply within one month
    • semver:
      • stability fixes
      • new backwards-compatible features
      • deprecations
  3. major-version
    • before EOL - with or without client approval/budget
    • when requested by a developer - with client approval + budget

For known violators of semver, upgrade decisions are more difficult. Also, some projects are semantically versioned but are not of the highest quality. These projects may have a reputation for being unstable during the first one or two patch levels after a minor version release.

  1. ruby
    • four level versioning: maj-min-tiny-patch
    • patch versions (fourth level) aim to be backwards compatible
    • tiny versions contain new features
  2. MacOS
    • three level versioning, but fails to increment major version
    • The first one or two patch levels after a minor version release are often unstable.

How do we upgrade things?

Currently, it's all manual. Finding a good tool for server management is important, but a separate project.

We have experimented with Sprinkle for configuration management, but have found it to be fairly difficult and relatively expensive.

When do we upgrade things?

TBD - How do we schedule these updates and stick to the schedule?

Who upgrades things?

Delegate Responsibility

Responsibility for upgrading app frameworks and libraries has already been delegated to the developers. Responsibility for the OS and packages has not been delegated.

We can either concentrate the responsibility to a single individual or spread it among many. If a single staffperson is interested in this role, that may be most efficient. Then again, spreading the responsibility among multiple people will broaden their skills.

Either way, it must be clear that upgrades are a primary responsibility, on par with new features and bugs. Non-billable maintenance must have the same priority as billable work, and this must be reflected in goal charts and other metrics.

Supervise

In general, little supervision is necessary, especially if a single staffperson is responsible for all OS and package upgrades. If responsibility is distributed widely among developers, oversight becomes more important.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment