Skip to content

Instantly share code, notes, and snippets.

@rjhintz

rjhintz/disposable_infrastructure.md

Last active October 1, 2015 16:06
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save rjhintz/8813a1db085ad109240a to your computer and use it in GitHub Desktop.
Save rjhintz/8813a1db085ad109240a to your computer and use it in GitHub Desktop.
Download ZIP
Disposable Infrastructure
Raw
disposable_infrastructure.md

#Disposable Infrastructure

Draft

Note: Comments welcome, especially opposing views or pointers to faulty logic.

##Overview This discusses "disposable infrastructure." ##What is Disposable Infrastructure? Disposable infrastructure are infrastructure components that are deleted from the configuration when they fail or have degraded past a boundary condition. The state of the resource may not be retained, even if it might be necesssary or helpful.

One observation is that adopting immutable infrastructure practices moves the boundary between versioned and unversioned closer to the lower end of the system/stack.

Disposable infrastructure can be compared and contrasted with an application's disposability, one of the characteristics of the 12 factor app. Disposability for an application means it's graceful during planned shutdown or unplanned, sudden death.

###Can All Infrastructure Components Be Immutable? Which infrastructure Components can, in principle, be immutable? Is there such as thing as stateless infrastructure?

As a practical matter, which infrastructure components can be immutable?

Which of these should be immutable? Are there some overriding dollar or wall clock costs or operational considerations associated with making components some immutable? Which components, by class, are at the margin? ###Scope What about components that are controlled by others?

What about components at the boundary where control is by others? ##Why is Disposable Infrastructure Important? What does it enable? What does it mitigate? ##Who Creates, Modifies, Uses, Terminates It? Disposable infrastructure is:

  • created by operations, sometimes by development, in response to design requirments.
  • modified by operations, rarely (?) by development, in response to work order
  • used by operations on behalf of end users
  • terminated by operations in response to policy

##How is it Created, Modified, Used, Terminated? Disposable infrastructure methods include:

  • for creation,scripts or more formal automation such as configuration management tools following design or policy guidelines applying current versioning
  • for updating, scripts or more formal automation such as configuration management tools following design or policy guidelines applying new versioning. (Note: the end infrastructure products aren't updated. The tools affecting them are updated.)
  • for using,components are typically used QA, and production systems. They can be used in development systems.
  • for terminating, obsolete resource are terminated by policy or, if orphaned, manually when identified

##Issues

  • How important is state?
  • What do you know about infra that is outside your control? (See other remarks about scope.)
  • Is it an unattainable myth?
  • Generally immutable infra refers to server type instances, what's the applicability, if any, to data stores, that is, those components handling persistent data?
    • Does distributed, "self healing" storage constructs such as Riak or Cassandra mitigate this issue? Mike Fiedler has a demo of killing MongoDB primary without change to application business metric (around 16:30 of his talk)
  • Special case of queues: in a distributed environment a queue service may deliver duplicate messages or reordered messages when network partitions/outages occur. In some cases, messages may be lost. Example, RabbitMQ
  • Do patterns for development of versions, automation, and deployment need to be developed and settled on before production use?
  • Are auto acceptance tests available to be run on the results of the automation process, especially for new versions?
  • Facilitates self service, but how to enforce standards?
  • What is the garbage collection process? How to get rid of tech debt in image.

###Is State Siloed? Florian Motlik has written:

The main criticism against immutable infrastructure – as stated in the Chef blog post – is that there is always state somewhere in the system and, therefore, the whole system isn’t immutable. That misses the point of immutable components. The main advantage when it comes to state in immutable infrastructure is that it is siloed. The boundaries between layers storing state and the layers that are ephemeral are clearly drawn and no leakage can possibly happen between those layers. There simply is no way to mix state into different components when you can’t expect them to be up and running the next minute.

Is this true for components that handle persistent storage? How about transactions in flight from ephemeral instances to persistent storage? Update: Many discussions carve out data store handling separately or recommend the use of distributed, resilient databases. See elsewhere in this document.

##Advantages Advantages include:

  • reduce dependence on instance stability
  • reduces technical debt associated with undocumented changes
  • may mitigate issue of "it worked on my laptop"
  • may help with "DevOops" Kief Morris
  • with a "narrower interface" better guarantees may be possible

##Disadvantages Disadvantages include:

  • unclear what happens to required or useful information in state of stateful resources that are marked for disposal
  • almost requires modern source control/versioning system
  • need a general automation policy, including tool selection, training, and transition
  • what happens if a resource degrades, but not over boundary level. Then another. Then several/many, so the aggregate is not resouce failure, but unacceptable degradation
  • easy to cause a problem with a wide dispersion quickly
  • "introduces dependencies to third parties during deployment. If you install packages in the system and your app repository is slow or down this can fail the deployment." Florian Motlik

##Barriers to Adoption Barriers to adoption include:

  • how many and what versions to adopt?
  • how to perform configuration?
  • what co-existence of versioning can be tolerated?
  • is there an undocumented security or compliance patch in a running version that won't be carried forward to the replacement version?
  • goes against human nature since it's a lot easier to make a small, probably poorly documented change on a single server
  • may be viewed as more risky by management since the transition makes it seem like "Version 1 of Everything"

##Contrary Views

  • Eric Windisch, Things will Change - Usenix Keynote UCMS'14, slides 20 and following
    • "It is naive to think we can simply throw away VMs or containers — we want to preserve their state for archival and analysis."
    • "The biggest problem with blind adherence to immutable infrastructure & 12-factor… is ignorance of the importance of the implicit state of a system which should not be deemed disposable."

See also video in References.

##References

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment