jamiegs

Command aliases

   acp                     !f() { git add -A && git commit -m "$@" && git push --set-upstream origin $(git symbolic-ref --short HEAD); }; f

   cleanup                 !COMMAND="git branch -D"; while [[ $# -gt 0 ]]; do case "$1" in -d|--dryrun) COMMAND="echo"; shift; ;; *) MAIN_BRANCH="$1"; shift;; esac; done; MAIN_BRANCH="${MAIN_BRANCH:-$(git symbolic-ref refs/remotes/origin/HEAD)}"; git for-each-ref --merged="$MAIN_BRANCH" --no-contains="$MAIN_BRANCH" --format="%(refname:short)" refs/heads/ | xargs -n1 -r $COMMAND;#

jamiegs

$count = (Get-Process | Where-Object { $_.Name -eq 'w3wp' }).count -1
$var = 1
$result = ''

$id = (Get-Counter "\Process(w3wp)\ID Process").CounterSamples.CookedValue
$app = [regex]::match((Get-CimInstance Win32_Process -Filter "ProcessId = $id").CommandLine, '"([^"]+)"').Groups[1].Value
$result += "w3wp = $id, $app`n"

while ($var -le $count) {
    $id = (Get-Counter "\Process(w3wp#$var)\ID Process").CounterSamples.CookedValue

jamiegs

Backgound

Previously our warmup and health checks were too coupled. Our load balancers make an http call to /healthcheck on a regular interval. This is to see if it should send traffic to the application or not. It will also kill the container if it's unhealthy after so long.

This was a problem because say mongo is down for an extended time it would take down the entire microservice. The microservice would continously try to launch containers, which would then fail healthchecks during warmup, and get killed again.

Most the application might still be in a good state though but because one dependancy is down, the whole thing is down.

So, we're splitting them out into multiple health checks.

jamiegs

Adaptive Container Scaling

Problems with AWS's recommended scaling method

• Based on a fixed percentage of memory and CPU reservation.
• Scaling on both cpu and memory metrics can cause the metrics to fight with each other where it'll scale out then back in and repeat.
• If you're scaling threshold is at 80% memory, current reservation utilization is at 79% and you deploy a container that requires 25% memory, It will fail to launch.
• ASGs are not container aware.

Solution

jamiegs

$EC2SettingsFile="C:\\Program Files\\Amazon\\Ec2ConfigService\\Settings\\BundleConfig.xml"
$xml = [xml](get-content $EC2SettingsFile)
$xmlElement = $xml.get_DocumentElement()

foreach ($element in $xmlElement.Property)
{
    if ($element.Name -eq "AutoSysprep")
    {
        $element.Value="Yes"
    }

jamiegs

__author__ = 'Jesse'
import urllib2
import base64
import zlib
import json
import StringIO
import gzip

def lambda_handler(event, context):
    data = event.get('awslogs').get('data')

jamiegs

cfg = rs.config()
cfg.settings = { }
cfg.settings.chainingAllowed = false
rs.reconfig(cfg)

jamiegs

2013-04-23 19:08:46,965 ip-0A6EE738 [FATAL] [QueueServices] [request_id=NULL] Unhandled exception
Type=System.OutOfMemoryException
Message=Exception of type 'System.OutOfMemoryException' was thrown.
Data:
  Signature=6ec8ce22
Stack Trace:
   at System.Runtime.Remoting.Messaging.Message.GenerateMethodSignature(MethodBase mb)
   at System.Runtime.Remoting.Messaging.Message.get_MethodSignature()
   at System.Runtime.Remoting.Messaging.MCMDictionary.GetMessageValue(Int32 i)
   at System.Runtime.Remoting.Messaging.MessageDictionaryEnumerator.get_Value()

jamiegs

At the same time it failed there was a message in the mongos log. 

2013-04-22 21:00:08,833 ip-0A2057A3 [WARN ] [Hudl.Config.ClusterConfigFileUtility] Timed out acquiring the configuration lock for key logger.Enyim.Caching.Memcached.PooledSocket during a GetConfiguration() call


--Mongos Host Error--

Hudl mongos service failed during startup:
Type = System.NullReferenceException

jamiegs

        /// <summary>
        /// Tests that a 200 is returned when visiting a page that doesn't exist.
        /// </summary>
        [TestMethod]
        public void ReturnsOkStatusCodeWhenExists()
        {
            var bs = new TestBootStrapper();
            var browser = new Browser(bs);

            var result = browser.Get("/users/", with => with.HttpRequest());