PNSQC

gistfile1.txt

Netflix Keynote:
@caseyrosethal

AWS - three regions. (US-W, US-E, EU)
300 + Microservices :: Dev ownership includes Deployment and Uptime

Software is getting more complex:
--how we interact with software
	mobile. IoT. rapid rate of change. 

--what software does
	natural language processing. self driving cars - algs more driving exp than any human.
	Artifically Incomprehensible -- AI Algorithms, black-box algs, you can't inspect the alg as a human.
	"Most precious resource will be 'getting your attention' -- relevance weight" (Recommendation Nexflix queue.)
	writing a test to tell me which date is more important -- interview @ noon, or anniversary.

--how we build software
	docker, aws, github, etc. -- off-loading different steps to different teams, no one person can know the whole stack.
	netflix many hundreds of deployment to production per day.

Netflix Test Suite (NTS)
	Manufactors allowed to test their devices against "prod" netflix stack.
	"Search goes down - Search box doesn't show up, doesn't bring down whole site."
	Tests don't pass when you want to test when search is down.


"Explosion of Complexity"  -- Humans needed for quality assurance.

Architect -- Fit all pieces of product together in their head. No human can do this now, can't track all moving pieces. 

No release team at Netflix. "Only hire good engineers" /troll.


Classical Approach to Engineering. 
 x => f ~> y = f(x) :: Given X input into F, we'll get output we can assert.


Beer Game.
System failed even when making logical rational choices.
Microservices can lead to shit results even when making logical and rational choices.
"Intuition Engineering" -- Human can build intuition when looking at a UI. 

Pain suit (service down, feel pain on body) - heath of the system by how you feel.

Flux (netflix) -- AWS Regions. Requests to each Region. Good, Bad (500), Failure (302) - Intuit info without looking at the actual numbers.
https://github.com/cfregly/fluxcapacitor
http://techblog.netflix.com/2015/10/flux-new-approach-to-system-intuition.html




Chaos Monkey
Engineers build to the standard of being resliant to server going down, and chaos monkey proof.

Chaos Kong
Kills full AWS Region. 

principlesofchaos.org
Chaos Engineering is the discipline of experimenting on a distributed system in order to build confidence in the system’s capability to withstand turbulent conditions in production.

Finding bugs at the edge of the system. 
Building experiments to find those edges.

"does the system work, not how it works." measure of the steady-state of the sytem.
Vary real-world events. (storms, giant pot holes, sudden rush of users, good and bad)
Experiment in Production. "The user is never wrong" -- paths in the grass.
Automate experiments to run continuously. 




------------------------------------------------------------------------------------------------------------------------------------------

Selenium Case Study
Alan Ark aark@eidpassport.com

Watir/Ruby

WebDriverWait -- wait until finding element.
Expected Conditions.
Use Page Objects -- base objects LoginPage.Login
Verify location on transitions (clicked links, form submission, ajax)
IE -- Use SendKeys(enter) instead of Click. (sometimes click just doesn't work)
Frames -- find element 'frame', switchto 'frame'
AutoIT Script -- dealing with login prompts, win32 objects. https://www.autoitscript.com/site/



------------------------------------------------------------------------------------------------------------------------------------------

Variability vs. Repeatability
Jonathan Li On Wing - Groupon


------------------------------------------------------------------------------------------------------------------------------------------


Version Controlling Jenkins
Wayne Warren - Puppet Labs.
https://git.openstack.org/cgit/openstack-infra/jenkins-job-builder/

Jenkins Job Builder -- JJB -- Python command line tool, generates jenkins yaml

Engineering :: Automation, Tooling, Infrastructure
Assurance :: Test Planning, Risk Analysis, Writing Tests

(Watch ~> Build ~> Test ~> Publish Results).loop

Github jenkins integration?

Yaml -> xml -> jenkins
Run specific jobs in job group by passing in GLOB

Project ~> Job Group ~> Job Template ~> Macros

Name jobs with Regex-based job views

Job Types:
Unit/Lint/Coverage || Packaging || Acceptance/Integration || Release

Merging pull requests, running master and develop and return diff.

Jenkins View Management ( define views with yaml )

------------------------------------------------------------------------------------------------------------------------------------------

Road to Continuous Delivery @ Tango.
Amit Mathur

Tango messaging platform -- communications, social, content

Commit Code ~> Unit/Feature Test ~> Deploy to Testing ~> Acceptance Testing ~> Deploy to Production.
----------CI----------------------
----------------------------------Continuous Delivery-----------------------
-----------------------------------------------------------------------------Continuous Deployment--

Monthly Release Cycle :: 2 week production client bake time.
Minimize regressions due to new servers deployed in production.


Deployment Automation || Continuous Testing || Test Intelligence

Testing Env Utilization
Dev (production mirror) ~> Internal (All Dev Feature branches combined + Prod) ~> UAT ~> Staging ~> Production

Deployment Automation (mosh, python scripting for AWS)

Restful Tests.
	Environment Mobility -- run in any enviroment including localhost
	Non-intrusive -- should clean up all garbage created by test.
	Security -- Creds leak, testing test in production, production in test.


Putting teams against each other -- Test Coverage, Test Execution, Test Quality, Product Quality || Total Score.

Limbo Sytem -- Quarantine.
	Test failure creates JIRA ticket.
	If that failure was analyized? good/bad
	Not analyzied puts it into Quarantine, or False positive > 90% 




------------------------------------------------------------------------------------------------------------------------------------------


Legacy in an Agile World.
Ray Miller -- NTT Data.

Solution Design Quality and Assurance.
	Design Quality.
	Design Assurance.

USCIS - Department of Homeland Security :: Lawful Immigration into US
E-Verify and Electronic Immigraiton System -- Huge Applications.
Release Timeframes have gone from Yearly to 3 weeks.

Specs vs. Customer Usage
<- QA | Delivery Assurance ->
TOEGAF - Enterprise Architecture software


------------------------------------------------------------------------------------------------------------------------------------------


Toolbox Testing Approach
James Gibbard - Intel

Goals :: Flexible, Pragmatic and High Quality

Thin Vertical Slice -- Off the shelf products are not flexible enough.

Use the robot framework......
Create remote wrapper in any language that'll talk to Robot.
Analyize screen with image -- MIT project (http://www.sikuli.org/)



------------------------------------------------------------------------------------------------------------------------------------------


Keynote
Build the Wrong Thing Faster and Learn from It.
David Hussman -- Devjam Studios.
@davidhussman 

Dudes Law :: Value = Why / How
Show 100% of 25% of the promised value.
Waterfall Paper (had agile feedback loops)

The longer time is (deadline) more often the goals will change, and more likely we'll be wrong, even if we were right.
Can't really estimate something you've never done.

"We need to assume most of what we're doing is wrong."
If you don't know where you're going it's really easy to itterativyly get there.

Levitin - the Organized Mind. (how we originize our mind)

from "How Much" (progress) to "How Wrong" (product)
Record users using the product and showing that to developers.


Planning to Discover and Learning from Delivery.
"Give examples of why we care about this feature" -- when setting up stories. Obvious to Complx examples


Graphing :: Cross Cutting Constraints (x) Product Learning (y)

The Checklist Manifest -- Atul Gawande

(Valuable - Desirable - Feasible) -- ven diagram of what should be done.

Checklist to know when we're ready. (broken, fixed, new)
	-story
	-features
	-tests
	-ui
Bugs == Mistakes. (changes the tone of the discussion)

How to Measure Wrongness.
 -Sonar.
 -Visualizing Quality Issues (pass/fail builds)
 -To Do | Doing | Done | Valuable?

Founders at Work - Jessica Livingston
Escape Velocity - Geoffery A. Moore

------------------------------------------------------------------------------------------------------------------------------------------

The Systems Thinking Tester. Leverage and Influence
Russel Smith

Today's problems come from Yesterday's solutions.


------------------------------------------------------------------------------------------------------------------------------------------

Using Machine Learning to Perdict Test Outcome.
Wayne Roseberry - Microsoft

Nobody can change code if there are failing tests.
2k/3k Developers all running tests every day - all failures get loaded into bug tracking system.
New failures create bug reports, or appends to old bug report.

Supervised learning - trying to perdict what bug failures look like. 
	-have bug report, trying to learn which ones look like they will fail, and match against actual bug reports

Figuring out what to use to predict the failures determine probability. (postponed, fixed, duplicate, etc...)
	-Figures out the difference between what it thought vs. what it actually was.
	-Updates weight or whatnot when it learns.

90 days of Log Data :: 80% Train / 20% Testing
 	-work on ETL of log data.
 	-failure message + stack trace.

Recall (97/100 cats) vs. Percision (120cats, only 97 really cats)

"When we see a certain stacktrace, we're tending not to fix it."

If test fails, have it re-run in a different state.


------------------------------------------------------------------------------------------------------------------------------------------


Moving up the Product Security Maturity Model
-- Intel Security


------------------------------------------------------------------------------------------------------------------------------------------

Misuing the Type System for Fun and Profit
Ian Dees @undees

Code Review - Unit Tests - Exploritory - "Just be Careful"
"Lean on the Type System"

Can't delete shared pointers. C++ 11

Denotational Design :: When developing software - relate them to precise and familiar semantic models.
	-sketch out
	-implement it in your language of choice

Sketch out API in "mathy" lanugage - Idris

If you can prove via a type - you've proved something.
Proof by Induction :: If it's true for zero, then it's true for x for x + 1

Make different class types for Matrix col/row so it's impossible to call method out of order.

David Sankel 'Intellectual Ascent to Data' C++ Now 2013 -- youtube.com/watch?v=vy5C-mlUQ1w


------------------------------------------------------------------------------------------------------------------------------------------