Skip to content

Instantly share code, notes, and snippets.

@strawberryjello

strawberryjello/ci.org

Last active August 12, 2016 05:43
Show Gist options
  • Save strawberryjello/7d6279f6186e11a55eba56c616093de0 to your computer and use it in GitHub Desktop.
Save strawberryjello/7d6279f6186e11a55eba56c616093de0 to your computer and use it in GitHub Desktop.
Download ZIP
Testing and CI talk
Raw
ci.org

Continuous integration (CI)

CI step-by-step for a single feature

  • checkout latest version of source code from mainline/master
    • verify first that the build is passing – if it’s failing, fix it first
  • develop feature
  • build source locally (in developer’s working copy)
    • compilation/producing a running version of the app
    • run tests
    • this step should be automated
  • when build passes, integrate changes into mainline (eg, push to master)
  • integration machine runs build
    • if conflicts occur from different working copies (eg, merge conflict from different devs), build fails – fix build
    • if build fails for any reason, fix build

Benefits of CI

  • bugs are found early because of frequent builds
  • especially useful for any team with more than one person – conflicts found as soon as they are introduced
  • reduced risk: at all times you know what works, and what bugs are present
    • this depends on how good your test suite is
  • frequent deployment -> rapid feedback

CI Practices

  • single source repository
    • rule of thumb: everything required for a build should be there
      • source, tests, config/property files, etc
      • security caveat for some files
    • exclude files generated by the build (eg, class files, executables)
  • automate the build
    • automate every step of the build – ideally requiring only one command (eg, Rake)
  • self-testing builds
    • TDD is a good way to produce tests, but not necessary – what’s important is that there are tests
    • test failure should fail the build
  • integrating the mainline every day
    • ie, pushing to master every day
    • requirement: build must pass before changes are integrated
  • every push should trigger a build on the integration server
    • CI servers automate this, as well as notifying everyone of the build results
    • manually triggering the build is also an option – it depends on the team’s discipline
  • keep the build fast
    • XP guideline: max 10 minutes
    • usual bottleneck: tests, particularly involving the DB
    • deployment/build pipeline
      • commit build: the fast one
      • slower builds can run slower tests (in parallel on other machines) regularly
  • test in a clone of the production environment
    • the more differences between the test/staging and production environments, the more possibilities of environment-related bugs going unnoticed
    • try VMs
  • everyone can see what’s happening
    • everyone should know if a build failed or passed
    • tools like Jenkins provide logs, statistics, and other data about builds that illustrate the status/health of projects
  • automate deployment
    • consider automated rollback for production
    • eg, Capistrano

What about pull requests?

  • in this case, the pull request must pass the build before being merged, which helps keep production environments pristine

Examples of CI services

Hosted

You set up the hosting yourself

Continuous delivery (CD)

You’re doing Continuous Delivery when

  • Your software is deployable throughout its lifecycle
  • Your team prioritizes keeping the software deployable over working on new features
  • Anybody can get fast, automated feedback on the production readiness of their systems any time somebody makes a change to them
  • You can perform push-button deployments of any version of the software to any environment on demand

Requirements

  • Build automation and Continuous Integration
  • Test automation
  • Deployment automation

Continuous Deployment vs Continuous Delivery

  • Continuous Deployment means that every change goes through the pipeline and automatically gets put into production, resulting in many production deployments every day
  • Continuous Delivery just means that you are able to do frequent deployments but may choose not to do it, usually due to businesses preferring a slower rate of deployment

Resources

Raw
recommendations.org

Recommendations

CI: where to start

  • automate the build
    • everything needed must be in source control
    • consider nightly builds
  • add automated tests
    • start with the features that regularly need to be fixed
    • building up the suite will take a while
  • speed up the (commit) build: 10 minutes or less
  • new projects must start with CI

For features that can’t be automation-tested for whatever reason, manual testing is a MUST

  • as much as possible, get someone else to test your code – that person will not have the same biases that you do as the author of the code, and will probably uncover bugs you’re blind to
  • when testing, try to break the app – use invalid inputs, do unexpected things with it, etc
    • for bugs found, listing down the steps you took before encountering them and including screenshots or stack trace snippets will speed up debugging
  • if you have enough time before a release, consider setting aside a day before the release date for regression testing – avoid squeezing in last-minute changes to minimize possible screw-ups
    • do testing on a test or staging server that is as close as possible to the production server

Include tests when fixing bugs

  • bugs in production -> there are missing tests, or one/some of the tests is/are faulty
  • if written well, the tests will prevent those bugs from reoccurring without being found
    • the alternative is manually testing for those bugs during each release

Write tests before refactoring/modifying anything

  • for refactoring, write tests that show the original functionality hasn’t changed after you refactor it (if those tests don’t exist yet)
  • for modifications, write tests that show how the modification is supposed (and not supposed) to work
  • especially for legacy apps – this implies familiarizing yourself with the original functionality before attempting to change it
    • acceptance/feature tests may be more useful than unit tests in this case

Do not release/deploy on Fridays

  • Murphy’s Law: if anything can go wrong, it will go wrong
  • if something breaks, you will wind up doing overtime to fix it
Raw
testing.org

Intro

Why test?

Background - Agile, XP, and TDD

  • the agile processes arose as a reaction to plan-driven methodologies – a lightweight compromise between no process and too much process
    • 1990s, the era of object-oriented programming languages (eg, Java, C++)
  • extreme programming (XP) is an agile process that aims to deliver quality software within short time frames
  • quality software is code that meets the specifications, or the customer’s requirements
  • you cannot guarantee quality software delivered in a short time frame if you have no way to verify that it meets the requirements
  • XP puts a heavy emphasis on testing: tests must be written before the code, not after
    • XP requires the programmer to write tests for her own code, instead of relying on testers

12 Practices of XP

  • planning and requirements (user stories)
  • small, incremental releases
  • system metaphors
  • simple designs
  • continuous testing (TDD)
  • refactoring
  • pair programming
  • collective ownership of code
  • continuous integration
  • 40-hour work week (ie, sustainable pace)
  • on-site customer
  • coding standards

Source: Extreme Programming Explained, 1st edition, 1999

Benefits of automated testing

  • tests, especially automated tests, are necessary when modifying existing code, either via adding new functionality or refactoring
    • they are your safety net: they help you catch regression issues, and prevent you from breaking working code
    • when the test suite is automated and easy to run, you’re more likely to run them
  • tests are a record of acceptable software behavior
  • writing tests helps you establish which behaviors are valid and invalid for which inputs and outputs
  • writing tests helps you understand the specifications/requirements better

TDD, ATDD, BDD

Test-driven development (TDD)

  • goal: produce well-designed, well-tested, well-factored code in small verifiable steps
  • the cycle: think, write test (red bar), write code to pass test, run test (green bar), refactor
    • the resulting design/architecture of the code base will depend on the nature of the refactoring
  • running tests multiple times while coding -> these tests must be fast
    • rule of thumb: the entire suite (excluding slow tests, eg for performance testing) should take no more than 10 minutes
  • steep learning curve: it could take a few months to adjust

Acceptance test-driven development (ATDD)

  • traditional acceptance testing:
    • customer/user meets with analysts to create user stories and acceptance tests based on the stories during the planning/requirements gathering phase
    • once the application is ready, the customer performs the acceptance tests, prioritizes any bugs/issues found, then reports to the analysts
    • after the bugs are fixed, the customer performs the acceptance tests again as a form of regression testing
  • ATDD focuses on automated acceptance tests written before the code itself
  • associated with specific tools (eg, Fit/FitNesse)

Behavior-driven development (BDD)

  • meant to augment TDD/ATDD, synthesizes/refines their practices
    • introduces terminology meant to be closer to business language, which spurs conversations between customers and developer teams that will increase understanding of the domain
    • given-when-then format for acceptance tests: “Given <a context>, when <an event happens> then <an outcome should occur>”
    • documentation can be automatically produced from scenarios/test cases via tools
  • emphasizes the business value of each user story
  • examples of tools: Cucumber

Testing in Rails

Minitest, RSpec

Minitest

  • included by default in Rails
    • Rails can auto-generate unit test stubs for most classes
    • test data by default provided by fixtures, which are another Rails built-in
  • syntax similar to that used by the xUnit family (eg, JUnit)

RSpec

  • syntax closer to natural English
  • better organization of test cases
  • examples of usage usually focus on factories such as FactoryGirl for test data instead of Rails fixtures, but fixtures are configured by default

Rails test types

Unit

  • the definition of what constitutes a unit differs – sometimes it’s a single class, sometimes it’s a group of classes, sometimes it’s a group of methods within a class
  • in general, a unit is tested in isolation, ie without involving other units
    • depending on preference, this may involve mocking units that provide needed functionality (eg, remote services/APIs, databases)

Functional

  • a form of integration test (ie, a test that checks the integration of different units) that tests an end-to-end piece of functionality
  • in Rails, controller tests are considered functional tests

Acceptance/Feature

  • automates the actions of a user via the app’s UI
  • tools can open a browser to simulate user actions such as clicking, or do headless testing without opening a browser
  • Rails tools: Capybara, web drivers (eg, Selenium, Poltergeist)

Resources

Background

Testing

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment