Building and

testing

docker containers

 
@sublimino
 Delivery Engineer
News UK, Visa, British Gas
www.binarysludge.com

Connected boilers


codes


Containment

data

efficacy

   
 

containerised component types

  • Queue
  • Datastore
  • Black box
  • Microservice
  • Infrastructure

design Principles

  • Continuous integration/deployment
  • Deterministic pipeline
  • Favour simplicity
  • JSON everywhere
  • Constant cycle-time optimisation
  • Partial or whole stack deployable 
    on developers' machines
  • Containers identical from dev
    through test to production

building containers - shared code

  • Shared code repository
    • npm module
    • Present in every repository/image (even couchbase/redis)
    • Contains linting rules, extensible task runner, read-only credentials
    • Idempotent npm post-install task creates test directories and manipulates filesystem
    • Any change triggers cascading builds in dependent jobs

building containers - base image

FROM ubuntu:14.04
MAINTAINER Andrew Martin "sublimino@gmail.com" ENV APT_PACKAGES curl build-essential wget git
ENV NPM_PACKAGES forever gulp http-server
ENV NODE_VERSION 0.10.29
# apt and npm installation
#
RUN apt-get update && apt-get install -y $APT_PACKAGES && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
RUN curl https://cdn.rawgit.com/isaacs/nave/master/nave.sh > /bin/nave && \ chmod a+x /bin/nave && \
    nave usemain $NODE_VERSION;
RUN npm install -g $NPM_PACKAGES;
# configure private npm
#
ADD ./node_modules/cb-shared/npmrc /root/.npmrc

BUILDING CONTAINERS -  Task runner

  • Stream processing and parallel task execution
  • Abstract invocation of language/library specificities
  • Tight dev feedback loop, reduced cycle-time

BUILDING CONTAINERS -  TASK RUNNER

Abstracts docker, fleet and test commands
$ gulp <tab> 
browserify              fleet:restart          test:functional
build                   fleet:start            test:functional:remote
build-properties        help                   test:functional:watch
build:watch             less                   test:integration
default                 lint                   test:integration:remote
docker:build            livereload             test:integration:watch
...
  • 4 different test runners
  • Defers test execution to cross platform BASH scripts
    • BATS acceptance tested
      • https://github.com/sstephenson/bats
    • OSX vs Linux - some GNU-ish complexities
  • Build server has no logic

building containers - a microservice

FROM harbour.example.com/cb-baseimage:latest
MAINTAINER Andrew Martin "sublimino@gmail.com"
# create installation directory
#
RUN mkdir -p /opt/d2
WORKDIR /opt/d2
# add package.json and npm install
#
ADD package.json /opt/d2/
RUN HOME=/root npm install ADD ./bin /opt/d2/bin
ADD ./lib /opt/d2/lib
ADD ./test /opt/d2/test
ADD exemplary.crt /opt/d2/exemplary.crt EXPOSE 4000 ENTRYPOINT ["bin/server.js"]

building containers - in summary

  • Manage dependencies via NPM
  • Common task API
  • Jenkins has no hidden logic 
  • Same containers from dev to prod

testing, testing, 123

testing a container

  • Isolate the component
  • Ensure its veracity as early as possible
  • Granular reporting of failures

components - Datastores



COMPONENTS - Queues

COMPONENTS - Black box


COMPONENTS - Microservice



COMPONENTS - other


TESTING PROCESS - 

unit/integration

  • Unit: "Smallest piece of testable software"
  • Integration: "Verify communication 
    paths and interactions"

  • Each runnable from outside the container
    for tighter feedback loop
  • But run inside the container to test as
    close to production as possible

Testing process - functional/acceptance

  • Functional/Component Integration: 
    "Limited component test using test doubles"
  • Acceptance/E2E: "Verify system meets
    external requirements from end to end"

  • Global setup/teardown and test override hooks
  • External services brought up with fig
  • Each repository's fig.yml configuration file:
    • Defines the component's invocation
    • ...and that of any immediate dependencies
  • (database fixtures in a separate image)

TESTING PROCESS - fig.yml

  • fig.yml  of c omponent under test is copied
    to  fig-test.yml
  • Dependent components'  fig.yml  files 
    merged into  fig-test.yml
    • Brings in transitive dependencies
  • Environment variables are templated
    • NODE_ENV 
    • Database names
    • IPs and ports

    fig.yml FOR "NHS" MICROSERVICE

    redis:
      image: harbour.example.com/cb-nhs-redis:latest
      ports:
       - "6379:6379"
    ... nhs:   build: .
      # image: harbour.example.com/cb-nhs:latest
      environment:
       - CBNHS_LOGLEVEL=debug
       - CBNHS_DBBUCKET=cb-api - CBNHS_PRE_AUTHENTICATE=0 ...
      ports:
       - "58000:8000"
      links:
       - redis
       - couchbase
       - api

    FIG-test.YML FOR "NHS" MICROSERVICE

    redis:
      image: harbour.example.com/cb-nhs-redis:latest
      ports:
       - "6379:6379" nhs:
      build: .
      # image: harbour.example.com/cb-nhs:latest
      environment:
       ...
      ports:
       - "58000:8000"
      links:
       - redis
       - couchbase
       - api couchbase:
      image: harbour.example.com/cb-couchbase-server:latest
    ...
    api:
      image: harbour.example.com/cb-api:latest
    ...

    in the test Setup script...

    • fig --file=fig-test.yml \
      --project-name cb-nhs \ up &

    • fig ps,  HTTP health checks and  /health   
      RESTful endpoints are used to determine availability

    • Polling preferred to sleeping                 
    • Any other necessary pre-test config
      • API oAuth authentication
      • Endpoints for exposing tunnels to Sauce labs 
      • JS/CSS build chains

    test demo


    actual test demo


    POST-TEST

    • On build server test completion, image is tagged  
      • :latest-dev                                   

      • :20150106-42 (date-jenkinsBuild#)

    • Tagged image pushed to private docker registry
              


    system integration test

    • Production-like environment 
    • Test that the interfaces between
      components match or "hang together"
    • Tests parts of the system not 
      covered by component tests
      • infrastructure
      • routing
      • systemd units
      • support containers
    • On completion, non-functional tests 
      are run in an isolated environment

    non-functional testing

    • Same configuration  as system
      integration environment

    • Tests
      • Load/capacity
      • Stress
      • Resilience

    container Deployment

    • Current design informed by Kubernetes
    • Automated by a Deployment Bot
      • Journalling service manifest
      • Service discovery via etcd
      • HAProxy routing via confd
      • Rolling updates
      • Cluster/service health monitoring

    a note on logging

    • systemd logs container output to journald
    • Every CoreOS host runs a 
      logstash-forwarder container
      with /var/log/journal
       mounted inside
    • Forwards logs, buffering in case of failure
    • Logstash runs on another host, decoding 
      incoming JSON and munging various
      fields

    Logstash Config - pt 1

    input {
      lumberjack {
        port => 54321
        codec => "json"
        ssl_certificate => "server.crt"
        ssl_key => "server.key"
      }
    }

    filter {
      mutate {
        remove_field => [ "__CURSOR", "__MONOTONIC_TIMESTAMP" ]
      }
      if [type] == "stdin" {
        mutate {
          # microsecs to millisecs
          gsub => [ "__REALTIME_TIMESTAMP", "\d{3}$", "" ]
        }
        date {
          match => [ "__REALTIME_TIMESTAMP", "UNIX_MS" ]
        }
      }
      # (continued on next slide...)

    LOGSTASH CONFIG - PT 2

      # (continued from last slide...)
    
      # if messages start and end with curly brace, interpret as json,
      # except messages from the logstash container, which do start and
      # end with curly brace and are NOT json!
      if [MESSAGE] =~ /^\{.*\}$/ and [_SYSTEMD_UNIT] !~ /^logstash/ {
        json {
          source => "MESSAGE"
          target => "json"
        }
      }
    }
    output {
      elasticsearch {
        host => "${ES_HOST}"
        port => ${ES_PORT}
        protocol => "http"
      }
    }

    BRING PAIN FORWARD: 

    deliver continuously




    Questions?





    @sublimino
    www.binarysludge.com