Your Build Smells

@pimterry

Your build

Who cares?

Gardening
not architecture

PAINFUL MERGES

Conflicts

are

avoidable

(usually)

Many of these conflicts are avoidable
Usually syntactic conflicts: two independent changes in the sme code
Both adding two fields, but on the same line

Sometimes semantic conflicts: two interacting changes, but easily reconcilable once you know both are required
Slightly changing the return type of a method, and using the method elsewhere

Rarely fundamental design conflicts: you want to do X, I want conflicting Y, X+Y are mutually exclusive
Changing the return type of a method to two drastically different things

Everything but fundamental conflicts is resolvable earlier, potentially invisibly
Syntactic and simple semantic conflicts would be unnoticable if we had make the changes in sequence.

Isolation

Feature flags

Branch for a reason

Painful merges

SMALL

FREQUENT

EASY

MERGES

NERVOUS PUSHES

Just push it
and see

BUILD FAILED

Run it again

Confidence

Accuracy

Thoroughness

Speed

Simplicity

Accuracy

No dependencies
Identical environments
Zero tolerance

Your build needs to be accurate: if the code works, it should pass
Consistently, and reliably, with no exceptions
If working code fails and messes up your flow, you run the build as little as possible
And you ignore actually useful failures when they happen

Unit tests should need nothing from their environment at all
Integration and system level tests might, but try to limit as much as possible.
Any external dependencies make your code worse
Bundle DBs/servers/service stubs with your codebase, start them in the tests

Identical environments - tests should run in identical contexts all the time
Vagrant, Docker, etc can help with doing this for dev environment
Unit tests should have no dependencies, should be trivial

Zero tolerance for failure
Occasional failures are poisonous: seems cheap, but undermines quality completely
As soon as you have any, you start disconting failures, and being nervous about ok builds
Chase them disproportionately
Better to have no tests than intermittently failing tests (if you can't fix them)
PHP - 5.3.7

Thoroughness

Monitor coverage (a bit)
Cover every risk somewhere
Test what might fail

Your build needs to be thorough: if it passes, the code works
Need to be confident, or you can't share your code with others
If build isn't thorough, you'll have to be manually thorough
You won't be

Make coverage visible in your team (monitors), and preferably your tools (NCrunch/Wallaby)
Coverage isn't perfect, don't chase arbitrary targets
But it is useful information, and overall trends and areas covered is important

You don't have to test every case at every level
If you have unit tests thoroughly checking certain behaviour, you probably don't need system tests too
Instead have system tests that do an overall chcek that it all works, and leave the detail to the unit tests

Don't test trivialities
If there's no way this test is ever going to fail, don't write it
Usually that means, test the same functionality, but further up the stack
Up is more complicated, so there's more to break
This is purely for QA; if you like TDD, TDD everything, but not for testing alone

Speed

Quick feedback
Watchable unit tests
Smoke test suites

Tests need to be fast
So people can get local confidence, and fast feedback from success or failure remotely
Makes pushing easier, reduces context switching

Fast feedback is the name of the game
See teams with builds that run for 4 hours, and then come back and tell you every single test failed
That would've been good to know 4 hours ago
Automating the smoke test suites as your first build step can help here
Successful feedback is good too; lots of small steps, not a single big one, to expose success en route and give more confidence

Unit tests should be of a watchable length. Order of seconds
If they're not, then I'm not going to run them very often, and that worsens my feedback loop
Think about syntax errors; if they took 5 minutes too arrive, how much slower would you be?

Integration and system test suites are going to be slower; fight anyway
Smoke tests can make help: predefined suites of a few quick tests
That cover most of the functionality at a basic level
One test per page, say
Rapid local feedback of obvious mistakes
Most test runner have some kind of tagging/categorization feature

Simplicity

One click
Zero setup
Easy debugging

Simple builds are easier to have confidence in
Easy to understand what it does, and thus to trust it
Easy to manage yourself, to check what the build is going to do

Run with one click/command
Very important when you have new people joining the team. Otherwise they'll do it wrong
Even with lots of people: I don't run builds where I have to run 10 different things
I pick the 'important' ones, and then I'm wrong, and the others break
Single click makes it really really easy for people to do, so they do it
Back to usability again: remove barriers to good behaviour

Zero setup
If you have setup, you have to remember it
When you try and do it, you'll do it wrong
Slightly different local environments are a recipe for absolutely zero confidence in the build
Again, vagrant/docker and ansible/puppet/chef/all of devops can help

This needs to be easily debuggable locally.
If it's not, when it breaks it's easier to fix the 'obvious' bug, push again and see than to really solve the problem.
Guide your users to the right choice

Nervous pushes

TESTED

QUICK

CONFIDENT

PUSHES

RARE RELEASES

Releasing
creates
value

Releasing
SOONER
creates
more
value

2
releases per day

100s
releases per week

100s
releases per day

Why not?

RISKY

HARD

Reduce the risk

Consistent environments
Automate everything
Measure everything

Kill the risk

Ensure your environments are the same everywhere
Like we talked about for accurate tests earlier: consistency gives confidence

Automate the hell out of this to make it happen
Very few excuses for this: any deployment steps easy enough to do manually should be very easy to do automatically
Disproportionate benefit to this kind of stuff
Not only your time, but improve overall quality, makes other pain more obvious
Quicker more-automated deployments are easier to do more often
So releases improve more quickly, and so does the process itself
Tighten the feedback loop

Measure everything
All four of the companies measured talk constantly about the different metrics they track
Message tends to be: publicize graphs of important stats, have everybody know them
Ensure people react when they see them deviating
Things like A/B testing and progressive rollout makes this easier, but isn't necessary

Make it easy

Automate everything
One click (or fewer)
Everyone deploys

Make it easy

Again, automation automation automation.
There's theme here
People don't like putting in brainless effort, and they aren't good at it
Computers do, and are. Delegate

One click deployments: once everything is automated this is very easy to do
The only argument against is people might accidentally deploy
If you've got a build that won't set up broken code, and you trust your build, that's not a problem
Again, usability; if you want people to deploy lots, make deploying trivial
Design your build to guide people toward good behaviour

Have everybody do it, no special priviledges
Github for example, everybody does a deploy on their first day
Lots of downsides to only one person knowing how to deploy
Bus factor
Biggest is that it makes deployment process one person's responsibility
Time they spend improving it is selfish
Deploying is the only point where your team adds value
Everybody should be invested in that

If it hurts, do it more frequently
Jez Humble

Great quote from Jez Humble, author of Continuous Delivery ### FULL NAME?
Applies to lots of this talk, but particularly deployment here
If you rarely do deployments, they're huge and painful and scary
So they break things
So you deploy less.
You have to break the loop

Automatically deploy every commit
Not necessarily successfully at first
But immediately gives you a very tight feedback loop
Highlights issues immediately, check improvements help immediately

This is scary, so start with just development environments
Low risk, easy to experiment
Do need to value their quality though, treat them like production envs

Nightly deploys help too
Easy for builds to rot
Running the pipeline every day stops this
Even if not rotting, it adds extra frequency
Known good build, should always pass

Rare releases

EASY

LOW-RISK

FREQUENT

RELEASES

Improvements

are

CUMULATIVE

Gardening
not architecture

Your Build Smells

@pimterry