Automated Tests for Database Procedures

Why not future-proof your database procedures just as the middle tier and front-end developers have been future proofing theirs? Not only can automated unit tests be built for each procedure but Test Driven Development (the practice of writing a simple unit test that fails and forces you to implement some simple procedure to satisfy the failing test, and once the test passes you enhance the test or add a new unit test to grow the procedure further) can be done as well.

Why Unit Test Procedures?

Every time there's a change in a procedure or schema, unexpected errors can happen. To mitigate disaster, you'll have been doing some kind of testing: manual testing, automated testing, ... 

An early adopter's mistake is to test every possible case you can imagine. For unit testing, this isn't your test target. You're goal is to have every line of procedure be necessary to pass your test procedures. Although this means that some lines of code are tested as a side affect, and that's OK.  When constructing your unit test, focus on testing only the one procedure you're targeting.

Confidence

Without unit tests whose job description is to test each stored procedure, then you're gambling that the testing in other tiers will uncover problems. Why live with this uncertainty? If you have a test suite that confirms every line of procedure works as expected, you'll have supreme confidence that your next release is truly an improvement (rather than new features with new bugs). 

Fast feedback

Since we're writing unit tests that test each procedure in isolation from other procedures (or at least as isolated as possible), then the tests will execute quickly. Likely you can test 100 procedures in a minute which scales to thousands of procedures in thirty minutes. In fact, if you run these tests continuously (triggered by code checkins) or whenever the you feel some uncertainty, you can execute the test suite to ask the question, "do I have a regression?" and get back the answer in less than a minute.

The tests "tell you" the regression's location

With individual unit test (test procedures) designed to test a stored procedure in isolation, you'll have signaling that shows what procedures are fine and which are in failure. This means once there is a regression it the tests will indicate approximately what procedure or procedures to examine.

Tools for Unit Testing Database Procedures

Here are a few that either I've personally used or I read through the documentation and meets my minimum feature list (asserts, pre test execution routines, post test execution routines, outputs a report, allows easy execution of entire test suite).

Oracle

PLUto

utPLSQL

(PLUnit looked promising but isn't maintained any longer, which makes it unusable since it's closed source.)

Oracle SQL Developer has unit testing tools built in (view->unit tests).  It's a bit complicated.  The learning curve to use this tool is higher than PLUnit.  Here are some links about this tool:

http://www.thatjeffsmith.com/archive/2014/04/unit-testing-your-plsql-with-oracle-sql-developer/

http://www.oracle.com/webfolder/technetwork/tutorials/obe/db/sqldev/r30/sqldev_unit_test/sqldev_unit_test_otn.htm

PLUTo is a minimalist setup.  utPLSQL has a rich API.  The Oracle one is hard to get into because Oracle's tech writing is so dull and uninspiring.  And the last time I tried to use it, the tool was disabled for some reason (license issues?).  My favorite is utPLSQL.  It's made popular mention on Steven Feuerstein's (of Oracle) blog as well.  You can install it in minutes and start with the examples.

T-SQL

T.S.T. Sql is a nice simple tool with some getting started videos and an InfoQ article.

Sql Server Database Test (SSDT).

Common Test Environment Configurations

To execute automated tests for a DB means you need a DB instance to put the procedures inside of and data for the procedures to execute.  As for data/schema, this should be created (inserted) as part of the automated test.  To answer the question of how many DB instances, there is a range to this answer:  You need enough. :-)

Usually you need:

• 1 instance for continuous integration, since you're going to want to execute your automated tests continuously.
• At least one instance for someone to develop automated tests.  Typically, every developer creates an instance on their own system whenever they want to execute, debug, or create automated tests.  If for some number of reasons you can't have an instance for each developer, then you'll need to manage them as a pool.

Test Design

The above tools add some supporting procedures so you can focus on creating automated tests in the form of test procedures.  The job of unit tests is to confirm that the code under tests operates as the developer intended it.
Good automated unit tests have the following characteristics:

• each test executes in microseconds
• independent of execution of other tests (said another way, you should be able to execute the tests in any order you wish)
• prefer having many simple test procedures to test a DB procedure (which are easy to understand and maintain) rather than a few test procedures that check many thing.
• use test data which is as simple as possible--just enough to make the procedure under test happy.
• test data should be created either by the test itself
• test procedure names should express: what procedure they are testing and the test scenario
• each of DB procedure under test should be tested in isolation (without dependency on other procedures)

System level tests are supposed to answer the question, is feature XYZ working as the user expects it?  These tests will work with large data sets and developing system level tests:

• each test executes seconds, minutes, or longer
• independent of other tests
• destructive tests should undo their "inserts" so as not to affect other tests (could be implemented by either refreshing the data afterwards or not committing the transaction).
• prefer creating tons of non-destructive tests and few destructive tests so you don't need to often refresh the data which is slow
• keep destructive tests which require data refresh in a separate test suit so you have a "fast" suite for continuous builds and your "slow" suite for hourly or nightly builds.

Test Data Creation

Ways that test data is created are: 

• restoring to the database a set of test data.
• sql script that inserts the data.
• sql code that creates test data that is used by your test procedures

Your DB instance creation along with schema should be automated as well so that any developer on the team can easily create a DB test environment and so your continuous integration system can recreate the test environment each time it executes.

Other Links

Best practices of testing stored procedures.

Recommended Behavior Driven Development (BDD) Toolsets

As a consultant who works with many different teams on mostly .Net,  Java and JavaScript projects, let me recommend my favorites.

Cuccumber versus JBehave
I don't have a clear favorite between these two.  The tools between them seems to be about at the same level of maturity (or immaturity).  Both allow you to use JUnit as a test runner.  Both allow Gherkin syntax.  I give an edge to Cucumber in that it needs a little less hand-holding configuration files to get things up and running.

Cucumber
Here is my favorite Cucumber setup which I'm using (as of Mar 2017):

If you just want to download cucumber jars the traditional way:

http://repo1.maven.org/maven2/info/cukes/cucumber-java/1.2.5/
http://repo1.maven.org/maven2/info/cukes/cucumber-junit/1.2.5/
http://repo1.maven.org/maven2/info/cukes/cucumber-core/1.2.5/
http://repo1.maven.org/maven2/info/cukes/gherkin/2.12.2/
http://repo1.maven.org/maven2/info/cukes/cucumber-jvm-deps/1.0.3/
http://repo1.maven.org/maven2/info/cukes/gherkin-jvm-deps/1.0.2/ http://repo1.maven.org/maven2/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar  http://repo1.maven.org/maven2/junit/junit/4.12/junit-4.12.jar

The above will allow you to develop and execute feature files and BDD automation.  Great for executing using maven and add these external jars to your build so you can build step definitions in Java.  But no sane person would do this without a nice syntax highlighting editor and "open on declaration" goodies plugged into Eclipse.

Natural

Natural is a feature file editor that uses code-assist to fill in Given, When, Then, and allows you to jump to the step definition.  Install Natural from the Eclipse Marketplace found in Eclipse at help->Eclipse Marketplace. Natural itself wasn't tagged in the depot correctly so you'll find it by searching for "cucumber," "jbehave," or several other BDD related tags.  Install it.

Test it out by creating a plain text file with a .feature extension.  I find when creating a new file with a ".feature" extension, I need to close the tab and re-open it so Eclipse can hand off editing the .feature file to Natural.

What Natural does for you when editing a feature file:

• content assistance in choosing Steps, 
• F3 (open declaration) which shows the Java definition for the step (assuming one exists), 
• Outline view, 
• syntax highlighting, and
• caution marks for Gherkin steps that don't have a Java definition.

Manual installation

Sometime back, when traveling Asia, I needed to do manual installs as the Eclipse Marketplace and Natural repository were timing out.  In those cases I did manual installs.

https://github.com/rlogiacco/Natural/wiki/Installation-Guide

1. Using Eclipse Install XText from this location: http://download.eclipse.org/modeling/tmf/xtext/updates/composite/releases/
2. Download an Eclipse archive (.zip) of Natural from this location: https://github.com/rlogiacco/Natural/releases
3. In the Eclipse->Install New Software, click "Add..." and this time select "Archive" and select the path to the zip file downloaded in the above step.

But Natural alone won't run your feature tests.  You need to install Cucumber-Eclipse, or as I prefer, run them via "cucumberized" JUnit test cases via @RunWith(Cucumber.class).

Cucumber-Eclipse
I don't recommend this tool as I've never gotten it to work for the last 12 month (2016/2017).  But If you want to give it a go....  As it's not in the Eclipse Marketplace for whatever reason, use help->install new software.  Click Add and add this url:
Select Cucumber-Eclipse and install.

What Cucumber-Ecilpse should do you for you
The context menu will have "Run feature tests" and "Debug feature tests" which should use the Cucumber CLI to execute the selected feature file, showing the test results in the Eclipse Console.  As of 2016/2017, I've not got this feature to work.  I think it can be made to work via fiddling with Runtime configurations.  I've given up on it and execute BDD tests via cucumberized JUnit test cases and JUnit suits.  (Naturally you can do the same with TestNG.) 

Good Organization and Configuration
Ask yourself how many features you'll build this year.  Then ask yourself what those categories look like.  Now go into eclipse and make a heirarchy, something like this: features->, features->.

So sprint by sprint, add your feature files into to that hierarchy and grow the hierarchy as needed.  I suggest representing the hierarchy as a package in the source code alongside the application it's testing.  If your Steps definitions have a strong relationship with their features, then put them alongside the feature files.  If there isn't such a relationship then don't do that.  (People have different feelings about 'global' namespaces of the BDD Steps.)

source/java/com/my/awesome/app
source/java/com/my/features

Most teams hook their JUnit runner to their BDD tests (using JUnit's @RunWith(...).  Put that test class in the features directory and make it responsible for running all your features.
source/java/com/my/features/RunFeatureTests.java

Using the above organization, follow the principle of "keep things that relate, together" and place the step definitions along side the feature files:
source/java/com/my/features/Foo/Foo.feature
source/java/com/my/features/Foo/FooSteps.java
source/java/com/my/features/ShoppingCart/Purchase.feature
source/java/com/my/features/ShoppingCart/PurchaseSteps.java
source/java/com/my/features/ShoppingCart/TakePayPal.feature
source/java/com/my/features/ShoppingCart/TakePayPalSteps.java
source/java/com/my/features/ShoppingCart/TakeVisa.feature
source/java/com/my/features/ShoppingCart/TakeVisaSteps.java

If you have global steps definitions (don't worry about this when starting out), then put that library in the features directory.  If you're driving a UI, you'll need a place to put your page objects too.  (Please avoid testing the UI unless you have to.  Also use the emergent design principle and develop your page objects as you need to.)
source/java/com/my/features/global_steps/*Steps.java
source/java/com/my/features/page_objects/Login.java
source/java/com/my/features/page_objects/ProductPage.java

SpecFlow
For .Net, all I ever seem to use is Spec Flow and it works good enough.  Here are some pretty good directions.  (FYI, don't believe him that this will work for Express versions of VS.Net as Microsoft turns off useful things like plug-in installation and debugger .)  The main thing is you need to install two things: SpecFlow libraries, and SpecFlow templates (for editing .feature files).

Don't forget to AGILE your Test Plan when transitioning from Waterfall to Scrum

Goal: Repeatable quality through automated tests!

But we need people to develop them. Traditional organizations call these people testers.


Tester-> Automated Tests!

In trad. organizations, testers rarely do this because of the traditions of Waterfall.

Tester-> Test Plan -> Automated Tests!

And this is where the trouble starts. Due to the divide-and-conquer and handoff approach of Waterfall, it made sense to split the role of software development into programmer and tester (I'll not debate the pros and cons of doing this, Agile comic SCRUM NOIR—A Silo to Hell! does a nice job.) but in the Agile context, these testers are challenged in integrating their work with a Scrum team: they can't write automated tests until late in the Sprint. This is a big problem when moving to a practice such as Acceptance Test Driven Development where automated test development starts on the first day of the Sprint.

Iterative Testing Problems:

1. Testers don't plan together with the developers during Sprint planning.
2. Testers never check-in an automated test on the first day of the Sprint.
3. Test Planning isn't done continuously (a little bit here, a little bit there, and then executed, then repeat) but instead is an upfront event that takes up 50% or more of the Sprint.
4. Testers complain they don't have enough information to get started.

The above happen in organizations in transition to Agile where people carry old habits and behaviors to their new roles, even if the behaviors aren't optimal. The Waterfall process was planning heavy and everyone (testers, developers, ...) were encouraged to spend a few months writing documents and reviewing them, creating a false sense that we "planned well." (The security was demonstratively false since the plans between waterfall phases always changed.) Since test and development are often in separate organizations, they coordinate around events set on calendars to deliver artifacts: Test Plans (among other things). The tester is responsible for the Test Plan and had dependencies on pretty near everything in order to produce it (development, architecture, business requirements, ...).

This habit is antithetical to the Scrum process since a Scrum team is doing incremental delivery of product and doesn't know until Sprint Planning what will be done during the next Sprint. This forces the Testers to produce Test Plans under immense stress if they preserve old habits.

This results in:

1. Testers hating Agile.
2. The Test part of the organization hates Agile and starts working against change initiatives.
3. Testers complain that they have to drive everything because the tester's need becomes the event rather than a scheduled milestone.
4. Many preconditions to creating a Test Plan: Testers demanding a lot of input documentation from other roles (design specs, arch specs, use cases, business case studies, hardware diagrams, call-flow, process flow,...) before they feel they can complete their Test Plan.
5. Equating Incremental testing with incomplete testing: Tester's consider their Test Plans incomplete because they are only supposed to think of testing the functionality they are delivering that Sprint rather than for the entire release.

Points 1 and 2 are effects which are mitigated by showing them how to be successful. Point 3 is discomfort about the culture change from Waterfall, where the process drives the events, into an Agile process where individuals drive the process. Testers and those who support Test Plan creation get used to it after a few Sprints. Point 4 is a "process over working software" habit from Waterfall where if testers (or their management) feels rushed, they can buy more time by demanding more process and documentation from others as dependencies. Point 5 requires another change in thinking which will happen after doing a few Sprints, witnessing that small high quality subfeatures will sum up to a large high quality feature.

If you change your process without changing how you do things then nothing's going to change (except for the labels). So points 3 and 5 are natural to "storming" (Tuckman model) and must be allowed to happen. Once we get to later steps, "norming" and "performing," and do it say in 3 Sprints, the dangers of the change rejection (points 1 and 2) will go away. Point 4 is a deep culture change which takes time until people to understand the Agile Manifesto Values and Principles, namely, Working Software is the primary measure of progress (automated tests are working software) rather than process checkpoints, milestones, and documentation completion.

No matter what, an organization in transition will have storming and that is healthy if conflicts are allowed to be exposed AND resolved (healthy stress/conflict drives change). To help "storm well," here are some activities.

How to Agile your Test Plan

The Scrum team containing programmers and testers must re-invent how they work together and be open to new ideas and ditch some bad ones (the heavy weight, comprehensive test plan). Each story needs it's own Test Plan. Look at solving this problem at two levels: Test Plan format and timing.

Making the format you already have, lightweight (format)

Rather than introduce any additional process/methodologies, make what you already do lightweight. If your usual Test Plans are 10-20 pages for features completed during a multi-month release, try and do the same plan but on one sticky for one User Story.

How can one stick get results as good as multipage test plans? We're leveraging the fact (and would be foolish not to) that within a few days we'll be building small features, so we don't need to document so heavily.

Agile Context (and why Agile development really is different than Waterfall):

• a lot of information is successfully retained in our tacit knowledge since we are working on a small team that interacts daily,
• we are dedicated to one sprint backlog and become experts in its execution,
• we'll start acting on our plan within days,
• we're working on sub features and only need to test the sub features we are doing this Sprint, and
• each User Story is independent of the others so each must be tested independently
• the majority (if not all) of the tests will be automated and checked in and will be our best documentation and reporting system

If you argue that your automated tests cannot act as your documentation, then you'd better work on your test design because you've got a problem that must be resolved.

Limber the mind

To change the results of your work you need to change yourself. A lot of what stops us are habits learned during the Waterfall context which need to be shed so we can develop sensible ones for the Agile context. This problem affects anyone when changing work contexts: I'm a science fiction author who spends hours getting my words right who sometimes spends hours getting an email right. This is complete waste for a one-off email. I had to learn that when I write technical planning documents or emails to reel myself back since prose isn't necessary. It took conscious effort to shift gears and doing pair work with another helped.

To do so I had to:

• decide it was important to change,
• be open to new ideas, in fact, be open to trying something so crazy it couldn't possibly work and then go for it.

James Whittaker of Google has a great facilitation style called The 10 Minute Test Plan which breaks down mental barriers that prevent writing a good and quick Test Plan.

Acceptance Criteria (format)

Acceptance Criteria are the most lightweight and commonly used of all Test Plan formats: a simple bullet list of what the application should do before the PO will accept the User Story.

Acceptance Criteria can also take the format of wire frames, call flows, non functional requirements, ....

Each User Story should have acceptance criteria before going into Sprint Planning. More can be added at any time, but it's important to have a rough list before Sprint Planning or the meeting will become overloaded and slow.

4D Analysis (format)

4D Analysis was introduced to one of my teams by friend and fellow coach XuYi. 4D adds three dimensions of additional analysis to simply using Acceptance Criteria. Adding more analysis isn't necessarily a good thing because that's what got us the 20 page Test Plan. However 4D is still one page and maybe your team'll find working on the first 3Ds helpful before getting to the Acceptance Criteria. The 4D analysis is attached behind each user story.

The Process dimension is further broken down into 3 types:

• User workflow (how it works from a User's perspective)
• Business process (how it works from a Business Analysis perspective)
• Technical flow (system engineer or architecture view)

The idea is that for an individual User Story you'll need one of the above process types and rarely, you'll need more than one.

Strangely enough, the feedback I've received from teams doing 4D Analysis is that despite the single sheet of paper, the amount of analysis is greater than they were used to from their traditional multipage test plan. Customize the 4D analysis to fit your team's needs.

BDD (process, format, and technology)

Get your PO adding Behavior Driven Development scenarios to your User Stories. Or at a minimum work with your PO and get them written down. Good BDD tests make wonderful test plans AND automated Tests AND traceability documents!

Given there exists bad entries in [playlist]
When trying to play this playlist
Then remove invalid playlist entries
examples:
|playlist|
|Never Played|
|My Top Rated|
|Whole Library|

More BDD articles:Well written BEHAVIOR Driven Development Scenarios, BDD Practices that Maximize Team Collaboration and Reduce Risk.

Timing

When should Test Plans be created? The two strategies are:

• Create Test plans during the Sprint
• Create Test plans before Sprint Planning.

The testers and developers include in their estimates made during Sprint planning the effort to create Test Plans and implement tests. But they need to learn to move fast and test like a jazz band rather than a symphony which has a conductor that has organized a lot of planning and process.

Lightweight test plans can be created before the Sprint by leveraging Backlog Grooming. Here's an agenda that broke backlog grooming up into the following: 30 minute kickoff, offline (outside of meeting) collaborative work, 1 hour grooming meeting.

Here's an example calendar for a three week Sprint:

MTWTF MTWTF MTWTF
         ^- Kickoff Preperation for Grooming
            ^- Groom results for 1 hour

Constraints:

• a team spends no more than 10% of a Sprint preparing for the next Sprint
• other than Sprint Demo and Retro, avoid meetings on last 2 days during Sprint crunch time
• have Grooming far enough in advance of Sprint Planning to fix problems uncovered in Grooming

Kickoff Agenda (team, PO, SMEs in attendance) < 30 minutes

• During meeting
• PO brings proposed Sprint Backlog
• Team members select stories to prep for grooming and are encouraged to collaborate with other team members
• Decide what stories need SME deliverables/support and make that status visible
• Offline, to be finished by Grooming Meeting start (Monday 2PM)
• Fill out prep document
• User Scenarios {Functional flow, User Experience behavior} (done by: programmer or tester)
• Y/N needs SME deliverables (architecture view, etc.)  (roles: tester, SME)
• Refine User Story Acceptance Criteria (roles: everyone)
• Update Assumptions (roles: everyone)
• PO and SME will visit each team member (individuals and interactions per the Agile Manifesto) before Grooming Meeting and see how they can help.
• ScrumMaster will make sure the process is successful and that everyone comes to the Backlog Grooming meeting prepared.

Grooming Meeting 1hour, review the results of the prep work as a group.

A team reviewing User Stories and grooming prep doc. They are standing up during the action moments for maximum collaboration. Stand up meetings are 30% faster than their "sit down" counterparts.

During Sprint Planning, the team will adjust further and are expected to refine further even during the Sprint.

Summary

Goal: Automated Tests!

But we need people to develop them.

Ideal:
Tester->Automated Tests!

Usually, teams do at least a lightweight Test Plan: 
Tester-> Acceptance Criteria-> Automated Tests
Tester-> BDD-> Automated Tests
Tester-> 4D Analysis -> Acceptance Criteria -> Automated Tests 

Some teams need more analysis. Find a way to do it using only 10% of your current Sprint to plan for you next Sprint:
Tester->Test Plan->{Acceptance Criteria, BDD, 4D Analysis}-> Automated Tests

Remember you're doing Test Plans to have automated test cases to defend your product from regression. Your Test Plans should be designed to serve this purpose. Having large Test Plan documents was never the goal. If you can produce 2-6 automated test cases from a one page Test Plan, and spend no more than 10% of your Sprint doing Test Plans, then you're on the right track.

Why STATIC is the keyword your mama warned you about

In the beginning, there was machine code and all was good. Then there was assembly and after that, a progeny of later cousins such as FORTRAN, COBAL, C, and BASIC, and these cousins had a GOTO command. They were procedural languages and GOTO allowed the programmer to command the procedure to immediately jump to a different location.

Later, as more software was developed, programmers noticed that *understanding* what was happening during a GOTO/GOSUB was easily forgotten. Later, GOTO was recognized as a face of evil so better to use procedures which have a name, and this name provided meaning.  Adding to the bargain, procedures defined variable scope so we didn't have to worry about creating unique variable names across the whole of the application. Everything was packed in a package of meaning, parameters, procedure, and scope.

Although programmers rejoiced, because the habits of those who had been using assembly had trouble adapting to the new paradigm, the new languages left a crunch behind in the form of GOTO. So many still built code with GOTO and big comment blocks to explain the intent of their GOTO blocks. As time passed habits evolved and people made their GOTOs to GTFO.

OOPS!

 Later, actually much later in computer years, some smart people wanted more sophisticated control of variable scope and designed object oriented languages. These languages allowed grouping together of procedures with scope and classifying the groups with a meaningful names.  These languages dropped GOTO and cousin GOSUB because by this time, it was accepted that they did more damage than good

But friends, nothing shakes confidence in a design more than seeing static. Unlike previous keywords such as GOTO and GOSUB, static begets more static design. Like a cult of evangelicals, static methods insist on converting everything they touch into static. One static method begets another, and that one begets another until as far as the eye can see are static fields and static methods in a hulking procedural design that shuttles behavioraless data structures through static methods with long parameter lists. Each static method "switch and casing" to discover state so a valid response can be made.  Where OO design encourages co-locating state with related procedures in a way that minimizes scope, static data structures global to the process are bandied about, convenient for communicating state for a few static methods but unfortunately turns into a "what's good for the goose is good for the gander" mess that is shared across all kinds of activities, increasing coupling, making the application a devil to add features to or unit test.

When can using STATIC be righteousness?

I most often use "final static" to create constants. Outside of that, I think really hard before I type the words STATIC.  How do you know you're in trouble?

• If at the top of many of your class declarations you have STATIC initialization blocks.
• If at any moment in your code you could have a reference to a STATIC class that couples you to some heavy I/O services (DBs, network, filesystem) that'll screw you when you want to write some unit tests.

Static methods force a procedural design--a style that is as 1970s as brown wall board, orange shag, and bitchen fros. And some dirty hippies always hitch a ride: static variables (as opposed to constants) and big global data structures. Although it's cool to throw on bell bottoms with an Atari T, and tuck a comb in the fro, fatty procedures with comment blocks every 100 lines is about as cool as banging your junk against the knob ridden dash of a classic Chevy Impala. It's only funny when it happens to somebody else. So unless all other avenues are exhausted, don't use static. And even then, minimize it's linkage by using the Singleton pattern which is unit test friendly.

And remember to tell Mom, "Thanks." She'll love you even if you use static, but your co-workers and sustaining team will love you a LOT less.

Other Sources

Technically correct article:
http://stackoverflow.com/questions/7026507/why-are-static-variables-considered-evil

Technically correct AND entertaining:
http://blog.goyello.com/2009/12/17/why-static-classes-are-evil/

Well written BEHAVIOR Driven Development Scenarios

In my work with coaching teams in doing Behavior Driven Development (BDD), it's been non-trivial (doable, but takes time) to teach what is a GOOD BDD example.  I'm finding that teams that "self start" in BDD are creating UI driven scenarios rather than behavior.  They end up creating scenarios such as the following.

(for an iTunes plugin that culls out invalid song entries in a playlist)
(THESE ARE EXAMPLES OF BAD BDD SO DON'T DO THIS!)
Given iTunes is launched and there exists bad entries in playlist Never Played
When user selects Music and playlist Never Played
Then show dialog listing invalid songs

Given showing user listing invalid songs
When user clicks OK
Then remove invalid playlist entries


There are bad smells in these scenarios:

1. the end user is mentioned
2. contains User Interface (UI) language (dialogs, selecting, clicking)
3. the scenario language contains details that only an experienced user of iTunes can follow it

While the nature of smells isn't that these things are forbidden, but that it's best to marshal your energy in expunging them as much as possible, especially if you're new to BDD.

Although UI scenarios will make most engineers happy because it's clear what the UI is supposed to do and will provide usable system tests, you will create the following problems:

1. High maintenance--Too much presentation language means the tests need to be updated when the presentation changes even though the behavior hasn't changed. 
2. Implies the UI design is finished--By virtue of writing the scenario in presentation language (rather than behavior), the presentation design is now fixed in the reader's mind.  The engineer may not wish to challenge this even if they see a better way.
3. The BDD test will be a UI test--If all the BDD scenarios are written in UI language, then all the testing will be through the UI.  Remember the Test pyramid!  You want as few UI tests as possible because UI tests are inflexible, slow, the most expensive to maintain, and prone to false positives.  In this case, the team is building the plugin, not iTunes so why should they have a Given/When/Then that also describes code delivered by the iTunes team?
4. The BDD tests will always be slow and slow is less valuable--If all the tests are UI tests, they will run slower than those designed to only test behavior.  A test failure that tells the team that something happened in the last fifteen minutes that caused a regression will be trivial to correct (say less than 30 minutes).  A test falure that tells the team that something happened in the last 24 hours is going to require using a debugger to figure out where the bug is and then figure out which change caused the bug (likely will be an hour or several hours).
5. Discussions are anchored to UI--Teams that can understand their work at the behavior level will be able to discover problems easier without having their thinking distracted by what's happening in the UI domain and clutter communication with others.

The above scenarios should be rewritten by focusing on the behavior:
(for an iTunes plugin that culls out invalid song entries in a playlist)

Given there exists bad entries in playlist Never Played
When trying to play this playlist
Then remove invalid playlist entries

This is something the user will pay for!  They don't care about the clicks or if the pointer is used at all!  They want the bad entries in their playlist removed!  If you have POs create such scenarios, they can easily communicate and debate them to product marketing about what behaviors the user wants without needing someone to translate what is the result of UI manipulations.  Give a Scrum team this scenario during sprint planning, they're going to need to discusses possible UIs to allow this behavior to happen.  But they don't need to choose right now anymore than they need to decide how many Classes they need to design and what are the public methods/attributes.  They just need to know if they can do any of the options during the Sprint.  When sprinting, they *will* need to decide on a UI implementation and they will need to implement the test driver which *may* need to drive the UI, or even better, trust that iTunes does it's job to hand off to their plugin and the team only tests that their plugin causes the behavior to happen.  (They likely will need at least *one* end-to-end UI test that ensures iTunes does hand-off to their plugin.)

By the way, for the BDD experienced in the crowd, it would be a great idea to parametrize the scenario to handle many/all playlists when the team is needing to add features for many play lists:

Given there exists bad entries in [playlist]
When trying to play this playlist
Then remove invalid playlist entries
examples:
|playlist|
|Never Played|
|My Top Rated|
|Whole Library|
...

Check out the other BDD resources at this blog such as:

• the section on GWT smells (GWT = Given When Then) in: BDD Practices that Maximize Team Collaboration and Reduce Risk

Other Resources:

I highly recommend reading the following which will NOT teach you how to write the code but teaches you how to think behavior and why it's important to do so (available in paper and Kindle):



Here is a great BDD tutorial if you are into music (guitar in particular) http://www.ryangreenhall.com/articles/bdd-by-example.html

How NOT to Motivate your Organization to Perform

The TED Talk video by Dan Pink (at the bottom of this post) reinforces a few things that I've witnessed in doing Agile consulting and it validates many of the Agile values around roles and responsibilities.

I did a technical training in Test Driven Development (TDD) and Refactoring to a group of highly skilled software engineers at a client in Shanghai. The engineers were grouped into pairs and were each implementing a set of features using TDD. Things were moving at a steady pace. With 2 hours left in the day, the attendee's energy was dropping so I offered a 200RMB (abt. $30 but really $200 in spending power) reward to the pair that could get a the functionality finished AND could pass my 3 bug injection tests (to see if they had good unit test coverage--did good TDD). The room was electrified at the announcement and the teams of two become very focused. At the end, I didn't have to pay out although one pair was pretty damn close.

What was reported at the end as interesting: the carefully refactored and well designed code became hacked and sloppy in effort to quickly finish the job. After watching the Dan Pink talk about the Candle Game, I wonder if some would have finished if I hadn't offer the incentive.

Dan's talk hits themes of how traditional management of carefully tracking workers and stepping in isn't as productive as slackening the reins and creating an environment of self direction. After all, software engineers are highly educated people. It's sad to see how management sometimes steps in their way and makes technical decisions for them.

A colleague and I argue about whether Scrum's Chicken and Pigs roles (http://www.benhallbenhall.com/2013/03/scrum-meeting-pigs-chickens/) do more damage than help. I say it depends. If management has a habit of doing micro management about how code is developed, then there will be waste made in bad decisions or time lost in Scrum meetings being taken over by well meaning but over-reaching management. It's like the Feds overstepping their boundaries into State and county areas. The Chicken and Pigs model stops this, sometimes creating a different kind of damage but I feel it's necessary to swing the pendulum the other way for a year or so until management and development (in the Agile world, this includes test) better understand the boundaries of what decisions are technical and what decisions are business. Mixing these together creates as much waste as a baseball game where the pitcher is also playing third base and giving orders (as opposed to advice) to the outfield.

BDD Practices that Maximize Team Collaboration and Reduce Risk

Maximizing team collaboration and reducing risk

Guiding Principles: KISS – Keep it simple silly & Incremental delivery

I’ll think about the future but only implement what I need this sprint, this day, this hour. I do that so I can start my implementation at the simplest level (but not too simple) and grow the feature minute by minute, hour by hour.

Because I work this way, I can check-in often. This enables other team members to have visibility on what I’m doing. This allows for cheaper integration (merges) for the team and myself. It allows me to get the latest changes from the source tree so I can receive new work from the team (and get visibility into what they’re doing). Checking in often reduces risk and increases code reuse. I can’t reuse my team member’s latest new shared libraries if it isn’t checked in. If I get pulled away from the work (personal or fire drill), another team member (who has some insight into my work because they’ve been seeing my many changes coming into their development environment, who has heard about what I’m doing at a high level during standup, who is collocated and has heard me working with my pair, who maybe has paired with me) can finish the task or story so the team is successful.

Because the team is checking in often, we are releasing our best design efforts and reducing latency for others to review and use our work.

Test Strategies

Guiding Principles: Tests run independent of each other, give timely feedback, are maintainable, deterministic, and test a feature once.

It’s best that EACH test run in a clean environment: clean data, clean server state, clean client state. This may not be practical if the tests take too long to give timely feedback.

Common strategies are any combination of:

• Test login only once, the other tests skip login: share selenium driver across tests, or have a way to generate a session token so your tests don’t need to login.
• Don’t commit data destructive tests to the DB or find a way to get the data "reset."
• If you don’t mind your tests running slow in different contexts, then make the above "workarounds" configurable so you get fast feedback in dev branch, but the "slow tests" that login and clean all the state/data in int branch.

Managing Data:

• Maintain a data dictionary—This is a single point where we declare what data in the DB we are dependent on. Three columns: short hand description that’s used in the code, database id, notes. The data dictionary reserves data and is a signal to the viewer that they shouldn’t write to the data or change it as there are tests relying in the data.

BDD in your Sprint

Behavior Driven Development implies that something is driving development. This something is failing or pending BDD tests. Immediately after Sprint planning, get those failing tests checked in and executing so that failures/pending are visible on your continuous build monitor. Why? So everyone can casually see the status of your project, just as you do when your commute takes you by a construction project.

Three levels of being driven:

1. Immediately get the entire sprint backlog features checked in as pending tests. Then before each feature is finished, implement the Steps to prove the feature works.

Risk: Too much automation work is left for the end of the sprint and we deliver features without automation. We don't discover requirements gaps early in the sprint because automation is postponed.

Value: Make visible what work is accomplished versus pending. Doing automation before implementation tests your understanding before you do it all wrong. :-)

2. Immediately get the entire sprint backlog features checked in as pending tests. Next, implement the Steps to prove each isn’t working (imagining the UI elements are built, coding to IDs or button names that you make up in your head). As each feature is implemented, tests should pass.

Value: if you can automate the tests, you've proven a deep understanding of the requirements or have revealed problems early in the sprint when there is time to fix that problem through collaboration.

Also, you've made visible on your build monitor a lot of status: Pending automation versus automation completed versus features completed.

3. At project start, enter all the pending features in the product backlog, and then sprint by sprint stakeholders see progress toward features as the teams use the above level 2, and as time passes, those features are split or remove or new ones added.
Value: Makes visible how the project is doing at the release level.

What happens if I get stuck automating?

1. Get help from a team member because pair working results in 80% effort toward correctness and we come up with more ideas to try and solve the problem.
2. Talk to the whole team about the issue as no single individual owns quality.
3. Escalate issue to organization
4. By Sprint end, maybe the team decides it’s not automatable and that they’ll test it manually.



Given When Then Design

These are BAD smells:
• Too many GIVEN, GIVEN, AND, AND,WHEN, AND, AND, THEN.
• Need an engineer or power user to understand the GWT. (An end user or end user’s boss whose never used the product can understand the GWT.)
• UI language mentioned in the GWT
• feature file is many pages long
• GWT language is not reusable. Lack of consistency of language.
What if my "feature" has no behavior:
• Go find it!!!
  • Talk to your PO and find out "why" the customer wants this.
• If you’re delivering a component that is *part* of a feature, then
  • Go look at the feature’s GWT and understand how your component supports that feature.
  • Implement Steps for that feature to test that your component delivers that part of the feature.

Test Steps

These are NICE smells:
• Step methods are about three lines long.
• No @Alias(es).
• Steps call into a shared library (rather than call interfaces on other Step classes).

Use Continuous Integration

It's hard to get started using CI if you aren't already. Here's how to get started.


Guiding Principles: you can’t automated what you can’t do manually and you'll always find a better way to do it next week (but don't wait).

If you tackle all your technical difficulties from head on, it's hard to make progress. So deliver your way to a CI by using evolving coding and learning, and incremental delivery. Because each CI server out there can work with any possible script, script it using methodologies that get the job done (for you). Start getting the value from your integration efforts as soon as possible.

1. Manual integration: integrate on a separate machine (using scripts that later will be executed continuously.)Scripts should be able to find new tests at run time rather than rely on static lists. A script should be able to build, install/deploy. If you can't yet create scripts, do it manually until you learn how.
2. Continuous Integration: Select CI software, install it on a server, and then automate the execution of scripts ina CI.

If you have feedback, agree, disagree, feel free to comment as you'll be adding value!

Posted with Blogsy