NO info DUMPING

No Dumping

In the pursuit of good Karma, riches and fame, employees want to provide value to their employers and colleagues. This leads to a compulsion towards DIY (do it yourself), which results in project teams not working in pairs. Working in pairs allows for superior information sharing but due to socialization of DIY, those who avoid pairing are not even conscious of the inefficient mechanisms that they have created to allow them to work alone. One of these mechanisms is the info dump.

What is an info dump?

The other day I was working at home and needed a stapler. I knew it should be in my wife's office so I went there and looked in the drawer where it was supposed to be. It wasn't there so I guessed that my wife, being a busy woman, possibly left it somewhere else. I looked around on the desk and then on some shelves. After five minutes had passed I realized I was going to waste a lot of time unless I got more information. "Honey! . . ."

She was in the living room busy doing work on her laptop. I interrupted her and she knew right where the stapler should be. She said she had left it in a pile of magazines shoved into a different drawer. It took a while to describe the details of the information, which drawer, which desk, what corner to look in, but she eventually dumped the information on me and I went back into her office. I tried to reproduce the steps but after some time I gave up and went back to the living room. "No, not that drawer!" she said and she gave me further instructions. I tried again until I was frustrated and asked her to come take a look.

After a moment she said, "I remember using it when I was cutting out photos." She opened another cabinet and looked. It wasn't in there, but then I remember a pile of photo paper near the printer. I pointed at the purple scissor handles sticking out from beneath the printer stand. It only took a minute to find them!

Why did this work so well?

As soon as she entered the room, the context of her surroundings jogged her memory of where she had last put it. We were working with up to date information about the room. She provided additional information where I could help because I was in the moment. It took very little time when we worked together. While trying to perform DIY, I used up over twenty minutes, randomized my wife's work, and I still didn't get the job done.

Engineering

Coworkers, bosses, and employees asking a small favor, they all want to get things done with minimal disruption to others. People working on the same project often divide the work and then DIY until they are really stuck like I was. This leads to waste and each individual becomes a silo of information of how the work was performed. This also leads to conflict when the team members integrate their work and their designs don't fit together. Instead, if they immediately work together, their collaboration will result in a usable design. When they each DIY, they heavily invest resources in each of their own designs so when they integrate their code, they have to decide who wins. They both lose in the end since the project will finish only as fast as the slowest item of work.

Conclusions

Now that you know what in info dump is, watch for it in your office and you'll see it happen over and over. Two people will get together to discuss adding functionality and invest time in one of the below Info Dump Anti-Patterns. Each requires a significant amount of time and then also requires future iterations of doing it over again once new information is found. From my experience, it always takes more time to use info dumps and try to work in parallel, than to serialize the work and have two parties work side by side at one workstation. At that workstation, they execute a "doing" strategy where, in the moment of "doing", the pair can immediately discuss, make decisions, and execute; discuss, make decisions and execute; . . . Below are some anti-patterns I've observed in the wild.

Info Dump Anti-Patterns

Each pattern has two roles: the "implementer" (the person doing the work), and the expert (the person who has a little more history and more knowledge about how the work could be done).

Mental Gymnastics - This occurs when two (or more) people discuss a coding related problem for 20 minutes or more. It's not possible know the code base well enough to be this detailed. The discussion will contain inaccuracies but the participants continue like it's some kind of limbering of the mind, or mental gymnastics. Even more waste happens when these two gymnasts argue over their recollections on how the code operates. When the gymnastics is over, the implementer goes to her workstation and within minutes, discovers that the code works differently than either of them thought. At that point, she either redesigns a solution by herself (DIY) or goes back to the expert and updates him, and then they have another session of mental gymnastics. I've seen this process repeated up to four times in one day.

Tome of Knowledge - The expert spends half the day (or more) writing a design document. At best, it is written immediately after reviewing the code so the information is accurate on that day. The tome is given to the implementer and hopefully the implementer does the work before the information is out of date. Even with an up to date tome, the implementer's knowledge is incomplete and has to visit the expert for a session of mental gymnastics.

No Do, just Code Review - The expert and the implementer sit together at a workstation to learn how the code works. The benefit is that the information is completely up to date and the session is interactive between the implementer and expert. But since they don't actual "do" any coding, they perform mental gymnastics to predict design changes and guess at how the new functionality will hook in. Ideally, the implementer immediately develops the code after the code review. But before the hour is over, the implementer starts to find challenges in getting the code to work as expected. So the implementer performs DIY or often needs to visit the expert for more mental gymnastics, tome updates, or code reviews.

Agile's PR problem--people who don't do it right give Agile a bad name

Jim Shore posted an interesting blog, The Decline and Fall of Agile.

It's well thought out and genuine and I agree with him except saying it's the "decline and fall of agile" felt like a fear tactic to get notice. (terror!)

I'll go this far: Agile is going to have a PR problem.

Scrum allows one to teach agile planning without agile engineering.
Sometimes you can't fix everything at once. When you go into an organization that is doing up-front (but not really since the requirements always change) and teach them to do agile, if you can get them to at least execute Scrum and have them doing it all rather than Scrum (but not really) you have improved the situation.

Are they going to struggle if they aren't doing XP practices? Hell yes. Some of them evolve a little to automate their tests and many evolve to Water Scrum (a waterfall life-cycle during the sprint so two weeks development, two weeks of "stabilization for testing", terrible stuff really, but they do a lot of planning just like Jim said. And that's an improvement because they have closer to truth about the state of things).

Also, I have been involved in agile roll outs that teach both Scrum and XP. It's really expensive to the organization to do booth, both in time and money. I'm with Jim in saying that it's going to be cheaper in the long haul, but it does create a high barrier to winning business.

The existence of consulting gigs to make Agile teams better is a better situation for the client because their mistakes are costing them less.

Just another Westside Story (the Wikis versus Sharepoint)

There has been a battle at my company over how to store and disseminate knowledge. We (speaking as a member of development) have used the Confluence wiki for many years and were happy with it. But less technical staff such as managers, marketing, and sales really hated it. So there was a shift at the IT management level to move everyone to Sharepoint. Their main argument was that Sharepoint was free due to our MS licensing agreements.

It turns out that it wasn't really free because it required many resources from IT to actually get the system up and running in an organized and useful fashion, and even then, development hated it because managing content using a word processor was heavy weight compared to editing a wiki page. (Using a wiki page is nearly as simple as writing things down into a text file.) The non-development types will disagree with this saying that Sharepoint is easy because it is like a word processor, a tool they are familiar with. But the type of work they do is more akin to publishing results where development uses a wiki for knowledge dissemination and creation, more akin to jotting down notes. If it isn't quick and easy to do, developers will just live without doing it at all and this will create some efficiency losses. For development tools, throughput is king, not tools for making things look pretty.

Sharepoint supporters will also say that there is a wiki in Sharepoint. But its really an abuse of the term. It comes with a rich-text editor which gets in the way of everything that makes using a wiki nearly as easy as creating plain text files. The Sharepoint wiki is a great way to persuade everyone to retreat back to the word-file/upload/download paradigm that is native to Sharepoint.
This creates problem's in development. The best team collaboration pattern is a community of cross-functional team members that change/maintain the team's knowledge sharing system with little effort. (It's like Web2.0 content communities versus Web1.0 web pages.)
IT's push to sharepoint caused some splinter groups in development: one faction moved to Sharepoint, a faction of development stayed with Confluence (though under threat from IT that they would pull the plug), and a new faction organized around media-wiki.
The later two groups formed out of frustration of using Sharepoint.

Recently, I was coaching one of the team's that kept their documentation in Sharepoint. Only one or two people bothered to put add/maintain information in Sharepoint. Many did use the site as readers. This created a culture of very little documentation which was out of date. When I moved some of their documentation to mediawiki, suddenly more people started updating any problems they saw and added more documentation. It was amazing to see how using the right tool enabled this to happen.

I've been working with wikis since 2001 and one thing that holds true is that there is a small learning curve that some members of the team may not cross, unless they are pairing with someone to get them going. I've found one simple exercise that quickly gets people using wikis is to ask them to add their contact information to the team's wiki. (Put it on the backlog if they aren't taking it seriously.) To perform this milk run, they are exposed to a few tools which gets them immediately productive, and with that critical mass knowledge, they know how to find more advanced features or how to use existing pages as pedagogical devices. Learning from existing pages is a property that Sharepoint lacks. Sharepoint relies on the usual hunting through menus and buttons because there isn't any "markup source" that can be used to learn from other people's postings.

Remember that wiki means quick. If a wiki doesn't fit this term, then find one that does. Otherwise, no one will use it and that free Sharepoint tool is going to cost you.

VBA script for cleaning .XLS when Gettings Stories out of Rally

Remember in a previous posting about The Mail Merge method of exporting your stories from Rally where I complained about the work necessary to remove all the markup from the spreadsheet? A listener to this agile confessional took it upon himself to create a VBA script to automate this step. Let's give Pete Turner an Internet thanks for sharing! :-)

Paste the VBA text file into your into your “personal macro workbook." Yyou should be able to open with ALT+F11. If that doesn’t work, try unhiding it using View > Window > Unhide.
You can run the macro with View > Macros

Have fun!

Maven: The coolest build tool ever!

I've been through the grinder on build tools: typing in VIC20 BASIC, building from the command line, building from .sh scripts, building from Make, building from Xmake, building from IDEs, building from nant.

It's been a glorious twenty-eight years of software development (including grade school years of software development). Now I'm hearing good things about maven. A few people love it; many don't like it. I've stayed away from the arguments and kept clinging to my Nant and Eclipse build tools, but now I've landed on a project that's using maven 2.0.9. I have an excuse to dive in!
And I really love it!

It was tough at first, but when I look at the time I dumped into learning Make and its implied rules, it really wasn't so bad. The documentation on "the internets" (thank you Mr. Bush) is pretty minimal, maybe enough if you already understand the big concepts. There is a new book out this month that looks promising: Maven: The Definitive Guide With only the Internet, I probably would have stayed a frustrated Maven user except a smart cookie named Jeff Ramsdale gave a colleague and I a thirty minute intro. To give back to the internets and to put my notes on maven somewhere, here is my high level understanding of organizing a maven project. I'm sure you'll correct me by commenting if I'm out in the weeds.

Lifecylce

Maven drives everything via a lifecycle. This lifecycle is built into maven:

1. validate - validate the project is "correct" and all necessary information is available.  Said another way, "go get my dependencies."
2. compile - compile the source code of the project
3. test - test the compiled source code using a suitable unit testing framework. These tests should not require the code be packaged or deployed. Don't mix your system tests here.
4. package - take the compiled code and package it in its distributable format, such as a JAR.
5. integration-test - process and deploy the package if necessary into an environment where integration tests can be run. Don't put unit tests here as unit tests != integration tests. These tests should be testing that the system or subsystem works.
6. verify - run any checks to verify the package is valid and meets quality criteria
7. install - install the package into the local repository, for use as a dependency in other projects locally
8. deploy - done in an integration or release environment, copies the final package to the remote repository for sharing with other developers and projects.

A portion of this lifecycle is called a phase, so the above is a list of phases. Maven has a configuration file called a pom.xml. When you invoke maven: $mvn, it looks at the nearest pom.xml and by default, it uses the "install" lifecycle. Since install is number 7 in the above list, maven goes through phases 1, 2, 3,...,7 and may perform operations based on what is in the pom.xml. You can also do a $mvn deploy and mention the phase right in the command line.

Profiles

Profiles are an abstraction layer on top of the lifecyle. Creating a profile allows you to create configurations which affect how things are done. For example, on our project, we have some integration tests which take two hours to run and we have two integration tests that take ten seconds to run. Before we check in, we want to run those quick integration tests but not the long ones. So we explicitly named the two integration tests in the configuration for the surefire plugin (a junit test runner in maven) which runs the two quick integration tests in the integration_test phase when $mvn is executed.

For the slow integration tests, we created a profile "all_integration_tests" which re-configures the surefire plugin to include running the slower integration tests too (using the naming convention of *IntegrationTest.java) when $mvn -pall_integration_tests is executed.

Goals

Goals are like arguments you use to tell a plug-in what to do. By the way, everything in maven is a plug-in so plug-in's are first class citizens. A goal is an argument that you pass to the plug-in. I don't have more to say about goals because I haven't had to use them much. :-)

After all, gotta save something for later.

The Mail Merge method of exporting your stories from Rally

Chris Sterling, a great agile coach who I get to work with, showed me how to export stories from Rally in a more sophisticated fashion than the PowerPoint method. These steps build upon what he had shown me. Again, this is all about getting the story information out of Rally and into a physical medium which can be easily interacted with by a large group of people, thus removing impediments to collaboration when we are in the same physical space.

This process will produce stories on "cards", two "cards" per sheet of paper. Before the planning event, I use a cutting board to split the sheets into two. I'd give myself an hour before the planning event to get through this process until you are experienced.

Getting the stories out of Rally and into Excel

Use the same steps mentioned in the PowerPoint method and stop after doing the Processing the CSV with Excel step. If you want a stack of cards at the end of this process that are sorted by rank, then make sure your spreadsheet is sorted by rank (or that Rally is sorted by rank) so you won't have to do this by hand.

Setup the Mail Merge

The following directions describe how to do this in Word 2007. Download the mail merge template and open it into Word. The first time you open the file it will give you an error message saying: "Opening this document will run the following SQL command:" because it is trying to load up the .xls file which I used for mail merge. You should tell it "No," though no harm is done if you say yes other than getting a cranky error message.

In the toolbar, click on "Mailing".
The buttons in the "Mailing" toolbar are arranged in a type of life cycle arrangement. Since I'm providing the template, you won't necessarily use all the buttons to get to the last one, "Finish & Merge." Notice the merge fields, which are the groupings surrounded by "<<" and ">>". These are the same as the column names in the .xls file. The data that is inserted into the merg fields will match the font characteristics of the merge fields.

Loading the Excel file & Preview

Because we are using a mail merge utility, it's wording things in the way of "mailings." For us, our set of stories are mail recipients, so the language of the buttons is going to be weird. Click "Select Recipients"->"Use Existing List" and select the .xls which was produced in the earlier step.
Now you can click on "Preview Results" and see how things look.
Now you can exit the preview and adjust the styles as you want. I made the first line largest and easiest to read and loaded it with what I've found to be the most important info we needed for planning. I also crammed as much info as I could onto the sheet of paper. You may have different needs so you may want to exit preview mode and edit the template. The "«Next Record»" merge field means to populate the 2nd "card" on the sheet of paper with the next record. Without this, you'll end up with two copies of the same "card" on each sheet of paper. The darkened background uses up more printer toner, though the team I was working with was taping the "cards" to a white board so they wanted them to stand out.

Finish & Merge

MSWord's mail merge preview has bugs so don't be surprised to see duplicate story cards show up. A phenomena I've seen is where the last story on the virtual sheet, shows up as the first story on the next sheet. This actually isn't the case. Click "Finish & Merge"->"Edit individual documents" to prove this to yourself before selecting "Finish & Merge"->"Print documents..." and sending the job to the printer.

Split the sheets in half with a cutting board, and find a roll of tape, and now you are ready for you planning event!

Further Improvements

An impediment list can be made stronger by exposing the cost to our processes. In the below list, the bigger the number the bigger the cost. I'm using a geometric series.

16 I'd like to avoid the "cleaning" of the markup from the spreadsheet. Anyone know how to turn those Excel cells into something that accepts the markup? Or perhaps there is a Rally configuration that will scrub that markup out of my .csv?

2 I did this process a few times with Avery card stock but this seemed wasteful since there is a lot of paper boarders that aren't used and the cards, albeit nice and thick, aren't necessary. It would be great to buy reams of paper which are perforated to tear in half.

Update: Thanks to Pete Turner, I've added a later posting about a VBA script which cleans up the .xsl file: VBA script for cleaning .XLS when Gettings Stories out of Rally.

Exporting From Rally to task cards--the Power Point method

Exporting from Rally is possible though not pleasant. I spent at least 2 hours trying to work it out the first time with someone's help. Even after I got the the process down, it still takes about one hour. In theory, I think I could do it in fifteen or thirty minutes, but every time I try, something comes up that I need to react to. (Like printing off the wrong backlog.) So I'd practice this once and then give your self at least an hour before the planning event to get ready.

Here are the tools you need:

• Rally -- This contains the backlog. You'll need to create a custom view to get the "Description" on the card. We'll use the CSV export feature.
• Excel (or OpenOffice) -- Use this to clean up any formatting HTML that rally has exported, read in the CSV and then copy past the rows into Word.
  
• Word (or OpenOffice) -- Use Word to create a .doc file.
  
• Power Point (haven't tried this with OpenOffice's Present) -- Read in the word doc which will covert the content into slides, and then print the slides.

The steps to get a Rally backlog into cards/paper:

Export to CSV

You need to show your backlog on a Rally view. My favorite is at "Backlogs & Schedules"->"User Stories" because it seems to be the most flexible with filtering. If that isn't good enough, you can create your own custom view.

When you select Action->export to CSV, Rally will export whatever you are showing, in the order you are showing. So if you are missing a column of information, like the story's description, you'll need to create a new view (see the icon with the plus above the Release column in Rally, circa 8/22/2008, use this to add what columns you need). You'll probably want to sort by the *Rank* column.
Columns I like to have in my custom "Export to CSV" view are : Rank, Formmatted ID, Name, Schedule State, Blocked, Description, Release, Plan Estimate, Notes, Iteration, Owner, Last Update Date.

Processing the CSV with Excel

In Excel, open the CSV that you've exported.



Cleaning the RTF-markup from the Story Description
If you've exported the story description, there is always some work in cleaning the data of rich-text markup. Take a look at the description column and you'll see what I mean.
If you've never used any of the rich-text features (bullets, underlining, copy and pasting from a rich-text document such as MSWord) then cleaning up any remaining formatting is tractable.
Using Find-Replace-All, you can clean up the markup. Replace the following with nothing:

Replace the following with a space: "&NBSP;"
Look over the Description column and remove more choice bits of markup cluttering your data:

Other things to do:
If you exported the Blocked field from Rally, you'll want to select that excel column and do a find-replace-all on "True" to "blocked", and "False" to nothing.

Make a Word Document

Swap the excel columns so they show up in an order that makes sense, and delete the columns that you don't want. Highlight the rows you want to print out, copy/paste them into a word processor, and then save the file to a MSWord .doc.

Adjust the layout in Power Point

In Power Point (I'm using power point 2007), the name of the game is adjusting the format of the font so that it fits on the slides without pushing into the next slide. When you get it looking pretty, print it. What I do to get things looking good is edit the master slide to be one big text box and change the font size to about 20. Then I leave the master slide view, go to the left hand slide selection area, do a ctrl-A, click on Layout->select my theme. And then all the slides are using my theme.

Power point is great if you are OK applying one formatting style to the whole slide. Ideally, you want to produce something like in this:


But I don't know how to apply different formatting to different sections of the slide using this method. The only way I know how to do that is using Mail merge which I'll post about later. With this method, you can get a backlog out of Rally and onto paper and then tape the paper on the walls for your agile planning events.

If you know of any refinements to this or any other ways to get data out of Rally and onto cards or paper, then I beg you to comment!

A Tale of Tasks Boards and Electronic Project Planners

Using electronic tools for agile planning initially seems like a great idea:

1. You have access to it anywhere you have access to the InterWeb,
2. You save environmental resources because you don't have to print things out,
3. You can quickly search through requirements since there is a DB,
4. Auditing and revision tracking can be applied so you can see what changed and pose the question to someone--why did you change to that?
5. You have a backup.

The problem with electronic planners is that:

1. They are easier to ignore than a big task board.
2. They are hard to use by a group during a meeting.
3. They don't allow spacial organization where task boards can have layout schemes that fit the problem at hand.
4. They suck the energy out of the users. The activity of getting up out of your chair and moving things around on a wall gets the blood and energy flowing.
5. They aren't as flexible. Task boards are limited to the imagination where electronic tools are limited by the code.

When I started doing eXtremeProgramming in 1999, we only used a task board. Today, many projects I work on use tools like Rally because nine times out of ten, we work with an off site product owner. This sucks and slows us down, but using Rally gives the product owner a way to manage the backlog.

Using Rally becomes an impediment to teams which are co-located. When the product owner and team can meet face to face to do release planning, backlog grooming, or sprint planning, physical cards allow work to get done faster and with more energy than an electronic tool. Remember a principle of agile is interaction over documentation, and fiddling with Rally is much less interactive and quickly feels more like a document review than a discussion over acceptance criteria on a story.

It's critical that electronic planning tools make it easy to convert to physical collateral which can be worked with in our three dimensional space where it can be slapped on a wall via a sticky, (or a card with tape) and moved around.

This isn't easy to do in Rally. But it is possible. Watch for my upcoming posts on this topic.