The following is an email I wrote to the upper management in our company when asked about things we can do to improve the process and what are the next steps of things we need to fix in our application.  It remains mostly unedited and contains my thoughts on what will make our company better when it comes to the software we're writing.  It wasn't written as a blog post.  It wasn't until after I finished with it that I realized that it was decent content for a blog post.  You'll find a very "XP-ish" theme throughout, and that isn't necessarily an accident.  Hope you enjoy! Please leave feedback if something moves you.

Overview

The following contains my thoughts on software development at Super Awesome Company. While I can address the specifics of the project that I see need to be fixed and modified (the “what”), I am instead going to focus on how these items get fixed. I am choosing to do this because over time the new website will have bugs that need to be fixed, enhancements that need to be made, or new features requested. For the long term health of the project it does not matter so much “what” is implemented but rather “how”. For that reason I am avoiding specific project level tasks in this document and speaking at a higher level.

Software has been being developed for over 30 years. While the internet is relatively young, software development by comparison is not. Super Awesome Company will benefit from a commitment to good software development practices. Further, it takes a great deal of commitment and discipline to reap the benefits these practices can bring. Many of my suggestions are based on these principles, which are principles I have committed to learning and adopting over the last several years.

The overarching suggestion I would make is make quality the focus. In order to produce quality, we must slow down. In order to go fast you must go slowly. Sacrificing quality should never be an option. Consider a car manufacturer who needs to get 100 cars built in a single day. If they can only build 80 cars a day, is it better for them in the long run to get 100 cars out the door but have problems or 80 problem free cars? Building deficient cars carries more expense that just fixing the broken cars, such diminished brand reputation. If the goal is 100 cars per day, measures should be sought to increase production while keeping quality high.

We don't need to look too far to see where in a rush we've done something that we've had to revisit several more times because we were "going fast". We've been rushed for nearly a year and I can't say that we are any further along than we would have been had we approached the project more methodically. In some areas we may be further behind where we would have been since we "made sacrifices" to get things done, which have ultimately only put us further behind. Again, to go fast we must go "slow" (methodical). Put another way, quality breeds speed.

Any process, principle, practice we choose should have quality at its heart. Regardless of what process, principles, practices we adopt they will only be successful if we are disciplined enough to follow them.

Planning

Requirements

Requirements need to be clear and understood by all parties. Before developers set out to coding we need to have a clear understanding of what the problem is. The developer is not able to provide a solution until the problem is fully known and understood. Often the perceived solution to a problem is not the best solution. Failure to fully understand the problem/scenario results in rework.

Releases

We need to have planned releases. The timeframe can be discussed and agreed upon by the team. In order to have successful releases we need to plan each release. Each release should be broken down into smaller time frames, iterations. During the iteration the work should be stable for each developer. In other words, they know what they’re going to work on and accomplish (see requirements above) for any given time period.

Metrics

We need to be measuring metrics for the purposes of planning. If we get 10 units of work done each week on average and we have 20 units of work to do, we can easily figure out what is left to do. Without any metrics gathering we can’t do any planning and we’re really flying blind. This will also help with project post mortems. For example, the “web order manager” project was originally estimated to take a week worth of development effort. Three months later, it is still not finished. This is a combination of putting someone with limited knowledge of the business estimating the project, an overly aggressive developer estimate, and having no metric to pull from on how long something should take.

Flexibility

We need to be disciplined in the process of building software, but also realize when the process needs tweaking that we should respond and not be afraid to tweak the process to meet the needs of our team

Design

Simplicity

We should strive for simplicity in our designs. “Clever hacks” are often not healthy for the life time of the project. Simple designs are easier to understand and improve upon. Simple designs also allow other developers ease in working on code. This is much harder to do in practice but it always pays off in the long run.

Design aspects dovetail with testing (later in this document). Typically complex designs prove to be hard to test. When we write code based on simpler designs we’ll find that our code is easier to test. We want both, simple designs and easy-to-test code.

Coding

Standards

We should implement a standard for code development. Right now you can find three different styles/methods of coding in the code base. This makes it hard for developers to work on code they did not author. Projects I work on outside of work require all new code meet the standard, otherwise it is rejected until it meets this standard. As a consumer of these projects I’m always grateful when I can move through any of the project pieces and the code always looks the same.

Testing

Given the environment and complexity of the code, we need to require the developers to be able to test their code in an automated fashion. Visual verification is orders of magnitude slower than automated testing. Once an automated test is written, it runs every time. By having a large collection of tests we can makes changes to certain parts of the application and use existing tests as a safety net to verify nothing else has broken. We currently have over 1,150 tests (about 30% of code has tests) that run whenever someone commits code into source control. Those tests run in about 20 seconds. Imagine how long it would take a user to test each of those cases visually every time a piece of code changes.

Two weeks ago I sat in a meeting adding functionality to shipping and enhancing rules around shipping hazardous items to Canada while people were finding issues because of a solid base of tests. Before the meeting was over both issues discovered were fixed and tests were added for these specific issues so we would be guarded against this happening again.

Unit Tests

All code must have unit tests that can be run in an automated fashion. If the code cannot be unit tested (tested in isolation from other components) it should be restructured in such a way that it can be tested. Very few pieces of code are truly not testable. As such we should have very few areas in our code base which are not tested.

Proving Bug Fixes Through Tests

When bug is found a test should be written before any attempt to fix the bug. This test proves to the developer the bug exists but secondly gives the developer a marker to know when the bug is fixed. As a benefit, this new test acts as a guard against this particular bug ever happening again. This test, when first written, should fail because the developer has not fixed the bug yet. The developer should then work to make the test pass, and by extension fixing the bug. The cost of adding the test is very small when compared against having the test guard against this condition ever happening again.

Code Coverage

We should implement a standard for new code coverage, code that is tested by an automated test. I would recommend somewhere in the 80%-90% range for code coverage. In order to write tests the developers will be forced to think about their code a bit more thoroughly.

Pair Programming

We should adopt pair programming as a method to both increase shared knowledge but also as a means to quality. Pair programming is two developers sitting at one computer for a period of time while code is written. From the outside it may seem impractical to devote two developers to a single computer, thinking it is inefficient. Industry studies however show that code written in a pair has a four-fold benefit:

1. 15% less code– Studies showed that less code is written in order to implement the same functionality. Less code means less to maintain and understand. Less code is also typically simpler.
2. Less bugs – Developers, when working in pairs, can catch each other’s bugs before they ever make it to production. We therefore save time that it would take to fix the bugs that would otherwise go undiscovered.
3. Shared domain knowledge – When “pairing”, each developer is training the other and sharing knowledge and rules around the business. This avoids silos and situations where only one person knows the code.
4. Increased Skill – When “pairing”, design, coding, and testing techniques are transferred between developers. This is often why many development shops pair their senior developers with junior developers; to raise the skill level of the junior developer.

Collective Code Ownership

No one person should be a silo/bottleneck for any area of the code. Any person should be free to make suggestions, fix bugs, or refactor any part of the code. Pair programming (see above) is one tool which seeks to address this.

Conclusion

While the above recommendations may seem like a lot, the above items all really go hand-in-hand. Consider:

• Collective Code Ownership is had through Pair Programming
• Standards are enforced by developers when Pair Programming
• Writing Unit Tests typically leads to Simple Designs
• A commitment to Code Coverage leads to more Unit Tests
• Gathering Metrics leads to better estimates and Release planning

Adopting the above will move us in a positive direction and ensure that all projects coming from our department will be of ever heightening quality. Once the quality bar is set, then we will begin to see the pace pick up; Remember, quality is a prerequisite to speed.

I don't typically link blog and I do my best to write substantive original material, but the following is too good to pass up.  It's how I feel and how I've approach work in the last year or so.  From Ayende's Reducing The Cost Of Change blog post this morning:

That mindset, at least for me, starts from the first line of code. I treat each piece of the project as utterly disposable. Since I don't really care how each individual piece works, I am able to roughly sketch a fair amount of the application very rapidly, and then focus on each of the items in isolation, and replace that with a much better implementation. I think that I stated before that I tend to rewrite most of my application core at least two or three times before I am happy with them.

When you have disposable pieces, it is no big deal if you mess up and need to start over, because the whole project is structured in a way that allows you to do so. Going back to using my current project as an example, the algorithm used for the core part of the system is crap. I thought it up while being on a coffee break, and it is enough to demonstrate what the software is supposed to be doing. I don't really care, because the moment that I do need the real algorithm, I can drop it in (need to change the implementation of a single method).

I can't improve on that, so I won't try.

In church last week we were talking about evolution when the term "Irreducible Complexity" came up.  For those who aren't familiar with the concept irreducible complexity is a term coined by Michael Behe to illustrate that complex system could not have evolved and therefore must have been created intelligently.  Behe defines irreducible complexity as:

A single system which is composed of several interacting parts that contribute to the basic function, and where the removal of any one of the parts causes the system to effectively cease functioning. (Darwin's Black Box p39 in the 2006 edition)

In his book Behe uses a mousetrap to illustrate an irreducible complex system. 

"If any one of the components of the mousetrap (the base, hammer, spring, catch, or holding bar) is removed, then the trap does not function. In other words, the simple little mousetrap has no ability to trap a mouse until several separate parts are all assembled. Because the mousetrap is necessarily composed of several parts, it is irreducibly complex." (Behe, 1996).

(This post does not seek to discuss the controversy of evolution vs. creationism vs. intelligent design)

How does this translate to software?  An irreducibly complex software is software that cannot be easily tested.  The "removal of any one of the parts causes the system to effectively cease functioning" really points to dependencies in software.  Do your tests rely on a database being there?  Do you get emails whenever some piece of code runs because as part of the method an email is sent?

Creationist Design

How often have you stepped into some code and have seen a method calling database, sending an email, doing some validation, and then returning some value.  Here's an example I pulled from Jimmy Bogard's post on separation of concerns:

   1: [DataObjectMethod(DataObjectMethodType.Select, false)]

   2: public static List<Customer> GetCustomers(int startRowIndex, int maximumRows)

   3: {

   4:     List<Customer> customers = null;

   5:     string key = "Customers_Customers_" + startRowIndex + "_" + maximumRows;

   6:  

   7:     if (HttpContext.Current.Cache[key] != null)

   8:     {

   9:         customers = (List<Customer>) HttpContext.Current.Cache[key];

  10:     }

  11:     else

  12:     {

  13:         customers =

  14:             (

  15:                 from

  16:                     c in DataGateway.Context.Customers

  17:                 orderby

  18:                     c.CustomerID descending

  19:                 select

  20:                     c

  21:             ).Skip(startRowIndex).Take(maximumRows).ToList();

  22:  

  23:         if ((customers != null) && (customers.Count > 0))

  24:             HttpContext.Current.Cache.Insert(key, customers, null, DateTime.Now.AddDays(1), TimeSpan.Zero);

  25:     }

  26:  

  27:     return customers;

  28: }

The code above will work however it's a bear to test, the code absolutely requires other items (database, HttpContext, ect) to be there in order to work.  I would venture to say that the person who wrote it did not build up the code through small iterations, but rather wrote it all in one fell swoop.

Evolutionary Design

In software, reducible complexity should be a goal.  A reducible complex system allows for pieces to be substituted for others so you can focus on one particular area.  Don't have a database in place, no problem, simply put a placeholder object in for all calls made to the database.  (Yes I'm alluding to mocking).  You can approach creating a reducibly complex system in two ways:

• Create the system as a whole, then reduce it's complexity
• Create the system incrementally

I prefer the former to the latter as I'm working on my TDD skillz.

Conclusion

While the analogies used here are not quite perfect, I do feel they illustrate the point adequately.  As a developer your goal shouldn't be to simply write code.  You should concern yourself with the lifetime of your application/code base.  Irreducibly complex systems, while working, can also have short life spans.  Think about the last time you've ever seen someone fix a broken mousetrap.  All the parts are too interconnected, the mousetrap is simply replaced.  If you're not careful your code may be viewed too difficult and time consuming to fix, and simply replaced.

Here's a conversation I just had with another developer who is struggling to get his team to adopt some agile practices:

developer: if you were starting from scratch, which would you think is more important to setup first? CI or a unit testing process?
me: unit testing
developer: wow
developer:  ok
me: why is that surprising?
developer: i look at the mess i'm in everyday and i'd think getting a good build system in place would be first...
me: I think unit testing provides more value in that it allows you to add features knowing you haven't broken other features
developer: testing above a development process?
me: yes
me: testing provides more customer value

Let me be clear in stating that I think both are great value and a necessity, however the challenge was to choose one, presumably the most important.  I chose unit testing because one of the reasons I enjoy writing software that meets someone's needs.  As a developer one of the most nervous moments is when your code is released into the wild.  Will it work?  Will users they use it how you intended?  As a developer one of the most satisfying experiences is watching a user fire up a program you've written and it works flawlessly.  I think most customers would agree with that as well.  Having code that works is what they want.  Having code that works builds their confidence in you and reassures them that know what you're doing.  Testing provides this for me.  In advance I can test different scenarios and program that in and put my program through it's paces.

Now let me speak a bit about CI.  While I believe that testing is a component of CI, the developer in this instance meant a build server that kicks off automated builds.  While an automated build is nice, it doesn't nearly provide the value that unit testing does.  It's the things your build server does that increases it's value.  One of those things in our build system is running automated tests.  If you remove automated tests from a build server all you have is a dummy machine that compiles code.  That provides some benefit, but minimally.  You can replicate the behavior simply by updating your code often and doing a compile on your own machine.

As an agile developer I seek deliver value to stakeholders.  When viewed through that lens, I think unit testing clearly wins.

What about you?  Would you test before build server?  Or would you choose the build server?  Why?

When I first graduated from college I worked on embedded systems where the process was a formal, military grade process.  When developing software I typically thought about what document I had to write next and what document(s) I may be missing.  In general, the MIL standards turned me off a bit to process, and for awhile left me thinking that process is an obstruction to true development.  As a 21 year old, you think code = development.  That was/is an immature point of view, but it was not a viewpoint that I was easily dissuaded from.  Further it's a point of view that many out there still hold dear, and not all who hold that view are recent college graduates.

Recently a small group of us decided to give Scrum a try and are using it to drive our current project.  Other than this effort, there is no formal process methodology in place.  My goal is to help change that in my new position.  Over the past several years many high value projects have never been worked on because there was no common backlog of all of the things that need to be done.

In my last job at Geonetric we adopted Scrum after the absence of a formal methodology and it provided a much needed framework around which to develop.  Many CEO's are not this up front with their deficiencies, but Eric is pretty awesome and very honest in his evaluation of Geonetric:

As we were in the middle of developing the newest version of our VitalSite product last fall, we weren’t making the progress we wanted-even though the whole team was running full tilt and putting in its best efforts. We had always been a bit informal about how we developed software-somewhere between draconian rigid requirements and completely freeform cowboy (and cowgirl!) coding practices. The problem was that being in the middle wasn’t working. So, we looked at some of the newest practices in the industry.

Eric goes on to list the benefits he's seen since adopting Scrum:

• Since adopting Scrum, the first two releases of VitalSite 5 have been on-target, meeting both scope and deadline requirements (5.0 in January and 5.1 in May).
• Software development is accelerating each sprint - we’re approaching twice the speed we had last year.
• There’s much more collaboration between disciplines
• The morale of the software team is higher
• Quality of the software we’re delivering is better

With results like that I can see why a CEO would like scrum.  As developer though, I like scrum in that it doesn't get in my way.  There is administrative overhead to Scrum to be sure.  However it's very "XP" in that there aren't scads of documentation to write.  Generally as a software professional I like to write code and solve problems and for the most part Scrum allows me to do that.  My development does not feel slowed down by Scrum, which I believe for most developers, is a must in order to be adopted willingly.

One side effect of scrum is that is causes me to think about the process in a pleasant way.  This morning at our standup meeting, a fellow developer and I got into a discussion about how his extra exploratory work in the evenings should be factored into our sprint.  Should we could his work towards our sprint commitment?  Will his work negatively/positively affect our velocity for future sprints?  From my experience, developers typically don't worry about this stuff however with Scrum the sprint commitment is something that should be taken seriously.  As such it was good to see developers, of which I was one, having talks about other things that just the code.

I'm sure many other processes out there could achieve the same thing.  Scrum for us, both in this job and the previous, provided a nice framework that was palatable to developers. In that respect I'm perfectly happy using Scrum.  I will keep my ears open for new ideas and processes, but I think for any development methodology to be successful in an organization, it has to be readily and whole-heartedly adopted.  We've enjoyed Scrum so far and as such, it's a not-so-bad development methodology.  If you aren't using a methodology or your current process isn't working, you should give Scrum a try.

We're all familiar with the Aesop's fable of the tortoise and the hare.  In the story, the hare, who is in every way is faster than the tortoise, loses a race to the tortoise.  The main principle of the story is that slow and steady wins the race.

In my development I am shooting to be a tortoise, really I am.  Read on and let me explain.

A few weeks ago I got some evil glares when I suggested at our .NET user group meeting that in enterprise systems that you don't have time not to test.  It's a common hurdle for those new to testing to say, "I don't have time to write tests."  Let's face it, we're all busy, that excuse is tired.  As an agile and lean practitioner I seek out ways to improve velocity and reduce waste, not take my already busy schedule and cram in another tool for the sake of another tool.  While writing tests does slow me down, it brings on tortoise like speed, which I would argue is a good thing.  Writing unit tests is one tool that provide me the ability to keep a more consistent velocity over the course of development.  Without tests I can surely write things faster, the problem arises as the codebase grows and each new feature or fix takes increasingly more time. Eventually, even simple requests become arduous to implement.  Slowly you see your velocity come to a crawl.

It's an insidious cycle that I've seen before and am currently in the throes of; an application is built from scratch implementing everything the business requires. The application is enhanced and bolted on to, until you realize that you could move much faster if you could start from scratch.  You make pleas to your boss and explain how much productivity would improve if you could shed the hideous code base.  One day he gives in, you rejoice and you make the leap, start from scratch, breathing a sigh of relief at how easy implementing the features are all the while reminiscing about the old framework and how poorly it was written.  And now that the application is rewritten from scratch the cycle, unless you were aware of it all the while, starts again.

You see, no one sets out to write crap code.  People do the best they can with the knowledge they have.  Code, following the second law of thermodynamics, tends towards chaos over time.  With unit tests in place I can refactor with more confidence and implement new features without the fear of breaking existing code.  If you have the ability to refactor, new code is no longer "bolted on" but rather "grafted in" becoming part of the system.  With a solid framework with tests in place you can much better stop the cycle of rewrites.  Quickly writing applications that degrade is the way of the hare.  Developing purposefully using unit tests causes me to be slow in the short run, but over the life of the application comes out far ahead.  In that way, I strive to be the tortoise.