The Shade Tree Developer

The Shade Tree Developer is moving to CodeBetter.com

2005-07-18T09:43:00.000-05:00

Sorry for the inconvenience, but I'm moving The Shade Tree Developer to CodeBetter. Besides keeping some great company at CodeBetter, I'll get to have a bit better blogging infrastructure like trackbacks and categories so you can skip over my recent rantings about legacy code to get to the TDD content. I might also try to remember enough about CSS to get a stylesheet that doesn't make my code samples look fugly. I'll put forwarding links on the blogspot location for the older posts.

The new address is http://codebetter.com/blogs/jeremy.miller/.

The new RSS feed is http://codebetter.com/blogs/jeremy.miller/rss.aspx.

An Atom feed is available at http://codebetter.com/blogs/jeremy.miller/atom.aspx.

Resources from the Continuous Integration talk last night

2005-07-15T07:53:00.000-05:00

If you were at the Continuous Integration talk last night at the Austin SPIN, here is the set of resources I mentioned. The slides from the presentation are here or at http://www.jwspro.net/aspin/. Thanks for coming out last night in the rain and putting up with me.

Articles

Continuous Integration by Martin Fowler and Matthew Foemmel
Evolutionary Database Design by Martin Fowler and Promod Sadalage
Integrate Often from extremeprogramming.org
Driving on CruiseControl by Lasse Koskela. Setting up CI in the Java world.
The Agile Difference for SCM by Brad Appleton from CM Crossroads
Mike Roberts on multi-project continuous integration

Tools

I published my company's .Net tool usage with descriptions and links here.
CruiseControl (Continuous Integration for Java)

Books

I know there has been a couple of books published on CI now, but I don't know any to recommend. I heartily recommend Michael Feather's Working Effectively with Legacy Code for retroactively applying Test Driven Development and CI.

Bunch of Good Links

2005-07-14T13:49:00.000-05:00

First, maybe the best blog post I've every read from Mike Spille. I've been guilty of the "guardrail to guardrail" effect a time or two. On a project last year we probably did some dumb stuff because of someone else's negative experience with a certain technology on his previous project.

Read the comments too --> "BTW, you left out the obligatory insult to VB."

In a similar vein, James Bach warns us against being absolutist about "Best Practices." One of my favorite quotes is "A maturity model is basically a gang of best practices hooked on crystal meth." Just to add my 2 cents worth, I think it's best to always keep an eye on the first causes. I've written before that a centralized architecture group handing down black and white mandatory best practices from Mount Olympus to the pitiful wretches in the trenches is a terrible organization anti-pattern. There is always some kind of exception to every rule and the guys doing the actual work should be capable of exercising their own judgment.

Lastly, integration guru Gregor Hohpe asks what the real difference is between coding and configuration. Here you go Gregor, the difference is that I can write unit tests for code and use a debugger to troubleshoot code. I don't know why people keep inviting Gregor to these kinds of conferences because he just mocks the proceedings in his blog later.

My StructureMap tool is very configuration intensive and I have an unnatural tolerance for external XML configuration. To compensate, I've tried to provide the usage of .Net attributes for some configuration to be closer to the code. I've also had to invest a lot of energy into creating better tools for troubleshooting and validating configuration files in StructureMap. Since I dogfood StructureMap, expect more and better troubleshooting tools (because my colleague gets irritated otherwise).

Impressions from the VSTS talk last night

2005-07-12T10:00:00.000-05:00

I went to a talk on Visual Studio Team System last night given by Chris Menegay at the Austin DotNet Group. I tried to go into this with an open mind, but my bias is clearly with open source tools and agile processes. I'm still not convinced VSTS provides any value beyond our current tools and process, but it'll help somebody. Either way, I think VSTS is a valuable addition to the .Net world. If nothing else, it's helped introduce lifecycle practices to shops that have none and started some healthy conversations.

I wrote this several months ago, and nothing I saw last night substantially changes my opinion.

http://jeremydmiller.blogspot.com/2005/04/bypass-vsts-and-get-yourself-poor-mans.html

Without further rambling, cue the "wah, wahhh, wahhhhhh" theme music and let the cliche begin...

The Good

I think VSTS could be great for project tracking. Being able to intelligently log checkins to a user story or a bug fix with the time spent on the task could be a great boon to project management. The Heisenberg Principle always applies to this kind of tracking. If I, the developer, have to go out of my way, my velocity is going to be slower. VSTS might change the cost/benefit equation somewhat. A lot of any Agile process is the ability to accurately forecast user stories during iteration planning. Using a good "yesterday's weather" report from past iterations could certainly help. We're still going to take a look at Trac though as an alternative.
The code coverage visualization is awesome. It's not clear yet, but it might be possible to create test coverage reports from other tools outside of VSTS (NUnit, NFit, etc.)
It is an integrated suite of tools that will be supported by MS. I personally don't buy the argument that assembling a stack of NUnit/NAnt/CC.NET/NDoc/Subversion is really that hard, but I've been immersed in the agile world for awhile and I'm comfortable doing that (I've also worked with some of the NUnit and CC.NET folks too, so I'm not really impartial). For a shop with no experience in these tools or with a hostile policy towards open source, I can see the value of buying the whole stack in one place.

The Bad

Call me close-minded, but I wouldn't touch MSF with a ten foot pole and a firewall in between. The process templates are editable, but it doesn't sound like there is a GUI yet for doing the editing. I'll be interested to see if anybody (with much more patience and ambition than me) tries to make an XP or Scrum template.
At this point it doesn't look like you can use any other source control system than what comes with VSTS. As much a pain as it has been trying to move from VSS to Subversion, I'm not very excited about another migration. The source control is clearly better than VSS (how could it not be), but I didn't see any particular value over Subversion today.
VSTS does not support Continuous Integration, only scheduled builds. You'll need to continue using CruiseControl.Net. I fully expect the CC.NET guys to have a plugin for VSTS and/or MSBuild for CC.NET by the end of the year.
The web testing looks weak to me. I still think we're looking towards either Selenium or Watir.

The Ugly

From the initial impression, the MSF for Agile process looks like a heavyweight process. It is clearly meant as an iterative process. While that's certainly an improvement on the older MSF, iterative alone doesn't automatically translate to being agile.
VSTS is clearly aimed at very large companies using laborious, high ceremony processes--everything that my employer is not. I think process automation is a great thing, but if your process is so complex that nobody can know the whole, I think you've got some serious issues.
Pricing. When alternative tools are available and generally free, I have a hard time justifying the cost.
The one piece of the presentation where I thought Menegay was completely off his rocker was the idea that VSTS became a directed workflow tool to tell everybody (expecially developers) exactly what they should be doing. As a tracking tool for project record keeping, I think VSTS is great. Using something like VSTS as your primary communication channel is absurd. Turning developers into mindless zombies doing exactly what the VSTS workflow tells them to do sounds like a recipe for disaster to me. Not to mention a steep drop off in developer retention.

Don't play with strange databases...

2005-07-11T10:03:00.000-05:00

...you don't know where they've been, or who's been touching them.

I've been burned pretty badly in the past by making assumptions about existing databases. Database documentation isn't always helpful either, because it is out of date or just wrong to begin with. Even if the documentation is "correct," the corresponding code may be using it differently than the database designer intended. Even worse, you often have to interpret the data, and that's dangerous when you don't know the database. The only accurate source of information is usually the subject matter expert for the system.

What's the answer? I don't know. All I can say is to approach someone else's database with great caution. Kinda like going up to a big, mean looking dog and trying to make friends. I wouldn't make any kind of assumption about the meaning of any table or field. Put some effort into understanding the schema and expect to go back and forth a little bit.

It's the year 2005. Isn't the existence of SOA supposed to eliminate the need to be doing integration directly against a database? I'm writing a little code to integrate our main application with another 3rd party system. The only way I have to do this right now is to just write SQL queries against their database schema. I feel dirty.

Even worse is when shops start writing their own extensions and back door queries against a 3rd party application. That's a great idea. Take a database that you're not supposed to touch directly, and that you don't fully understand, and write all kinds of custom code into it. That'll really make it easy to upgrade the software package later.

Don't Make It Worse

At a previous employer they use an ancient inventory system that is written in a rare 3GL language. If you think of your IT infrastructure as the cardiovascular system of a manufacturer, the inventory system is the heart muscle. Everything else talks to the inventory system. The business logic of the system was bound up in the UI screens, so the only front door integration point was a screen emulation package from the vendor that was only certified for low volumes. At this point an intelligent IT organization in the SOA era would start looking for alternatives. My old employer beat the integration problem by writing hundreds of PL/SQL procedures to duplicate the business logic in the screens in lieu of a real service layer. To support other functions they added about 300 custom tables to the out of the box database. Just to make things worse, the database customizations were different region to region.

At one point I needed to write an integration from the inventory system to my application. I went and asked the SME where I could find a certain piece of information. He told me to look in table A and I turned around to leave. He then said, "you could also get it out of table B, or table C come to think of it." Oh, ouch. When I left they were struggling with a batch job that was supposed to run nightly that was taking about 25 hours to run and locking all kinds of database tables. They've experienced some problems with scalability, who would have guessed?

Even though the system is horribly obsolete and arguably an opportunity cost slowing down new development, they have no chance of upgrading to the newer J2EE version with web services because their database is way too wired. Idiots.

Harris Boyce warns us that O/R isn't a silver bullet

2005-07-11T09:09:00.000-05:00

Harris Boyce wrote a response to my ADO.NET contagion post: "Soul Vaccination" for Data Access Layers.

I think that Harris is quite right in warning us about overusing Object/Relational mapping.

While there are still plenty of scenarios where I would forgo an O/R approach for the classic data access layer, I still think that the "Persistence Ignorant" (POCO/PONO/POO whatever) approach for business classes is the way to go for the sake of testability. I think the decision hinges primarily on the amount of business logic in your application (and secondarily on the comfort level of the developers with OO coding in general). Even if I'm not using an O/R tool, I usually use some type of Inversion of Control to keep the rest of the code loosely-coupled from the data access.

Before diving into O/R mapping, you might take a look at these resources. Understanding some of the underlying patterns and mechanisms of O/R mapping might help alleviate some of the pain or help in making design decisions.

Patterns of Enterprise Application Architecture by Martin Fowler has a couple of good chapters about O/R patterns. The chapter on organizing business logic is a canonical read for deciding whether or not to use a Domain Model approach that would lead to using O/R for persistence.
Applying UML and Patterns by Craig Larman has a great chapter on creating a small persistence framework. This book is my choice as the best introduction to OO programming around.
This article from Scott Ambler introduces the basics of O/R
My article on C# Today from a couple years ago, but it's derivative of the guys above.

Continuous Integration Presentation at the Austin SPIN

2005-07-10T17:43:00.000-05:00

I'm giving a presentation on Continuous Integration at the Austin SPIN meeting on Thursday, July 14th. The meeting info is here. It's an adaptation of the talk Steve Donie and I did for Austin Agile a while back, just less .Net-centric for a broader audience.

I'm excited (OT)

2005-07-10T14:01:00.000-05:00

Communications technology is undeniably changing our world. "The World is Flat" type effects may act as a downward force limiting our salaries, but my wife just let me sign up for the NFL Sunday Ticket and that's pretty darn cool. I live in Texas, so every Sunday at noon my football viewing choices consist of mediocre Cowboys on one channel and mediocre (but improving) Texans on the other channel. I grew up in Missouri, so I'm a lifetime Kansas City Chiefs fan. Thanks to a little technology I'm gonna be watching my beloved Chiefs in my living room this year instead of trying to sneak over to a sports bar to see the game.

And if the Chiefs' new linebackers still can't stop my Grandmother from running up the middle, I'll just change to a different game.

When I was a Mort...

2005-07-08T08:57:00.000-05:00

Yesterday at lunch we were discussing (mocking) Rhocky Lhotka's defense of "Mort's" on his blog. What the heck is a Mort you ask? A Mort is supposed to mean an opportunistic programmer who is most concerned with delivering business functionality rather than wasting any time on silly ivory tower concepts like technical quality. Unfortunately the term "Mort" has become a pejorative term synonymous with low skill developers using data aware widgets to drag'n drop their way to one tier applications. I'll admit that I commonly use the term Mort as a putdown, but there was a day and time when I was a Mort, too.

My first programming experience was writing some ASP and Access tools for my engineering group in the late 90's. At that time the engineering and construction world was pretty crude in terms of IT automation. There were a lot of data silos, and quite often the most junior engineer (always me) was stuck manually typing information contained in one database into Excel sheets. We had to generate a lot of paperwork and track a lot of audit style information. I hated the paperwork aspect of engineering (hence my gravitation towards agile processes), so I set out to create some automated tools to create the Word and Excel documents from information in an Access database that was edited by ASP pages. This work led me to a position with the project automation group creating an ASP system to verify and manage a very poorly written, but mission critical, data exchange.

I clearly created a lot of business value and I was pretty proud of myself. Then I left the company to relocate to Austin, and everything I left behind collapsed in a few months because no one could support it. Some of it was rewritten by actual IT folks (and consider this a long overdue apology to you all), but most of it just disappeared. So what did I do wrong? Here's a laundry list of really bad things I did because I just didn't know much about good practices in software development.

Source control? What's that?
(Boss) Jeremy, where is this stuff running? (Me) It's running on personal web server on my box. Hey wait a minute, what's that awful sound coming from my hard drive and why won't my box reboot anymore?
(Boss) Do you have that stuff backed up? (Me) What does 'back up' mean?
Coding directly into a production database of a 3rd party product that was notorious for database corruption on projects ranging from little 100 million dollar projects to multi-billion dollar projects
ActiveX controls on ASP pages. Believe it or not we had some issues with installation.
ASP pages running against MS Access via ODBC in production
Connecting directly to production Oracle databases from ActiveX controls on an ASP page
Using the old Remote Data Service (RDS) library. Great stuff for productivity, huge security hole. Using RDS meant that I had the user name and password for the fragile Oracle production database embedded in each ASP page where any yokel could go "View Source" and hack our sensitive data.

I was focused on business needs and delivered, but I was dramatically ignorant of anything approaching decent development practices or design. I distinctly remember one humbling episode where I asked a senior automation expert for help on comparing two different databases. He took a very short look at my several hundred lines of VBScript and banged out a SQL statement that did exactly what I needed to do in one line of code. His exact words were "you've got good ideas, but you would be much more effective if you just knew more." Ouch.

All right, I just shared some of the truly stupid and ignorant things I did in my Mort days. What did you do in your Mort days?

What's so wrong with being a Mort? After all, they're pretty common and they definitely add value to their employers. They're usually closer to the business and have a better understanding of the problem domain. For one thing the Mort's of the world are in mortal danger if I have to work with any more awful Mort-ish code (regardless of what language it's written in). For another more serious reason, the Mort jobs are the first developers to be offshored. Lastly, if you're developing software, learn about your craft and climb the skill ladder. The whole Mort/Elvis/Einstein taxonomy may be crap, but the more you know the more effective you'll be and that seems pretty pragmatic to me. Make your great business solutions stand the test of time.

Great CMM quote at lunch today

2005-07-07T15:32:00.000-05:00

At the Agile Austin lunch today we had a great quote. We were discussing using a continuous integration build to auto-generate documentation specifically for CMM compliance when somebody said "[doing CMM paperwork] is like paying protection money to the mafia."

Quarantine the ADO.NET Contagion in your Code

2005-07-07T10:28:00.000-05:00

From my perspective the .Net world is clearly becoming more sophisticated in our approach to persistence. Reusable persistence solutions like NHibernate or NEO are starting to become more popular, reducing the need or desire to work with ADO.NET directly.

However, most of the .Net systems out there that I encounter still have old fashioned data access layers rolled by hand. This doesn't have to be that bad, ADO.NET is pretty easy stuff, and the .Net community is generally pretty strong in data access. The problem is that ADO.NET usage has to be done right, every single time. Forget to close a connection or an IDataReader somewhere, then add in enough volume through the system, and watch your application roll over and crash because the database has too many open connections. The same thing with exception handling occurs. You have to have the same "try/catch/finally" code every single time you use ADO.NET code.

The second problem is a little more endemic. Because ADO.NET is so easy to use, developers coding a web page or a business project give into temptation to just write the data access code right where it is needed instead of separating out the data access concern with a little thought. These people are often colloquially known as "Mort's." To Mort, please stop doing this. Especially if you're writing code that I have to support. Just in case it's not obvious, writing the ADO.NET code straight into a business object basically eliminates any possibility of writing easy unit tests.

My advice is to simply count the number of times the following lines of code show up in your application. Offhand, I'd say the maximum number of times this code should be called to be no more than 3-5 times.

connection.Open();
command.ExecuteNonQuery();
command.ExecuteReader();
connection.Close();

You can eliminate a grotesque amount of duplicated code by simply centralizing the low level ADO.NET manipulation into one or very few classes. It's much easier to write the error handling and connection management once than it is to get it right everywhere. It'll also allow you to create a better answer for tracing and instrumentation. All other higher level data access layer classes call into the central execution class to actually execute database commands. Other than the possible exception of DataSet's, nothing from the System.Data namespace should ever leak out beyond the classes that are specific to data access.

Here's an example of writing the inner code execution class from our data access layer:


using System;
using System.Data;
using StructureMap.DataAccess.ExecutionStates;

namespace StructureMap.DataAccess
{
 [Pluggable("Default")]
 public class DataSession : IDataSession
 {
  private readonly IDatabaseEngine _database;
  private readonly ICommandFactory _factory;
  private readonly IExecutionState _defaultState;
  private readonly ITransactionalExecutionState _transactionalState;
  private IExecutionState _currentState;
  private readonly ICommandCollection _commands;
  private readonly ReaderSourceCollection _readerSources;

  [DefaultConstructor]
  public DataSession(IDatabaseEngine database)
   : this(database, 
   new CommandFactory(database),
   new AutoCommitExecutionState(database.GetConnection()),
   new TransactionalExecutionState(database.GetConnection()))
  {

  }

  /// 
  /// Testing constructor
  /// 

  /// 
  /// 
  /// 
  /// 
  public DataSession(
                    IDatabaseEngine database, 
                    ICommandFactory factory, 
                    IExecutionState defaultState, 
                    ITransactionalExecutionState transactionalState)
  {
   _database = database;
   _factory = factory;
   _defaultState = defaultState;
   _transactionalState = transactionalState;

   _currentState = _defaultState;

   _commands = new CommandCollection(this, _factory);
   _readerSources = new ReaderSourceCollection(this, _factory);
  }



  public int ExecuteCommand(IDbCommand command)
  {
   try
   {
    return _currentState.Execute(command);
   }
   catch (Exception ex)
   {
    throw new CommandFailureException(command, ex);
   }
  }

  public int ExecuteSql(string sql)
  {
   IDbCommand command = createCommand(sql);
   return this.ExecuteCommand(command);
  }

  private IDbCommand createCommand(string sql)
  {
   IDbCommand command = _database.GetCommand();
   command.CommandType = CommandType.Text;
   command.CommandText = sql;
   return command;
  }

  public IDataReader ExecuteReader(IDbCommand command)
  {
   try
   {
    return _currentState.ExecuteReader(command);
   }
   catch (Exception ex)
   {
    throw new CommandFailureException(command, ex);
   }
  }

  public IDataReader ExecuteReader(string sql)
  {
   IDbCommand command = createCommand(sql);
   return _currentState.ExecuteReader(command);
  }

  public ICommandCollection Commands
  {
   get { return _commands; }
  }

  public IReaderSourceCollection ReaderSources
  {
   get { return _readerSources; }
  }

  public void Initialize(IInitializable initializable)
  {
   initializable.Initialize(_database);
  }

 }
}

using System;
using System.Data;

namespace StructureMap.DataAccess.ExecutionStates
{
 public class AutoCommitExecutionState : IExecutionState
 {
  private readonly IDbConnection _connection;

  public AutoCommitExecutionState(IDbConnection connection)
  {
   _connection = connection;
  }

  public int Execute(IDbCommand command)
  {
   try
   {
    setupConnection(command);
    return command.ExecuteNonQuery();
   }
   finally
   {
    cleanupConnection(command);
   }
  }

  private void setupConnection(IDbCommand command)
  {
   command.Connection = _connection;
   _connection.Open();
  }

  private void cleanupConnection(IDbCommand command)
  {
   try
   {
    command.Connection = null;
    _connection.Close();
   }
   catch (Exception)
   {
   }
  }

  public IDataReader ExecuteReader(IDbCommand command)
  {
   try
   {
    setupConnection(command);
    return command.ExecuteReader
(CommandBehavior.CloseConnection);
   }
   catch (Exception)
   {
    cleanupConnection(command);
    throw;
   }
  }
 }



}

In this example, all data access classes have to use a class called DataSession
to execute an IDbCommand object or a sql string. The DataSession class delegates to an IExecutionState interface object. The AutoCommitExecutionState deals with all of the connection opening and shutting, as well as providing a consistent error handler to report the actual sql (with parameters) that failed. By centralizing this operation to one spot, we could add extra code to provide more instrumentation or simply change the error handling across the board. Most importantly, other classes don't have to be obfuscated by mucking with so much ADO.NET manipulation.

My Poor Friend

One of my previous career stops was at a very bad internal IT shop. The leadership in our division was making it painfully clear that developers were merely a commodity, and undeserving of anything like good working conditions, raises, or a career path. After years of abuse working on a truly horrendous VB6 system (inspired by the architectural writings of a "Super Mort"), a friend of mine named Scott secured a transfer to a different organization. Scott was baited by promises of getting to do hard core J2EE development, the hottest thing going at the time.

Upon arrival at the new digs, Scott was told they needed him to help clean up the existing system first to improve stability before he could work on the new J2EE work. It turned out that the existing system was a 900 page ASP classic application that had been written by people that had grown into a mission critical application. The instability problems turned out to be because none of the shadow IT developers had closed any of the ADO connections in the ASP code, causing the database (I don't know what the database was, but it wouldn't surprise me if it had been MS Access) to crash and burn with too many open connections. Scott's new Job? Go through each and every ASP page and add code by hand to close connections, commands, and recordsets.

I know what you're thinking, just write a fancy Perl script with some regex magic and go on. No such luck, the code was too inconsistent. Just in case you've never been afflicted by classic ASP code, take the worst code you've ever written and put it into a blender. That's about what you'll find in ASP (my ASP code circa 1998 was pretty smelly).

I Don't Think Much of the DAAB

Just to be opinionated, I don't see much value in the original Data Access Application Block or the improved replacement in the Enterprise Library. In my opinion, they just don't add much value. It's still a low level API with not much there to guarantee good data access practices. I know a lot of people like them and they're better than nothing, but there are so many better tools out there. For that matter, I just don't think it is difficult to make a data access layer specific to your application that is superior to the DAAB in terms of connection management, transaction management, error handling, and instrumentation.

Attitudes towards Stored Procedures

2005-07-06T10:25:00.000-05:00

My philosophy towards stored procedures has shifted so much over the past five years that I'm not sure I have a strong opinion on them anymore. At one point I would have told you with great fervor that you had to be doing data access through stored procedures to have a good application architecture, now I'm not quite sure. On the other hand, I've seen developers write hundreds of lines of procedural code to do set-based manipulations that would have been relatively simple to express as a SQL statement.

For me today the decision to use stored procedures comes down to the particulars of the application, the comfort level of the rest of the team, the relationship with an application DBA (if he/she exists), and the relative pain or ease of configuration management for database code. For whatever reasons, a lot, if not most, shops do a terrible job with configuration management for database code and structure. As a consultant I worked on an engagement with a client that went through way too much manual process rigamarole just to get a stored procedure moved into the development server. Of course, they handled the database code moves in an unrelated process separate from the code, allowing all kinds of bad things to happen.

I spent a lot of my early career writing Oracle PL/SQL, so I’m pretty comfortable with using and writing stored procedures in general. And to be honest, I think a lot of ex-DNA developers (victims) like me are inclined to write sproc’s because they were often easier to change and deploy than COM(+) DLL’s, especially with a clustered server architecture. Case in point, my current employer has a deep VB6/ASP/Sql Server background and stored procedures in T-SQL run rampant throughout our architecture. Developers who aren't comfortable with object oriented programming will often use stored procedures as an extensibility mechanism (I don't like this, but it does work).

When I moved to a shop doing TDD that was mostly stocked with J2EE veterans I was a little shocked at their disdain for stored procedures. In their world order, stored procedures were evil. Not very many of them could explain exactly why stored procedures were problematic, but they all knew it for established fact. Later on I started to understand their point of view. J2EE has a multitude of proven persistence mechanisms that isolate middle tier code from the database and eliminate a lot of manual SQL creation. In most cases, there was also a fear that using stored procedures would inevitably lead to business logic in the database. A general discomfort with relational databases probably explains a lot of the hostility towards stored procedures in some developer circles.

For many people the most important reason was that stored procedures are inherently more work to test than plain old objects. Just as importantly, unit tests that touch the database are slower to execute. A domain model approach coupled with some kind of metadata-driven persistence mechanism can lead to much easier unit testing and faster development overall. Stored procedures often simply interfere with persistence strategies.

Some O/R tools support stored procedures, but it’s often more trouble than it’s worth. If you want both O/R mapping and stored procedures, I’d actually recommend rolling your own tool. Writing a complete solution like NHibernate is hard, but creating a just good enough custom tool isn’t that bad. Reflection in .Net is pretty easy to use, and the System.Convert class makes type coercion relatively painless. If your domain model is small, just hard code the database mapper classes and move on with your life.

Enter the DBA

Another obvious factor in an organization’s attitude towards stored procedures is the relative strength and influence of DBA’s. Developer/DBA conflict can be even more harmful than the spitball fights between developers and testers. If a DBA is the only person allowed to write SQL, or has to approve every piece of SQL, you’re going to go slower. I hate having this kind of territorial bottleneck in the development process, but when you’re stuck with this situation my advice is to:

Keep your DBA engaged in the development work of the rest of the team. Don’t let them work in a vacuum. It helps tremendously if the DBA actually has a stake in the success of the project.
Aggressively use mocks or stubs to build business logic and user interfaces so you aren’t stuck waiting on a DBA. This also has the advantage of firming up the data access requirements before engaging the DBA.
The DBA must follow the exact same configuration management practices as developers. Ideally, you want a DBA to check any changes into source control and monitor the continuous integration build and fix any problems from their check-in's. Stored procedures are just code and should be treated as code. Make sure you help the DBA out with check in policies because it is a different world for most of them. It’s also cool as a way to give the DBA’s some visibility into the project at large. I'm plenty paranoid about this one because of some early issues on a project last year when the DBA decided to start optimizing some PL/SQL and broke the application for a while.

Good and Evil in the Garden of Stored Procedures

2005-07-05T13:03:00.000-05:00

Periodically I see or engage in an argument over the usage of stored procedures. I think a lot of these types of arguments are somewhat unproductive because the combatants are thinking and talking about radically different things. Besides, I think the argument over the proper role of stored procedures in data access is often obfuscated by the misuse of stored procedures for things other than data access. My opinion has changed quite a bit through the years, so I try to stay out of these arguments.

However, my development group has to make a change soon in our philosophy for using stored procedures (or hit a brick wall). The way my company has used T-SQL in the past is jeopardizing our future development (besides, I’m an Oracle guy and I think T-SQL is butt-ugly). We’re a product development company that has grown by leaps and bounds from a startup venture and the existing codebase has grown somewhat chaotically. To keep the growth going and accommodate new customers both here and abroad we’re going to need to localize the application for different languages and port our main application to Oracle. Toss in the ongoing adoption of TDD and continuous integration as a means of creating better software quicker, and the usage of stored procedures has to be reevaluated.

What’s Good about Stored Procedures

Before I start ranting about the grotesque misuse of stored procedures I’m dealing with, here is what I think is useful about stored procedures.

Using stored procedures externalizes the SQL from the compiled code. I really like having the SQL in an easy to read format instead of built by string concatenation in the code. This is also great if you need to optimize the stored procedure for performance. I haven’t seen that come up very often, but it only seems to happen on systems where the SQL is embedded in the code. There’s also the issue of allowing a DBA to alter or write stored procedures independently of the code. I have generally had control of both database and code throughout my career, so I’m always suspicious of DBA’s writing stored procedures instead of developers. I do think that dynamically constructed SQL can result in systems that are hard to debug, but that can be beaten with just a little judicious instrumentation in your data access layer.

Performance is an advantage of stored procedures, but maybe not enough by itself to justify stored procedures over other approaches. I think the performance advantage is either largely mitigated by the ability of newer database engines to cache execution plans or simply a case of premature optimization. If you have an opportunity to batch up a set of updates or simply reduce the amount of information you send over the wire to the database a stored procedure certainly leads to a performance improvement. Otherwise, I think the stored procedure advantage is too slight to be a serious consideration.

Security is another advantage that is often cited as an advantage of using stored procedures. On one hand, stored procedures quickly mitigate vulnerability to SQL injection attacks. On the other hand, the database can be tightened down so that no ad hoc SQL can be executed, stopping security breaches right at the start. Personally, I abhor this idea myself because it drags down the development team’s velocity. I don’t see why it’s necessary, but it is an option.

Most of this goodness can be happily achieved with nary a stored procedure. Parameterized queries can eliminate SQL injection vulnerabilities while providing comparable performance. The data access layer we built (I might release this as part of StructureMap someday) for our last project already externalizes SQL from the code for ease of maintenance and decoupling from the database.

When Stored Procedures Go Bad

While there is nothing inherently evil about stored procedures, developers often misuse stored procedures. I think stored procedures should be nothing but basic CRUD. I wrote in an earlier post that “IF/THEN/ELSE” logic in stored procedures is a design smell.

Stored procedures are a terrible place to create business logic. Development environments for the .Net languages or Java are far better suited for developing applications. Intellisense support, debugging, and refactoring are all inferior with stored procedure languages. T-SQL is a procedural language that pretty well forces you into the transaction script style of organizing business logic. One thing I’m observing in the reams of T-SQL I’m wrestling with is the obscene amount of code duplication. Now that we have to support Oracle as well, this duplication stands out like a sore thumb. An OO language can do a much better job of code reuse.

Here’s my list of things I’ve seen put into stored procedures that should never be in a stored procedure.

User input validation that creates the validation messages seen on user screens. I can’t even begin to describe how stupid this is.
Creating HTML, enough said.
Security authorization logic. This is definitely a middle tier responsibility. I’ve used joins to some sort of permission table in select statements to filter records. I think that’s fine, but using procedural logic or even role checks inside stored procedures is nuts. Security is volatile and sensitive. Put it where it’s easy to test. For that matter, always leave a way to test your business logic without security checks if you can. Security always makes code harder to unit test. By wrapping security up with business logic in a bag of stored procedure goo, you’ve essentially shot yourself in the foot for testability.
Business logic. It’s easier to write and test in C#, period. Splitting logic between stored procedures and middle tier code is even worse, but I’ve seen it done many times.

Insight from Scott Bellware

2005-07-01T15:58:00.000-05:00

Fellow Austinite and champion ranter Scott Bellware has some great insight on the impedance mismatch between application architecture and the newer SOA paradigm.

Overall, I think SOA is an advance in the construction of software solutions, especially in large IT infrastructures. What always bugs me is the seeming disdain (or ignorance) the SOA guys have for good OO or even application design. I'm familiar with the situation and perpetrator of the (very foolish) guidance of mandatory SOAP web services between the UI and backend (even if the code is all running in the same process space!) "just in case" they need a connected service later. In no way does the existence of SOA services abrogate the need to create well-factored solutions. Write a well-structured code base and your application will always be able to expose SOA endpoints later. And if you're doing SOA, keep the internals of your service or system completely independent of the messaging transport. Like Scott implies, serializing your real domain classes as part of the SOAP contract strongly couples your clients to the internals of your service.

Create new entry points to enable unit testing of Legacy code

2005-06-30T12:41:00.000-05:00

If you've been reading the Shade Tree Developer this week you'll know that I'm fighting with a legacy component. I was trying to get a piece of code into NUnit to test my new functionality. A class in the stack works by giving it a key value to a record in the database. I tried for awhile to use an existing stored procedure to push in test data. Guess what I found? I had relational integrity issues with setting up the test data, but so what, that's just what a database is. I wasted a bit of time pushing past the referential integrity by getting the correct data and stumble right into security authorization being performed inside the f***ing stored procedure!

It was about this time I remembered what I had been reading in the Feathers book on legacy code. Sometimes you have to change the code just to get it into the test harness. A quick "CTRL-ALT-M" action with ReSharper, and voila, an entry point into the business logic that bypasses the database and security checks. Add some ObjectMother magic to construct the fugly data structures, and I've got a working unit-ish test.

Moral of the story? Don't be afraid to change the code first before writing a test (well, maybe afraid but not terrified). Remembering to backup and look for an easier way to write a test was probably a good idea too.

External configuration should be human-readable

2005-06-29T11:52:00.000-05:00

Ranting ahead...

For the love of all that is holy and good in the world make all of your external configuration readable and editable by human beings just in case I have to maintain your system later. Yeah, I know sometimes you have to encrypt configuration for those pesky security auditors, but at least make the unencrypted data readable. First of all, the mechanics of making little configuration tweaks shouldn't be difficult. Second, debugging and diagnosing problems is awfully difficult when you can't even read the configuration.

Xml by itself doesn't bother me, but I'm dealing with files that are partially SOAP serialized. "Simple" whatever protocol my ass. SOAP is not human readable!

Hulk Angry When He No Have ReSharper

2005-06-29T10:30:00.000-05:00

I'm a ReSharper junkie, and I get angry when I don't have it. I've spent the majority of my .Net career working with converted Java developers. To a man, they all complained about the crudeness of Visual Studio.Net compared to IntelliJ or even Eclipse. I wrote them off as whiners because I was used to working in either VB6 or Visual InterDev (I still have a soft spot in my heart for doing what's now called Ajax development in VI) and VS.Net was clearly more polished (except for that little code and continue goodness). Then the ReSharper 1.0 beta builds dropped. My project team of ex-IntelliJ jockeys jumped at it immediately. Even with the sometimes spectacular instability of the ReSharper beta, it made an obvious improvement in coding efficiency. I know this because several times I started to code on the flight home without a working copy of ReSharper and became frustrated quickly when the code navigation and refactoring was missing. ReSharper quickly improved with each passing build, and the phrase "I'm having a ReSharper moment" left the project vocabulary.

Why am I complaining about this today? I'm wrestling with a legacy C# codebase this month and it's not entirely "ReSharper-able." One of Michael Feather's pieces of advice in his book Working Effectively with Legacy Code is to lean on the compiler when making changes to legacy code. Going farther, I really want to be able to use ReSharper's "Find Usages" capabilities to trace code in an attempt to understand interdependencies between classes and trace through code. In order to insert any kind of unit tests I'm having to extract methods, classes, and interfaces. The safest and quickest way to do this by far is to utilize the automated refactorings in ReSharper.

Out of the box the code could not be traversed by the static code analysis of ReSharper. The reasons? The solution files aren't using project references between projects. They are compiling all assemblies to a common hard-coded folder and making all references to this folder, creating a force field impervious to ReSharper code navigation and refactoring. This was largely done to get around problems they had with circular references between class libraries. A lot of classes are loaded and executed by reflection. Part of the problem is that the class library projects are much more fine-grained than they really need to be. I'm guessing that the assemblies were subdivided so granularly to apportion out the work between developers and avoid checkout conflicts with Visual SourceSafe's pessimistic locking, but I don't know that.

So far we're only tackling a small portion of the legacy code at first. The long range steps we're trying first are:

Get to project references immediately. Enable both ReSharper refactoring across projects and the almighty task in NAnt. It also solves some reference conflicts
Collapse the assembly structure to mitigate the circular dependencies and simplify deployment and build
Eliminate as much of the load by reflection instantiations as possible. Minus automated unit testing, strong typing is absolutely essential. Since we don't have an automated NUnit test suite to speak of, we have to have the compiler enforce strong typing as a first line of defense when it's time to modify the code
Get rid of VSS, move to Subversion, and get some automated builds put together for continuous integration
Automate the migration of build products (think shared libraries, frameworks, and staged builds) between our different codebases. This is a soft spot in our infrastructure at the moment that is slowing us down. We're certainly not the first team to struggle with this by any means.

Another thing I highly recommend; my partner is an uber-buildmaster and one of the first adopters of the original CruiseControl. Fellows like him are awfully nice to have around when you're in this sort of situation.

I have played with IntelliJ briefly when I thought I was getting a chance to go onto a J2EE web project. For the agile style of incremental development (red bar, green bar, refactor), IntelliJ is a coder's paradise. I think this largely due to JetBrains dogfooding their own tool using an XP process. I know pockets of MS use agile processes (and I'm dubious about counting MSF Agile in that column), but I don't think the VS team really takes it seriously. VSTS is clearly aimed at the heavyweight Rational Unified Process product.

VS 2005 with VSTS is a definite improvement for agile development, but I think it still falls short of what I've observed in IntelliJ or Eclipse. I'm not entirely sure that some of the enhancements like unit testing integration are actually as good as the existing open source equivalents, and the refactoring in VS 2005 is inferior to ReSharper. I'm honestly more excited for the upcoming ReSharper 2.0 release than I am for the improvements with VSTS.

Taming the Legacy Beast - Dose It With Some NAnt

2005-06-28T14:18:00.000-05:00

Yesterday I griped about legacy code. I carped about VSS, whined about the tight coupling, groused about the arcane configuration setup, and spewed invective about the stupid manual post compilation steps it took just to make the code work. Today it's time to start cleaning things up and applying some ibuprofen to the legacy headache.

Even though the work we're doing is supposed to be throwaway in a couple of months, we decided to "NAnt-ify" the development environment. First we're skimming the VSS project and setting up a new Subversion repository. The next step is to create a bare bones NAnt build to automate the application setup. The last step is a simple CruiseControl.Net configuration to get it into our existing continuous integration server (this is sooooo much easier than it was just a couple of years ago. Hat's off to the CC.NET guys).

If you can't easily make a repeatable build and get the application working on your own box in short order, you're not gonna go anywhere fast. That whole NAnt/CC.NET helps keep the code clean in source control so your coworkers aren't idle while you go hunt down the new class file you forgot to add. And one last time, VSS is a piece of crap. Using the far superior clients for Subversion (or almost anything else) that are actually intelligent enough to determine what files have been changed, added, or deleted makes for a smoother source control experience. And no, we're not going to wait until November for the promise of Hatteras in VSTS when Subversion is here now and free.

You know it's a bad day when...

2005-06-27T15:46:00.000-05:00

You're typing "iisreset" or "net stop iisadmin /y" a lot.

Legacy Code and the Life and Death of an Application

2005-06-27T11:50:00.000-05:00

My colleague and I are making minor extensions to some legacy code (in C#) for the next month or so (and that generally implies that the Shade Tree Developer will feature quite a bit of ranting in the near future). After that my team will be making some large scale architectural changes to an existing product. My job for the rest of the year is going to revolve around legacy code.

We very frequently make customer driven modifications to our application. We also have to keep the product ahead of the competition, and that means being agile (in the literal sense) with our development practices. Our existing codebase has some definite weak spots. The development organization has the beginnings of a long term "Grand Unified Theory" to transmogrify our existing systems and development environment to extend the company's investment in the current products and prevent the downward spiral into "legacy code."

Rewriting the applications is completely out of the question. So first, here's the symptoms we're trying to prevent and eliminate. As we get farther into the "Grand Unified Theory," I'll blog about our strategies for getting out of the legacy code trap.

Life and Death of an Application

In my finite experience and research, applications (or services if you're of the SOA persuasion) reach the end of their useful lifespan when the cost of altering or extending an application is greater or riskier than throwing it away and starting over. I've seen a lot of people make a parallel to the concept of entropy. The natural state of any system will tend towards entropy (in software terms, the big ball of mud). You have to exert energy into an application to reverse or stave off application rot (entropy).

Most applications undergo a twilight period where the previously shiny, well-factored application descends into a morass of spaghetti code that becomes more and more difficult to work with. The applications are still providing business value, but they are starting to be more of an opportunity cost rather than an enabler because of the increasing friction incurred by further modifying the application. This twilight slide into retirement is what most people mean when they say “Legacy Code.” The code works, but no one wants to touch it for fear of bringing the down the whole thing.

What is Legacy Code?

Here's a couple of takes: "Before we get started, do you know what 'Legacy Code' means? It’s code that works" -- via Roy Osherove.

Yeah, but this is true only if you leave it alone.

Michael Feathers described legacy code as "code without tests." What I think he’s driving for is code that can be safely modified and rapidly verified.

I'm going to add a corollary to Mr. Feather's definition; legacy code without tests is code that is difficult to test. Twice in the last year I’ve transitioned from greenfield development projects that were written with TDD to working with brownfield code that had not been written with TDD. In almost startling contrast, the test-first code was vastly easier to extend with new unit tests than the code written test-last. To paraphrase the noted development sage Ferris Bueller, “the legacy code was so tightly coupled that if you put a chunk of coal between the classes you would get a diamond.”

What made the legacy code made writing unit tests difficult?

Overuse of static methods. Static methods result in tight coupling. I ranted about this here. Keeping state in static fields makes it even worse.
Tight coupling to external configuration. It’s often impossible to run any piece of the code without an external configuration file. Dealing with configuration slows down unit testing. I’ll blog more on this soon, specifically ways to avoid the coupling.
Poorly factored code and tight coupling between concerns. Database, business logic, and presentation logic all mixed together.
Blobs

Legacy Code isn’t Necessarily Old

Of course, many applications aren't well-factored to begin with -- instant legacy code. Entropy also creeps in with any gap in communication or participation between the designer’s vision and the development team. I’m dealing with one of those right now. The original vision and design was thoughtful and innovative, but it was thrown over the wall to a team of developers who had not participated in the original vision. It’s particularly bad when the original prototype is coded by a skilled architect and then handed off to the developers with instructions to “just finish it off.” One of the advantages to continuous design is that the rough edges are smoothed down over the course of development. Remove the designer from the coding and those rough edges will only become sharper over time.

Shamefully, I think I’ve been responsible for a bit of this myself. One way or another the technical vision needs to be socialized and shared throughout the development team if you want to create a consistently structured application. Ideally, the development team should be creating the technical vision themselves.

The Development Environment Matters Too

Since much, if not most, of the cost of modifying an existing application is verification and regression testing, the ability to quickly migrate the application to a testing environment is paramount. Repeatable automated testing is such a great long term investment in an application that it’s a shame so few projects invest upfront in test automation. At this point in my career, I would consider development infrastructure like build scripts, database scripts, and continuous integration servers to be a vital part of the application itself. If that infrastructure is lacking or nonexistent, the application is that much harder to modify. I'm working with legacy code right now, and the lack of an automated development build with unit tests is making me feel, well, like I'm naked in a hailstorm.

The worst case I know of was a very large VB6 application that had been accreted around a nice tidy core over the course of several years. Simply setting up a test environment manually could take up to 10 business days, and even then they were never very confidant in the installation. That application was a core piece of the manufacturing system and changed often. Not only was the migration slow and hard, but the haphazard nature of the environmental control allowed several pieces of bad code to get through testing and into production, leading to factory downtime (and take it from me, factory downtime is a bad, bad thing. When the billionaire tycoon founder himself comes looking for someone’s head in IT, run away and get some plausible deniability). The real gallows humor was that the application was customized somewhat for each factory and business line and therefore had to be regression tested for each and every configuration.

"#region/#endregion" Tags Smell Bad

2005-06-27T07:56:00.000-05:00

Ranting ahead...

I'm unilaterally declaring overuse of the "#region / #endregion" blocks in VS.NET a code smell. When I first started playing with VS.NET in 2001 I thought this feature was going to be the best thing since sliced bread. Boy, was I wrong. It seems like every time I expand an innocuous looking region out spills a big bag of entirely unrelated code.

Much like the excessive comments code smell, the existence of a large code region generally implies that the code in the region really should be in a different method or even a different class. The region block is no substitute for good factoring. The region block is often used as just a bit of perfume sprayed over smelly code.

If you want what I would call a bad example, go take a look at the source code of the original application blocks from Microsoft Patterns and Practices. Instead of factoring functionality, they wrote more or less procedural code segregated by region blocks that were nested 4-5 layers deep. From what I understand, the code style was mandated by their internal QA code standards. The result is thoroughly unreadable code. Finding the code that does anything is like playing a game of hide and go seek in a thick forest.

The .Net community's tendency to take anything written or published from MS as the gospel without any kind of critical analysis is a huge pet peeve of mine. The original application blocks were a good example. The code was rotten and they were reportedly buggy in spots -- but I saw nary a word of criticism anywhere.

What an Amazing (Code) Smell You've Discovered

2005-06-23T15:02:00.000-05:00

I'm naming a new code smell today, "Obsessive Tracing." You know exactly what I mean. If you see a long method or class with a *lot* of Debug.WriteLine("1") or Debug.WriteLine("I'm in here now!") methods sprinkled throughout the code, it's a good bet the code smells to high heaven. Those trace statements are in there because the code is prone to breaking and hard to understand.

The point of a code smell is to recognize a problem so you can begin to move in a different direction. I'm not sure what you do with legacy code, but with new code the key in my opinion is a well factored solution for ease of understanding and strong unit testing. Excessive amounts of debugging often means your unit testing isn't granular and comprehensive enough. Excessive debug statements might also mean a developer could benefit from reading up on the capabilities of their debugger.

Geek points for nailing the movie line in the title. No Chris Fields you don't count, that one's too easy for you.

Flimsy evidence that agile delivers the goods...

2005-06-23T07:51:00.000-05:00

Does Agile development actually work in real life? Here's anecdotal proof in the affirmative (since the XP Refactored book sold a bunch of copies mostly based on negative anecdotes I figure I can get away with this too).

Production Moves in Ceremonial Waterfall

Production moves in big IT waterfall land are living nightmares of pure desperation and panic just waiting for the other shoe to drop. We usually had most of the team work an entire weekend in shifts just to be on hand as code and data was manually migrated. It's not unusual to see the cots come out in project warrooms. I don't think that's what the Godfather gang have in mind when they "take it to the mattresses," and the mafia seemed to have more fun with it than we did. As I've mentioned before, teams often made last minute coding changes right before go live because integration problems weren't discovered until making the migration and everybody was afraid to miss the installation window.

Agile Deployment

Shifting attention back to the shiny agile world of Test Driven Development and Continuous Integration, we made a final push into a production staging environment with our new product yesterday (with no known defects I might add). After yesterday's move, I left work on time and took my toddler to the YMCA pool with nary a cell phone at hand because I was perfectly confident everything was fine (I'll probably pay for that this morning).

On a previous project last year for a previous employer we made our second release one Tuesday morning. I was quite happily working on a user story in the mainline code branch for the following release when the project manager came up to tell us the new release had been put into production without a hitch. The developers had completely forgotten about the production move!

Where I Think Agile Development Helps

So why do I think the production moves went so much better with agile development than my earlier projects in waterfall world (to be fair, the earlier projects were COM based with a fair amount of middleware and the later projects were .NET systems)? The waterfall shop had a admirably organized (but radically inconsistent) ceremonial change management process, but pitiful build management. I think most of the reasons I list below are applicable for any disciplined iterative and incremental process (I'm thinking RUP here), but agile processes put more emphasis on early and frequent feedback over almost anything else.

Earlier, more frequent, and repeatable testing. Test Driven Development is really a design technique as much as anything, but it certainly creates a large body of automated unit tests that provide a safety net to keep new bugs from emerging. Add any kind of automated acceptance or regression test suite and that safety net gets much tighter.
Continuous Integration. CI can be thought of as a lot of trial runs at automated code installation. Much the same way that TDD forces you to write code that is testable, maintaining an automated CI build forces you (if your thinking cap is on straight) to think hard about making your application or service easier to deploy and version. Some people will excoriate me for this statement, but I think a NAnt build is the best documentation for system installation because it unambiguously specifies what has to happen for the application to function (and it tends to stay synchronized with reality easier than a document). I'd still supplement that with a Wiki page though.
Defects can be fixed faster and stay fixed. CI allows a team to push code fixes to testing much faster without sacrificing environmental control. In ceremonial waterfall land, code pushes required filling out forms and convincing an external group to make my code move first, often up to a six hour process if other projects got there first. By automating builds under the control of the project team, I think the control over the environment is much better even without a controlling environment authority outside the development team, and certainly much faster. When you do write an automated test for a defect as you fix it, the defect stays fixed. Lost time due to regression failures can be mitigated by automated testing.
Project teams are, or should be, more self-reliant and self-organizing. What I mean is that a project team should be able to carry out its own configuration management without waiting for a sluggish external group. In agile projects and organizations I've consistently had fewer dependencies on external IT service teams. I think that goes a long way towards reducing project friction. It does require carrying a broader skillset on a project team and people willing to step outside their normal skill silo. I think that might be one of the largest barriers to agile adoption in big shops. Flame away on point number 4, I probably deserve it.

Our organization has started down a path of using the WiX tool to create installers as part of our NAnt/CruiseControl.Net build. Using this, our testers will now be able to work off the automated installation in testing environments at all times, in effect testing the application and installation at the same time. I know several other organizations in the blogosphere are trying this, too. I'm only peripherally involved, but I'm excited for the potential. More updates when I know more.

I detest Visual SourceSafe

2005-06-22T13:40:00.000-05:00

Ranting ahead, obviously...

All our new projects are in bright, shiny Subversion repositories and humming along just fine. I've got about a month of detour through existing products in Visual SourceSafe and I'm already aggravated on the first day. Don't get me wrong, every source control system has some sort of problem, but VSS is the absolute worst.

The client is a piece of crap; it falls over and corrupts in a swift gale, it's slow, and hangs during CC.NET builds almost as often as not. Branching and merging with VSS is a joke. In a world where Subversion and CVS are free and reliable with multiple clients and integration into IDE's, and SourceGear Vault is inexpensive, why is VSS still so popular in .Net shops?

I've played quite a bit with the vss2svn perl script from Tigris.org. I have no idea if it works or not because VSS keeps crapping out on corrupted history trails. VSS won't even behave long enough to get rid of it.

One thing I'd really like to see from the .Net community as a whole is to stop taking everything from Microsoft without question. There is a whole world of development tools and knowledge out there that doesn't originate in Redmond.

How's this for scary though, in a former job we used VSS under the dark of night on the side because the officially mandated source control system (CC/Harvest ) was even worse.

Thinking about Developer and Tester Interaction

2005-06-21T09:49:00.000-05:00

The following is essentially a lecture and reminder to myself and my team, I apologize in advance for being preachy.

We're wrapping up a release candidate build this morning and quite naturally the subject of testing and bug fixing is on my mind. Because of resource constraints (i.e. not enough testers) we were not able to do much testing (acceptance testing by real testers, not just TDD) within iterations. Doing the old-fashioned (and stupid) big bang, waterfall testing cycle at the end under schedule duress is always tricksome.

It is absolutely imperative that the developer/tester interaction be as smooth as possible. Everybody needs to be focused on a common goal of finding and removing defects from the code, and that means paying attention to what the other half of the equation needs. Verbal abuse or general incivility from either side needs to be smacked down.

If you're going to do any kind of Agile development, one of the first and hardest lessons to learn is that the role boundaries must be blurred, or at least have much more transparency between project roles. You can't bury your head in coding and designing, you have to actively cooperate with testing on test automation and code migration between development and testing environments.

What Developers Owe Testers

Don't waste the tester's time with buggy code. At a minimum, unit testing and either pairing rotation or code review should stop the majority of the problems from reaching the testers. If you can get acceptance tests to the developers before coding is "complete," all the better still.
When you fix a bug, try hard to write an automated test that validates the exact conditions of the defect. At a bare minimum, exercise the code end-to-end to ensure the bug is really fixed. Don't assume the defect is really fixed because you dealt with the root cause (I'm currently wearing some egg on my face from this one).
If anything is blocking the testers, stop what you're doing and get them working again. The application or even a user story cannot be considered complete until it is tested. If the testers are sitting on their hands because a testing environment is invalid or the application is down, your project is down. Again, a reminder to my team and I because we dropped the ball on this one day a week or so back.

What Testers Owe Developers

The singular thing we developers really need from the testers is just communication. When a defect is found, we have to understand a couple of things.

A description of exactly what the error is
Exactly how to reproduce the error
The desired outcome of an operation in clear, unambiguous language
Contextual information if it exists. Stack traces, audit trails, input data
Severity of the defect. Is the defect stopping the tester from testing any farther?

The worst thing a tester can do is just say "It doesn't work" without any context or adequate description of the desired outcome (or my colleague's ultimate pet peeve, "it says NullReferenceException"). I had an incident a couple years ago with an analyst slash tester. She had given us the original requirements for a data exchange between two systems that wasn't much more than a big SQL statement in a stored procedure (mildly modified from the previous system we were replacing). When she was testing the exchange, she kept coming back to us and telling us the results were wrong. Her SQL script she was using to test the exchange was giving different results. I think at some point I said something like "!~@#* it, let's just use your SQL" so we could move on.

My favorite tester used to show me all defects at the tester workstation and walk me through the problems before his team would log a defect. That helped dramatically. It obviously helps if the testing team is colocated with the developers.

Testers are Pigs

Made you look, didn't I? All I mean is that the testers should be an active participant in the standup meetings and the project as a whole. One of the agile practices I like is to have a bug report discussed every single morning in the standup, even if it's just to say there are no bugs. It's a great way to ensure the developers are getting the testers what they need and vice versa. Bug tracking tools are fine and dandy (except ours is terrible), but nothing beats face to face communication.

Here's the original story about the derivation of Pigs and Chickens if you've never heard the phrase.