Saturday, January 19, 2008

What would you do when your program does not do what your documentation says it should

You are a development manager.
You develop enterprise software mainly used on the web.
Your software gets extremely customized for customers by professional services - either from a division of your company, by partners, or consulting companies.
Your company hosts a few customers while the majority deploy your product in their own environments.
A large customer reports an issue which gets investigated by customer support. Support cannot determine if it's a defect at all. Defect ? Of course, it goes to development.

Development is tasked with investigating whether the API in question is defective. Development determines that the code and end user docs don't match, but it`s not known which is correct. So it looks like if you follow the docs it is a defect, if you follow the code it is working correctly.

What would you do ?

1) follow what is written - declare it a defect and fix it
2) follow the code - declare the documentation out of date, fix the docs, and suck up to the customer that reported it

We ended up choosing (1). We changed the code to match the docs because we didn't know why they differed. Our automated dev tests passed. But we failed QA because they had some obscure tests that regressed, ultimately saving our behinds. Reason ? We would have helped one customer and injured several. We certainly didn't have enough characterization tests to prevent changes in the system that were "legacy" and quite unknown.

If your software gets installed at customer sites, you most likely need to create end user documentation. A typical iteration will see a spec created, followed by the code and tests, and lastly the docs. Since the functionality of programs will change over time, and because the docs have the highest propensity of not being kept in sync with the requirements and with the program, I would argue to trust your docs infrequently compared to your working code.

No comments: