• COTS or Not?

COTS or Not

Introduction

Perhaps the most widely used acronym in government IT projects is COTS – Commercial Off-The-Shelf. Prevailing wisdom amongst many major government IT procurers and their suppliers is that ‘COTS is good; the alternative – bespoke software development - is intrinsically bad’. And although in many circumstances this principle is sound, in recent years the author has seen numerous IT project failures that are attributable to it being wrongly applied.

This paper is intended to be read in the context of procurement of secure government IT systems of significant complexity, where the use of COTS products designed principally for commercial markets always needs careful analysis. The differences between a project to deliver a typical commercial IT system and one to deliver a secure government IT system are worthy of separate discussion; I won’t dwell on them here. Suffice to say that, although some of what is written here might be applied to commercial sectors, such read-across should be done with care.

Background

A few decades ago, large-scale bespoke software development projects were much more common in the government sector than they are today. The justification for this bespoke development was frequently that commercial products did not offer sufficient security. Some people cite failed government projects that tried to develop bespoke operating systems or secure network protocols as evidence of an industry that was then immature and which is now more pragmatic and better managed, perhaps forgetting some very successful spin-offs from secure government systems that the IT industry now takes for granted.

In recent years, I have commonly seen IT projects in the government sector get bogged down in ‘customising’ their COTS software (of course this happens in commercial IT projects too, but I’ll restrict this discussion to the government environment).

At this point, I need to clarify what I mean by COTS and bespoke. When used in a secure IT system, almost all COTS software is going to need some customisation. But what seems like quite a clear distinction can soon become quite fuzzy. An EDRM (Electronic Document and Records Management) package is a piece of COTS software, but its customisation will frequently involve software development rather than just configuration. A bespoke system will almost certainly use a COTS framework such as J2EE, or .NET, a COTS database, a COTS application server and so on. So it is really just down to a difference of approach: A system integrator can choose an integrated COTS package-based solution, and write bits of middleware when they really have to, or can choose to develop a bespoke system employing COTS components where it is expedient to do so.

This paper is all about IT project failures involving COTS package-based solutions that tried to meet requirements that were either unrealistic or which really required a more bespoke approach. Characteristics of these failed projects are:

• Implementation takes far more time and effort than the solution architects anticipated, sometimes leading to the project to be abandoned before completion.
  
• If the team does manage to complete the project, the resulting solution is costly, if not impossible, to maintain long term, with huge problems arising when moving to new versions of COTS products, necessitating major rework or acceptance that the system will have to remain using old COTS software well beyond its support period.
  
• Performance may be far below what users can reasonably expect and below what the COTS vendor in good faith led the project to believe was achievable.
  
• Requirements being abandoned at a late stage when it is discovered that they are not attainable. This then results in less business benefit, additional unplanned business change, dissatisfied users and frequently greater security risk.

So why is this failure prevalent?

Let’s take for granted that these IT projects went through a supplier selection process that ensured that the COTS products being adopted were well-engineered and mature, with many satisfied customers. In the failed projects I have in mind, this was the case.

Let’s also take for granted that these IT projects had a system integrator that genuinely wanted to deliver a good solution for their customer and provided a team that were reasonably knowledgeable and competent in the products and technologies they were using. Again, this was the case in the projects I have in mind, this was for the most part the case.

I believe that the reason for this failure can be distilled down to a simple fact: These projects believed that by adopting a COTS package-based solution they were managing their implementation risk effectively – they didn’t do nearly enough analysis of the real technical risks they faced, and didn’t put adequate mitigation in place. They didn’t sufficiently come to terms with the fact that, to make a COTS package-based solution work effectively, you must have a ‘COTS requirement’. What I mean by this is that the further the customer’s set of requirements diverges from what the COTS vendor might deem to be typical, the greater the implementation risk. Every additional piece of customisation, every piece of code put around the COTS software to cater for some special requirement, increases the project’s implementation risk. Something these failed projects had in common is that, if their solution architectures had been shown to the lead engineers in the COTS product vendors, the response would have been somewhere between surprise and horror at the way in which their software was being used completely outside the bounds of what was intended.

If a COTS package doesn’t meet all the business needs, then getting the COTS vendor to change their product, rather than relying on the system integrator to do excessive, clumsy customisation is a good way to reduce risk. However, the reality is that most IT projects simply won’t have sufficient leverage with product vendors to do this, or don’t maximise the leverage they do have.

To put it another way, it’s innovation that risky – whether this innovation consists of large-scale bespoke development, or whether it consists of customising COTS products in ways never intended, without effective risk mitigation it still has the same outcome in terms of reducing the likelihood of project success.

In some ways, a project imposing the restriction of maximising use of COTS makes matters much worse. Let’s consider the real-world example of a project trying to add a stronger access control model into COTS software. A bespoke solution would have free reign to implement security wherever the architects deem most appropriate: at the database layer, in a module called by business logic – or whatever best suits the solution. In contrast, a system integrator customising a COTS product will have very limited choices offered to them by the COTS, often implementing security at a layer in the software stack they know is just plain architecturally wrong because they have no choice.

Avoiding these pitfalls

Where an IT system procurer and its system integrator have an effective working relationship, these pitfalls should be avoidable early on in a project. The system integrator should be strong enough to point out to their client where the requirements take the solution beyond reasonable customisation expected of a mature COTS product and into the territory of risky misuse. However, this much-needed frank discussion sometimes fails to occur due to competitive tendering situations, optimism at early project stages and poor definition of required COTS customisation.

Ultimately, the answer is for procurers of complex IT systems to go to the trouble to understand the solution implications of their requirements. My experience is that most suppliers will focus on trying to deliver what the customer asks for, even if they quietly have reservations as to whether the requirements reflect the real business need or warrant the associated cost and risk. This means that procurers of complex IT systems need to have a strong client-side team. Unless the client has staff involved in producing system requirements that understand the implications on suppliers’ ability to deliver, there’s always the prospect that projects unconsciously take on delivery risks that aren’t justified by associated benefit.

‘COTS requirements’ or ‘bespoke requirements’

If your IT project has requirements that are a good match to available, mature COTS package solutions, then this is almost certainly the lowest cost, lowest risk implementation option. After all, if the product vendors are doing their job properly, most organisations should have their needs fairly well met by these COTS products.

However, if your project has to meet unusual or especially demanding requirements, perhaps because your security requirements are more demanding than most other organisations, maybe because of the sheer scale of the data you need to handle, maybe because there are particular functional requirements that are unique to your area of business, then you need to look hard at whether a COTS package-based solution is likely to satisfy these needs effectively.

Don’t fall into the trap of setting in stone that the project must use a perceived market-leading COTS package, then forcing the package to meet all your requirements, even when it’s ill-suited to the task. Make sure you do rigorous requirements engineering: get a sense of how valuable – how beneficial to the business – how tradable – each requirement is. And understand, in detailed informal collaboration with potential suppliers, what the impact is on the solution of meeting these requirements.

I’ve frequently seen an approach adopted that amounts to documenting requirements without any consideration of whether they are hard or easy to implement, then formally issuing the requirements to potential suppliers without any real discussion of what might be tradable (and marking a few peripheral requirements as ‘desirable’ or ‘highly desirable’ instead of ‘mandatory’ is no substitute for requirement trading). The idea behind this approach seems to be one of “Let’s see what the suppliers come up with. If they can’t meet the requirements, or we don’t like the solution, then maybe we’ll make some concessions”. What this approach fails to consider is something called ‘solution elasticity’. If you issue requirements document A to suppliers, leave them to develop their solution, then change the requirements to B, then some time later to C, you’ll almost certainly get a different (and worse) solution than if you asked for a solution to C in the first place.

Work closely with potential suppliers rather than keeping them at arms length until you have selected a supplier and hence a solution – try to understand the areas of requirement that they find challenging. Don’t be afraid to go back to business sponsors and tell them that some requirements are too risky to deliver. And when a set of requirements suggests a highly bespoke solution, acknowledge it and what goes with it – sometimes COTS isn’t the right answer.