Structuring a Proof of Concept

A common practise in IT, as run up to a sale or a project is to proof that the intention of the undertaking can be fulfilled.

The challenge

A PoC needs to strike a challenge between effort and coverage. A final proof of a project is its completion, so the temptation lures to try to proof everything. On the flip side: if they core functionalities aren't covered the proof has little value.

The second challenge is to define concise successs criteria. Quite often, especially for standard product PoC, it is left to 'how users like it' - which isn't a really qantifiable result.

Use cases

A workable approach is to define use cases, that cover a typical scenario, like 'Sale of an ice cream'. This scenario needs to be broken down into business steps until a step can be looked at: 'did work / did not work'.
The breakdown needs to be business level, business language. So 'Can click on customer info' should rather read 'Customer info is retrievable'.

Use cases and steps are hierarchical, typically 2-3 levels are sufficient for most PoC. Deeper levels are a smell that you are looking at a pilot or full fledged project, not a PoC.

So, in a nutshell: A PoC line item needs to have a binary answer. If a binary answer isn't possible break the line item into smaller units. Stick to the domain specific language (usually: the business steps)

Measurements

When a use case line item has a binary outcome (works / doesn't work), the simplest measure is to check if everything worked to declare the PoC a success. Usually doesn't help.

The next level is to define a pass percentage. Like 70% of 200 line items must pass. Again a simple solution. Challenge there: nice to have and essential features have equal weight. You could end with an outcome that has all nice-to-have features, but might miss essentials.

So the next level is to define weights for each items, including a showstopper flag for must-have features. Weighting discussions are popular battle grounds for feuding fractions, since the weight determines outcomes, especially for concurrent PoC execution.

Another weakness of this approach: works/doesn't work as binary value doesn't cover: 'Does it work well?'. Like 'Is a pair of sneakers suitable to get from Boston to New York?' The binary answer: Yes you can walk, but the real answer: use a car, train, bus or plane.

Balanced Scorecard to the rescue

Looking at the definition of Usability, one can find 3 criteria:

• Does it work?
• Is it efficient?
• Is the user pleased?

I would treat the first column as a binary value and the later two as scales from 1-5. This allows to generate a balanced score card that reflects important aspects of a proof. Depending on the nature of the system, you could add additional columns like 'failure resistance, error recovery, risk'.

While it doesn't relieve you from the weight bickering, it provides a clearer picture of actual outcomes.

As usual: YMMV