Does asking the team actually work?

The status report is the weak link

In a survey of software project managers, status reports were biased around 60% of the time, and the bias was twice as likely to be optimistic as pessimistic. That isn't dishonesty. It's a set of documented, well-understood biases acting on one person who is accountable for the date. The full list, with the research behind each, is in why status reports don't reflect what's actually happening.

Why independent judgements beat a single one

A group's combined judgement beats an individual's under specific conditions: the people have different information, they judge independently, and their answers are combined without an editor. Independence is the one that has to be designed in.

At Google, researchers found strong correlations in trading among employees who sat within a few feet of one another — proximity alone was enough to make judgements move together. It's why planning poker uses a blind reveal, and why a confidence vote is anonymous. More on the conditions in can the wisdom of crowds be applied to project status?

Companies have tested it

• Hewlett-Packard ran internal markets to forecast figures in its official sales forecast, with between 20 and 30 participants who didn't know the official number. The market outperformed the official forecast in six of eight cases. Chen & Plott (2002)
• Siemens ran an internal market that predicted a software project would definitely fail to deliver on time, even as traditional planning tools said the deadline could be met. Wolfers & Zitzewitz
• Google conducted what researchers described as the largest corporate prediction-market experiment they knew of, including markets on project completion. Cowgill, Wolfers & Zitzewitz

The groups were small. At HP and Siemens, the traders numbered only between 20 and 60 employees. You don't need a crowd the size of a company — you need the people close enough to the work to know.

Genchi keeps the principle and drops the machinery

Prediction markets are hard to run on every project: they need a currency, trading, incentives and enough participants to keep prices moving. A confidence vote keeps what made them work — the people closest to the work, answering independently and anonymously, combined without an editor and tracked over time — and replaces the trading floor with one click in Slack. The result is a project delivery prediction for every project, not just the few worth building a market for.

Where it's weaker

• A whole team confidently wrong. If nobody on the project can see the problem, no amount of asking will surface it.
• Optimism about your own work. Google's markets showed an optimistic bias, strongest on project completion — people lean hopeful about work they control. That's why the direction of the trend and the spread of individual votes matter more than the absolute number.
• Organisations that punish bad news. If a worried vote is held against someone, the votes go green and stay there. See how do you get honest status in a political organisation?
• Sudden late surprises. A key person leaving or a vendor failing the week before launch can't be predicted by anyone.

References

• Snow, A.P., Keil, M. & Wallace, L. (2007). The effects of optimistic and pessimistic biasing on software project status reporting. Information & Management, 44(2), 130–141.
• Chen, K.-Y. & Plott, C.R. (2002). Information aggregation mechanisms: concept, design and implementation for a sales forecasting problem. California Institute of Technology.
• Wolfers, J. & Zitzewitz, E. Prediction markets in theory and practice. National Bureau of Economic Research.
• Cowgill, B., Wolfers, J. & Zitzewitz, E. Using prediction markets to track information flows: evidence from Google.
• Cowgill, B. & Zitzewitz, E. (2015). Corporate prediction markets: evidence from Google, Ford, and Firm X. The Review of Economic Studies, 82(4), 1309–1341.