Weighting Metrics

I'm reading Richard Feynman's book What Do You Care What Other People Think? [1] - a fascinating account of the things that Feynman did and believed in: the power of science and the experiment (there's even a xkcd cartoon about that).

Feynman worked on the Challenger Commission which investigated why the shuttle Challenger exploded and concluded with the discovery of the O-ring failure in one of the solid rocket boosters. One of the most memorable incidents was Feynman's live O-ring in ice water experiment.

However, after dealing with metrics on various issues recently, a paragraph in the book where Feynman discovers the results of a go or no-go decision on the state of the O-rings under cold conditions. There are four named experts and four answers: 2 x no, 1 x yes, 1 x don't know - which effectively splits the vote 50-50 (for some reason don't know = yes).

However Feynman points out that the foremost experts on the properties of the O-rings both stated no and one of the four experts was not present at the original meeting. Taking this into account we get the following:   2w x no, 1v x yes, 1u x don't know, where w > v > u. Simple mathematics returns not a 50-50 split but a split where the no vote would overwhelm (even by a microscopic margin) the yes/don't know combined vote.

Suffice to say here that weighting of the inputs into the calculation here was critical to getting the righ results. This is not to say that finding the weights is not hard, but as we see in the case above even simple ordering would have sufficed.

The metrics are simple, the relevance and weighting unfortunately are forgotten and it is these that really tell you what the metrics mean and how to analyse them.

References

[1] Feynman R. (1988) What Do You Care What Other People Think? Penguin Books. 978-0-141-03088-3