Something of a technical question but very important never the less because it goes to the heart of the bigger question on when and how statistical (and other) sophisticated risk techniques could introduce risk into your business. Used inappropriately, they could generate spurious answers which are treated as Gospel. Andy Garlick offers two clear examples of when Monte Carlo should not be used in his book Estimating Risk.
Despite its very general philosophy, there are some cases where Monte Carlo does not work. This can be the case when the sampling process does not adequately look into the feature which the output requires. Two examples come to mind:
1. A contract is priced at P80. A variation order is then issued to allow an additional task to be done. It is desired to keep the price at P80. What should the value of the variation order be? The way to do this is to run Monte Carlo both with and without the extra task, and look at the difference in the P80s. Because of the sampling errors in the P80 estimates, however, it is difficult to calculate this difference with any accuracy. Of course, you can explain to the client that this does not really matter if you embrace the Monte Carlo approach – it is just a matter of living with probabilistic prices. On the other hand, you may prefer to be able to do an accurate estimate.
2. You are doing a system reliability analysis where a component has an unreliability of 10−4. You are using 1000 samples but realise the event will not happen. So you do 100 000 simulations and the event happens about 10 times. But another part of the system – considered to be independent – has an unreliability of 10–2. But, you still don’t see it. So you do 10 000 000 samples – which takes a week and you still do not have a very accurate sample. When you do some more samples, you run out of memory. Monte Carlo is simply not suitable for small probabilities.
Source Estimating Risk by Andy Garlick, Gower Publishing, Farnham, 2007
Have you got advice on the best use (or misuse) of tools and techniques in projects? Share your thoughts and read Andy’s book in full onwww.gpmfirst.com