Yesterday I wrote about common cause and special cause variation. Certain amount of randomness is always present in any system, and that causes fluctuations in the outcome. This is common cause variation. On the other hand, something unusual might happen from time to time which also affects the outcome. These variations in outcome are called special cause variation.
Statistical theory says that one should not make immediate changes due to common cause variation. This randomness is in the system and is independent of specific actions that might have happened at the time. When you do make changes due to common cause variation, then it is called tampering with the system.
To continue from yesterday's example: If I deliver a training and get a lower rating, then a natural behaviour is to look at the training and see if I can make some changes for next time. But if variation is due to common cause, then it is just a natural variation. I could do everything exactly the same next time and end up with a higher rating.
In this situation, it is a mistake to mess around with the training class. If I change the content, and end up with a higher rating the next time, I might think the new content caused the higher score, when it might have just been the natural variation that caused it. The training after that might be another lower score, and I might then change something else.
If you think about it, these actions are just injecting further randomness into the system, destabilising any signals that I might otherwise observe.