We now reconsider the randomization proposed for the experiment on computing machines, in order to discuss a criticism that presumably has occurred to the reader. When this experiment is being planned, we possess the knowledge that there is some advantage to the machine on which the second calculation of any sum of squares is made. In the light of this knowledge, it seems wise to make sure that each machine receives the advantage in five of the replications, rather than to leave this to be decided by the tossing of coins. In fact, if the decision is made by tossing coins, the chances are only about 1 in 4 that each machine will be used first in five replications; they are also about 1 in 4 that one machine will be used first in 7 replications. The criticism is that the randomization is unlikely to give as accurate a comparison as the proposed alternative. The criticism is sound, and the experiment was actually planned so that each machine would be used first in half the replications.
Randomization was also applied in the design actually used, though it is a different type from that originally proposed, being subject to the restriction that each machine must appear first exactly five times. The process of randomizing therefore consisted in choosing five numbers at random from the numbers 1 to 10; these turned out to be 1, 3, 6, 8, and 9. Accordingly, machine A was used first in replications 1, 3, 6, 8, and 9. This is one of the cases where it is difficult to see why failure to randomize could have led to any serious danger of bias. It probably would have been quite satisfactory to use machine A first in the first 5 replicates, and machine B first in the remainder. Nevertheless, it might happen that the advantage in the second calculation of a sum of squares would diminish in later replications; and, to save the trouble of trying to guess whether biases of this type are likely, the randomization was used as a precaution.
Adapted from:
Cochran, William G., and Cox, Gertrude M., "Experimental Designs", John Wiley & Sons Inc, 1957, pp.6-8.