Level of Significance

The level of significance of a statistical test of hypotheses refers to the upper bound (expressed as a percentage) placed on the probability of Type I error for the test.

Notes:

• When you specify the level of significance for a statistical test, you are effectively stating your maximum tolerance for the chance of Type I error occurring when you use the statistical test. For example, when choosing a 5% level of significance, you have the assurance that
  
  P(Type I error) <= 0.05
  if you use a statistical test with this level of significance.
• The definition given above is more general than the one usually encountered in an introductory textbook on Statistics. In some cases, the level of significance is simply defined as the probability of Type I error (see, for example, Johnson and Bhattacharyya, 1996, p. 332). This restricted definition is applicable when the null hypothesis is simple but not when it is composite. In the latter case, P(Type I error) is not uniquely determined since it depends on what makes the null hypothesis true. For example, when testing
  
  H₀: p <= 0.5    versus    H₁: p > 0.5,
  P(Type I error) depends on the value of p which makes the above H₀ true (any value of the population proportion not larger than 0.5 makes this H₀ true). Thus, in general, it makes sense to specify a reasonably small upper bound for P(Type I error) and this is precisely what is achieved when we specify the level of significance for a statistical test.
• Usually, the level of significance of a test is set at 1%, 5% or 10% with 5% being the usual default value. Part of the reason for choosing one of these conventional levels (apart from their relatively small size) is because of constraints related to the construction of statistical tables. Nowadays with widespread availability of suitable computer software, you are not so restricted and can choose any sufficiently small number for the level of significance for your test. Your choice is usually dictated by the seriousness of Type I error for your problem and the extent of control you desire for the chance of occurrence of this error. A word of caution is in order:

  P(Type II error) increases as P(Type I error) decreases, assuming everthing else (e.g., sample size) remains the same.

  What this means is that when you decrease the level of significance, you shrink the critical region of your test and hence reduce the chance of rejecting the null hypothesis. Consequently, the probability of a Type I error drops but the chance of a Type II error increases.
• Finally, note that the alpha-level of a test refers to its level of significance expressed as a probability (e.g., the alpha-level of a 5% level of significance test is 0.05).