What is the p-value?

The p-value indicates the probability that the observed result or an even more extreme result will occur if the null hypothesis is true.

The p-value is used to decide whether the null hypothesis is rejected or not rejected (not rejected). If the p-value is smaller than the defined significance level (often 5%), the null hypothesis is rejected, otherwise not.

You want to make a statement about the population and have set up a hypothesis for this. Since it is usually not possible to survey the entire population, you survey a sample. Now this sample, due to chance, will most likely deviate from the population.

If the null hypothesis applies in your population, e.g. the salary of men and women does not differ in Germany, then there will certainly still be a difference in the sample, e.g. a difference of 300 euros per month. Now the p-value tells you how likely it is that a difference of 300 euros or more will occur by chance in the sample if there is no difference in the population.

If the result is a very small probability, you can of course ask yourself whether the assumption about the population is true at all.

If the p-value is 3%, for example, then it is only 3% likely that a sample is drawn in which the salaries of men and women differ by more than 300 euros.

When is the p-value used?

The p-value is used to either reject or retain (not reject) the null hypothesis in a hypothesis test. If the calculated p-value is smaller than the significance level, which in most cases is 5%, then the null hypothesis is rejected, otherwise it is not rejected.

Example:

• The null hypothesis is that there is no difference between the salaries of men and women.
• Now a sample is taken with the salary of men and women. These are our observed results.
• We assume that the null hypothesis is true, that is, that there is no difference between the salaries of men and women.
• In the observed result (sample) we now find out that men earn 150 € more per month than women.
• The p-value now indicates how likely it is to draw a sample in which the salary of men and women differ by 150€ or more, even though there is no difference in the population.
• If the p-value is e.g. 0.04, it is only 4% likely to draw a sample of 150€ or more extreme, if there is no difference in salary in the population.

Let's say, in the upper case a p-value of 0.04 or 4% comes out, what does this p-value mean now? It means that if there is no difference in salary in the population, it is only 4% likely to draw a sample that is 150€ or more extreme.

The probability of 4% is of course very low, so that one can ask oneself whether it is at all true that men and women in the population earn the same or whether this hypothesis should rather be rejected.

The question from when the null hypothesis is discarded answers the significance level.

Significance level

The significance level is determined before the test. If the calculated p-value is below this value, the null hypothesis is rejected, otherwise it is not rejected. As a rule, a significance level of 5 % is chosen.

• alpha < 0.01 : very significant result.
• alpha < 0.05 : significant result.
• alpha > 0.05 : not significant result.

The significance level thus indicates the probability of a 1st type error. What does this mean? If there is a p-value of 5% and the null hypothesis is rejected, the probability that the null hypothesis is valid is 5%, i.e. there is a 5% probability of making a mistake. If the critical value is reduced to 1%, the probability of error is accordingly only 1%, but it is also more difficult to confirm the alternative hypothesis.

One tailed p-value

Let's say you are examining the reaction time of two groups. Then it is often not of interest whether there is a difference between the two groups, but whether one group has a larger or smaller value than the other. In this case, you would have a directional hypothesis and then calculate what is called a one-sided p-value.

The one-sided p-value is then obtained by dividing the two-sided p-value by 2. Here, of course, care must be taken whether the difference or effect under consideration is at all in the direction of the alternative hypothesis.

Example

Your alternative hypothesis is that group A has greater reaction time values than group B. When analyzing your data you get a two-sided p-value of 0.04.

Now you have to check whether group A really has larger values in your data. If this is the case, the two-sided p-value is divided by two, so you get 0.02.

If this is not the case and the effect or difference goes exactly in the other direction than formulated in the alternative hypothesis, your p-value is 1-0.02, i.e. 0.98.

Don't worry, if you use DATAtab, you can specify what kind of hypothesis you have, and DATAtab will help you evaluate it.

Calculate p-value

For the calculation of the p-value a suitable hypothesis test must first be found. If the suitable hypothesis test is found you can calculate the p-value in the statistic calculator on DATAtab. The best known hypothesis tests are:

• t-Test
• Correlation analysis
• Chi-Quadrat-Test
• Regression analysis

For the calculation of the p-value a distribution function is needed which describes the realizations or drawings of the sample. If this distribution function is known, it can be determined how probable it is that a drawn sample is less than or equal to a considered value. Classical representatives of these distributions are the t-distribution and the Chi-square distribution.

t-distribution

Chi-square distribution

Statistical tests and the p-value

In order to reject or maintain a hypothesis one needs the p-value. The procedure how the p-value is used for statistical tests, is now the following:

• Definition of the critical p-value or the significance level e.g. 5 %
• Definition of a statistical test procedure e.g. t-tests or Correlation analysis
• Calculation of the test statistics from the sample e.g. the t value in the t-test
• Determination of the p-value for the test statistics e.g. p-value for given t in t-test
• Check whether the p-value is above or below the specified critical p-value e.g. p-value 1% leaks below the critical value of 5%