Weighted Cohens Kappa

The weighted Cohens Kappa is a measure of the agreement between two ordinally scaled samples and it is used whenever you want to know if the measurement of two people agree. The two people measuring something are called raters.

In the case of "normal" Cohen's Kappa, the variable to be measured by the two raters is a nominal variable. With a nominal variable, the characteristics can be distinguished, but there is no ranking between the characteristics.

The Cohens Kappa takes into account whether or not the two raters measured the same thing, but it does not take into account the degree of disagreement. What if you don't have a nominal variable, but you have an ordinal variable?

If you have an ordinal variable, that is, a variable in which the characteristics can be ordered, then of course you want to take that order into account.

Let's say your expressions are dissatisfied, neutral, and satisfied. There is a smaller difference between dissatisfied and neutral than between dissatisfied and satisfied. If you want to take the size of the difference into account, you have to use the weighted Cohen's Kappa.

Therefore, if you have a nominal variable, you use the Cohens Kappa. If you have an ordinal variable, you use the weighted Cohen's kappa.

Reliability and validity

It is important to note that with the weighted Cohens Kappa you can only make a statement about how reliably both raters measure the same thing. But you cannot make a statement about whether what the two raters measure is the right thing!

So if both raters pretty much always measure the same thing, you would have a very high weighted Cohen's Kappa. Whether this measured value fits with the reality, thus, the correct is measured, the weighted Cohens Kappa does not tell you! In the first case one speaks of the reliability. In the second case, one speaks of validity.

Calculating the weighted Cohen's Kappa

How is the weighted Cohen's Kappa calculated? Let's say two physicians have evaluated how satisfied they are with the therapeutic success of patients. The physicians can answer with dissatisfied, neutral and satisfied.

Now you want to know how big the agreement between the two doctors is. Since we have here an ordinal variable with the rank order dissatisfied, neutral and satisfied, we determine the agreement with the weighted Cohens Kappa.

In the first step we create a table with the frequencies of the respective answers. In doing so, we plot one rater on each axis. Here we have our two raters, each of whom rated whether they were dissatisfied, neutral, or satisfied with the success of one person.

Let's say a total of 75 patients were evaluated. Now let's count how often each combination occurs. Let's say 17 times both raters answered dissatisfied, 8 times rater 1 answered dissatisfied and rater 2 answered neutral, 4 times rater 1 answered dissatisfied and rater 2 answered satisfied and so on and so forth. For the ratings that lie on the diagonal, both raters agree.

The weighted Cohen's Kappa can be calculated using the following formula:

Where w are the weighting factors, fo are the observed frequencies, and fe are the expected frequencies. Instead of the frequencies, we could also use the calculated probabilities, i.e. the observed probabilities po and the expected probabilities pe.

We have already calculated the observed frequencies. If we would not calculate with the frequencies but with the probabilities, we would simply calculate each frequency by the number of patients, i.e. 75, then we would have the observed probabilities.

But we still need the weights and the expected frequencies. Let's start with the expected frequencies.

Calculate expected frequency

To calculate the expected frequency, we first calculate the rows and the columns sums. So we just sum up all the rows and all the columns.

For example, in the first row we get a sum of 29 with 17 + 8 + 5. We now divide this by 75 of the total number of cases.

Now we can calculate the expected probability for each cell by multiplying the row probability by the column probability. So for the first cell we get 0.35 times 0.39 which is equal to 0.13, for the second cell we get 0.44 times 0.39 which is equal to 0.17.

Now if we multiply each probability by 75, we get the expected frequencies.

Calculate weighting matrix

If we did not use any weighting at all, our matrix would consist only of ones and in the diagonal of zeros. If both raters answered the same, there would be zero in the cells, otherwise one. It does not matter how far apart the raters are in their answers, if they answered something different it is weighted by 1.

The linear weighting matrix can be calculated with the following formula. Let i be the index for the rows and j for the columns. K is the number of expressions, so in our case 3.

So now scores that are close together are weighted less than scores that are far apart.

What about the quadratic weighting? If we use the quadratic weighting instead of the linear weighting, the distances are simply squared again. This way, scores that are far apart are weighted even more heavily in relation to scores that are close together than in the linear case. The weighting matrix is then obtained with the following matrix.

So with this we can now decide whether to use no weighting, use the linear weighting or the quadratic weighting. We just continue to calculate with the linear weighting.

Calculate weighted kappa

Now we can calculate the weighted kappa. We have the weighting matrix, the observed frequency, and the expected frequency. Let's start with the sum up here. We simply multiply each cell of the weighting matrix by the respective cell of the observed frequency and sum this up. So 0 times 17 + 0.5 times 8 to finally 0 times 9.

We now do the same with the weighting matrix and the expected frequency. 0 times 10.05 plus 0.5 times 12.76 to finally 0 times 3.84. If we now calculate everything, we get a weighted kappa of 0.396 with this.

Calculating Cohen's weighted kappa with DATAtab

To calculate weighted Cohen's Kappa online, simply go to the Statistics Calculator, copy your own data into this table, and click on the Reliability tab.

DATAtab automatically tries to assign the appropriate scale level to the data, in this case DATAtab assumes it is nominal data. If we would click on Rater 1 and Rater 2, DATAtab would calculate the unweighted normal Cohens Kappa. In our case, however, these are ordinal variables. So we simply change the scale level to ordinal.

If we now click on both raters, the weighted cohens kappa is calculated. We can now choose whether we want to have linear or quadratic weighting. We see here the cross table, which shows us how often the respective combinations occur. Then we get the results for the Cohens Kappa. With this data, we get a weighted Cohens Kappa of 0.05

If you don't know exactly how to interpret the results, you can just click on Summary in Words: An inter-rater reliability analysis was performed between the dependent samples Rater1 and Rater2. For this, the Weighted Cohens Kappa was calculated, which is a measure of the agreement between two related categorical samples. The Weighted Cohens Kappa showed that there was moderate agreement between the samples Rater1 and Rater2 with κ= 0.5.