Logistic regression is a special case of regression analysis and is used when the dependent variable is nominally scaled. This is the case, for example, with the variable purchase decision with the two values "buys a product" and "does not buy a product".
Logistical regression analysis is thus the counterpart of linear regression, in which the dependent variable of the regression model must at least be interval-scaled.
With logistic regression, it is now possible to explain the dependent variable or estimate the probability of occurrence of the categories of the variable.
For an online retailer, you need to predict which product a particular customer is most likely to buy. For this, you receive a data set with past visitors and their purchases from the online retailer.
You want to investigate whether a person is susceptible to a certain disease or not. For this purpose, you receive a data set with diseased and non-diseased persons as well as other medical parameters.
Would a person vote for party A if there were elections next weekend?
If you need to calculate a logistic regression, you can easily use the Regression Analysis calculator here on DATAtab
What is a logistic regression?
In the basic form of logistic regression, dichotomous variables (0 or 1) can be predicted. For this purpose, the probability of the occurrence of value 1 (=characteristic present) is estimated.
In medicine, for example, a frequent application is to find out which variables have an influence on a disease. In this case, 0 could stand for "not diseased" and 1 for "diseased". Subsequently, the influence of age, gender and smoking status (smoker or not) on this particular disease could be examined.
Logistic regression and probabilities
In linear regression, the independent variables (e.g., age and gender) are used to estimate the specific value of the dependent variable (e.g., body weight).
In logistic regression, on the other hand, the dependent variable is dichotomous (0 or 1) and the probability that expression 1 occurs is estimated. Returning to the example above, this means: How likely is it that the disease is present if the person under consideration has a certain age, sex and smoking status.
Calculate logistic regression
To build a logistic regression model, the linear regression equation is used as the starting point.
However, if a linear regression were simply calculated for solving a logistic regression, the following result would appear graphically:
As can be seen in the graph, however, values between plus and minus infinity can now occur. The goal of logistic regression, however, is to estimate the probability of occurrence and not the value of the variable itself. Therefore, the this equation must still be transformed.
To do this, it is necessary to restrict the value range for the prediction to the range between 0 and 1. To ensure that only values between 0 and 1 are possible, the logistic function f is used.
The logistic model is based on the logical function. The special thing about the logistic function is that for values between minus and plus infinity, it always assumes only values between 0 and 1.
So the logistic function is perfect to describe the probability P(y=1). If the logistic function is now applied to the upper regression equation the result is
This now ensures that no matter in which range the x values are located, only numbers between 0 and 1 will come out. The new graph now looks like this:
The probability that for given values of the independent variable the dichotomous dependent variable y is 0 or 1 is given by
To calculate the probability of a person being sick or not using the logistic regression for the example above, the model parameters b1, b2, b3 and a must first be determined. Once these have been determined, the equation for the example above is
Maximum Likelihood Methode
To determine the model parameters for the logistic regression equation, the Maximum Likelihood Method is applied. The maximum likelihood method is one of several methods used in statistics to estimate the parameters of a mathematical model. Another well-known estimator is the least squares method, which is used in linear regression.
The Likelihood Function
To understand the maximum likelihood method, we introduce the likelihood function L. L is a function of the unknown parameters in the model, in case of logistic regression these are b1,... bn, a. Therefore we can also write L(b1,... bn, a) or L(θ) if the parameters are summarized in θ.
L(θ) now indicates how probable it is that the observed data occur. With the change of θ, the probability that the data will occur as observed changes.
Maximum Likelihood Estimation
The Maximum Likelihood Estimator can be applied to the estimation of complex nonlinear as well as linear models. In case of logistic regression, the goal is to estimate the parameters b1,... bn, a, which maximize the so-called log likelihood function LL(θ). The log likelihood function is simply the logarithm of L(θ).
For this nonlinear optimization, different algorithms have been established over the years such as the Stochastic Gradient Descent.
Multinomial logistic regression
As long as the dependent variable has two characteristics (e.g. male, female), i.e. is dichotomous, binary logistic regression is used. However, if the dependent variable has more than two instances, e.g. which mobility concept describes a person's journey to work (car, public transport, bicycle), multinomial logistic regression must be used.
In the case of multinomial logistic regression, several binary logistic regressions are calculated. Each instance is transformed into a new variable that investigates the following: Is it the "yes" or "no" characteristic? In the above example, the one variable with three instances now has three new variables, each with two instances ("correct", "incorrect"). Three logistic regression models are now created for these three variables.
Interpretation of the results
The relationship between dependent and independent variables in logistic regression is not linear. Therefore, the regression coefficients cannot be interpreted in the same way as in linear regression. For this reason, odds are interpreted in logistic regression.
An independent variable is called good if it correlates strongly with the dependent variable.
An independent variable is said to be good if it allows the groups of the dependent variable to be distinguished significantly from each other.
The odds are calculated by relating the two probabilities that y is "1" and that y is "not 1".
This quotient can take any positive value. If this value is now logarithmized, values between minus and plus are infinitely possible
These logarithmic odds are usually referred to as "logits".
In a linear regression, the coefficient of determination R2 indicates the proportion of the explained variance. In logistic regression, the dependent variable is scaled nominally or ordinally and it is not possible to calculate a variance, so the coefficient of determination cannot be calculated in logical regression.
However, in order to make a statement about the quality of the logistic regression model, so-called pseudo coefficients of determination have been established, also called pseudo-R squared. Pseudo coefficients of determination are constructed in such a way that they lie between 0 and 1 just like the original coefficient of determination. The best known coefficients of determination are the Cox and Snell R-square and the Nagelkerke R-square.
For the calculation of the Cox and Snell R-square and the Nagelkerke R-square, the likelihood from the so-called zero model L0 and the likelihood L1 from the calculated model is needed. The zero model is a model in which no independent variables are included, L1 is the likelihood of the model with the dependent variables.
Cox and Snell R-square
In the Cox and Snell R square, the ratio of the likelihood function of the zero model L0 and L1 is compared. The better the whole model is compared to the zero model, the lower the ratio between L0 and L1. The Cox and Snell R square is obtained with
The Cox and Snell pseudo-determination measure cannot become one even with a model with a perfect prediction, this corrects the R-square of Nagelkerkes. The Nagelkerkes pseudo coefficient of determination becomes one if the complete model gives a perfect prediction with a probability of 1.
The McFadden's R-square also uses the null model and the total model to calculate the R2.
Chi2 Test and Logistic Regression
In the case of logistic regression, the Chi-square test tells you whether the model is significant overall or not.
Here two models are compared. In one model all independent variables are used and in the other model the independent variables are not used.
Now the Chi-square test compares how good the prediction is when the dependent variables are used and how good it is when the dependent variables are not used.
The Chi2 test now tells us if there is a significant difference between these two results. The null hypothesis is that both models are the same. If the p-value is less than 0.05, this null hypothesis is rejected.
Example logistic regression
As an example for the logistic regression, the purchasing behaviour in an online shop is examined. The aim is to determine the influencing factors that lead a person to buy "immediately", "at a later time" or "not at all" from the online shop after visiting the website. The online shop provides the data collected for this purpose. The dependent variable therefore has the following three characteristics:
- Buy now
- Buy later
- Don't buy anything
Gender, age, income and time spent in the online shop are available as independent variables.
|Purchasing behaviour||Gender||Age||Time spent in online shop|
|Don't buy anything||female||33||65|
|Don't buy anything||female||43||34|
After the above table has been copied to the Statistics Calculator, you will get the following results:
Statistics made easy
- Many illustrative examples
- Ideal for exams and theses
- Statistics made easy on 251 pages
- Only 6.99 €
"Super simple written"
"It could not be simpler"
"So many helpful examples"