Coefficient of Determination | Applied Biostatistics | BIO733_Topic182
Watch on YouTubeVideo summary
When constructing a regression model, it is crucial to evaluate its effectiveness using various statistical measures, one of which is the coefficient of determination, commonly known as R-squared. This statistic serves as a key indicator for determining whether a fitted model adequately explains the data. The foundation of this measure lies in understanding the three sources of variation present in any regression analysis: total deviation, explained deviation, and unexplained deviation. These variations are mathematically related through sum of squares, where the total sum of squares (SST), representing the overall variability in the dependent variable Y, is equal to the sum of the explained sum of squares (SSR) and the unexplained sum of squares (SSE). The SSR quantifies the variation accounted for by the regression model, while the SSE measures the remaining variability that the model fails to explain.
The coefficient of determination is calculated as the ratio of the explained sum of squares to the total sum of squares, effectively expressing the proportion of the dependent variable's variability that is predictable from the independent variables. This value always ranges between 0 and 1, providing an intuitive scale for model performance. An R-squared value close to or equal to 1 indicates that the regression line fits the data points perfectly, meaning the model explains nearly all the variation in Y. Conversely, an R-squared of 0 suggests that the model explains none of the variability, leaving the data points widely scattered around the mean. Generally, values exceeding 0.5 are often considered to indicate a significant relationship, with higher values signifying a better model fit and lower values suggesting a poorer fit.
However, relying solely on R-squared has notable limitations, primarily due to a phenomenon known as the inflation of R-squared. Because adding more independent variables to a regression model will invariably increase the R-squared value—even if those additional variables are not statistically significant or relevant—it can be misleadingly manipulated to appear that a model is improving when it may simply be overfitted. Furthermore, R-squared does not provide information regarding causality, omitted variable bias, multicollinearity among explanatory variables, or whether the chosen set of independent variables is the most appropriate. It also fails to indicate if transforming existing variables could improve the model or if there are sufficient data points to draw solid conclusions. Despite these caveats, R-squared remains a valuable tool for discussing how much of the variation in the dependent variable is explained by the fitted regression model, provided its constraints are understood.
Read the full video transcript
When we build a regression model,
it's very important
to observe
that if the model
is effective.
And the model that we have built is a
good model or not.
Though Box said
that all the models are wrong, but some
are useful.
But still we use various
measures that help us to determine
if model is
effective or not.
One of such measure
is the coefficient of determination.
When we fit any regression model, there
are three sources of variation.
One is the total deviation,
explained deviation, and unexplained
deviation.
All these three deviations
give us
three sources of variation.
Where they can be stated in the form of
equation where total deviation
equals explained deviation
and
unexplained deviation.
So, if you measure these deviations for
each value of Y I and Y I hat
and square each deviation and add up to
square deviations, we have sum Y I minus
Y bar square,
which is called total sum of square,
which is equal to sum Y I hat minus Y
bar square, which is explained sum of
square, plus sum Y I minus Y I hat
square, which is unexplained sum of
square.
These quantities may be considered
measures of dispersion or variability.
So, unexplained sum of square, also
known as sum of square of residuals,
and it is denoted as SSE.
It measure the amount of variability in
Y that is not explained by the model.
To know the amount of variability
explained due to regression model,
we must know that the total variability
present in the data.
The total sum of square, which is also
abbreviated as SST, is a measure of the
total variation in the Y variable.
Since SSE is the minimum of the sum of
square residuals of any linear model,
so SSE is always smaller than the total
sum of squares.
So, these explained sum of square can be
obtained by taking the difference
of the total sum of square and
unexplained sum of square.
Hence, SSR, which is regression sum of
square, also called explained sum of
square,
can be obtained as SST - SSE.
So, using these
sum of squares, we obtain the
coefficient of determination,
which is one of an important measure
to
find out if the fitted model is good
model or not.
So, the coefficient of determination,
which is denoted by R square,
is a statistic that explains the amount
of variation explained in the dependent
variable due to the regression model.
Even if it's a simple linear regression
model or a multiple linear regression
model,
the interpretation of R square stays the
same,
which is defined as the ratio of R
square,
which is SSR / SST.
And since we know sum of square of
regression can be obtained by the sum of
square of total total subtracted sum of
square of error divided by sum of square
of total.
So, using this expression, one can
easily calculate R squared.
Here are some important
things to note for R squared that R
squared ranges between 0 and 1.
R squared
if it's close to 1 or it's equal to 1,
it indicates that model exactly explains
the variability in Y.
This is the case when
in the scatter plot, you will notice
that all the dots are falling right on
the regression line.
And if R squared is equals to 0, this
indicates that the model does not
explain any variability in Y.
Where you will see that there will be a
huge scatter in a scatter plot.
But that's a rare case. Mostly,
R squared values
are
between 0 and 1.
If R squared value that is larger than
0.5, it's usually considered a
significant relationship.
A smaller value of R squared
represents
that model is not a good fit, but a
larger value of R squared represents
that this is a good model.
But there's a problem with R squared,
which is called inflation of R squared.
In any regression model, if we keep
adding independent variables to the
regression model,
R squared, though sometimes small, it
increases whether the variable added to
the model is important variable or not.
So, R squared can be
manipulated by adding variables into the
model, it will increase or by removing
variables from the model, it will
decrease for sure.
Here are some other
caveats about it.
The independent variables are a cause of
the change in the dependent variable.
It does not tells us
this information.
It does not tells us that if omitted
variable bias exists.
It does not tell us the correct
regression was used.
It does not tell us the most appropriate
set of independent variable have been
chosen.
It does not tell us if there is any
collinearity present in the data
on the explanatory variable.
It does not tell us the model might be
improved by using transformed version of
the existing set of independent
variable.
Or
it does not tell us that if there are
enough data points to make a solid
conclusion.
But it simply helps us to discuss
that how much amount of variation in the
dependent variable is explained by the
fitted regression model.