When fitting a model to a data, the main objective is to obtain estimates of parameters of interest, such that the differences between predictions and observations (residuals) are minimal. This is usually accomplished by minimizing a "sum of squares" objective function. Three of the most commonly used minimization methods are Ordinary least squares (OLS), weighted least squares (WLS) and Extended least squares (ELS) approach.

Ordinary Least Squares (OLS)

The ordinary least squares estimation minimizes the sum of squared residuals, often called the objective function value. The OLS objective function value is given as

OFVOLS = Σi(ypred-yobs)2

where ypred denotes the predicted value of yobs based on the model, and yobs are observed concentrations. The OLS method makes the following assumption that all observations have equal variance. i.e., the residuals are homoscedastic.

Weighted Least Squares (WLS)

The weighted least squares estimation minimizes the below objection value, given as

OFVWLS = Σi.Wi(ypred-yobs)2

where Wi denotes the weight, which is inversely proportional to the variance of the observation. The variance of the observation varies directly with the magnitude of the observation, as in radioactive counting. The WLS approach assumes unequal error variance (heteroscedastic). Thus WLS gives less weight to observations with larger error variance and vice versa.

Extended Least Squares (ELS)1,2

In the extended least squares estimation method, the variance model can take any form including non linear functions. The ELS estimation minimizes the below objective function value.

OFVELS = Σi.[{ (ypred-yobs)2 /Wi}+ln (Var ( ypred)]

where Wi is given by Var ( ypred). The ELS equation can be derived by maximum likelihood method, and under this assumption the objective function value is proportional to twice the negative log likelihood.

References

  1. Peck CC, Beal SL, Sheiner LB, Nichols AI. Extended least squares nonlinear regression: a possible solution to the 'choice of weights' problem in analysis of individual pharmacokinetic data. J Pharmacokinet Biopharm. 1984 Oct;12(5):545-58.
  2. Peck CC, Sheiner LB, Nichols AI. The problem of choosing weights in nonlinear regression analysis of pharmacokinetic data. Drug Metab Rev. 1984;15(1-2):133-48.