Loss Functions

Loss Functions for Regression Problems

Loss functions for regression problems are of the general form

where is a scalar intercept and are the regression parameters. Here, is the response vector with n elements and is the predictor matrix with n rows and p columns.

nsoptim::LsRegressionLoss

The classical least-squares (LS) loss function for regression, as defined by

class LsRegressionLoss : public nsoptim::LossFunction<PredictorResponseData>, public nsoptim::ConvexFunction<LsRegressionLoss>

A regression loss function implementing the un-weighted least-squares loss defined as 1/(2n) * sum_{i = 1}^n (y_i - {} - x_i’ . )^2

Public Functions

operator WeightedLsRegressionLoss() const

Cast the un-weighted LS loss as weighted LS loss.

template<typename T, typename = typename std::enable_if<std::is_same<T, RegressionCoefficients<typename T::SlopeCoefficient>>::value, void>::type>
T ZeroCoefficients() const

Get the zero coefficients for this loss type.

Return

zero coefficients.

template<typename T>
double operator()(const RegressionCoefficients<T> &where) const

Evaluate the LS loss function.

Return

loss evaluated at where.

Parameters

  • where: point where to evaluate the loss function.

template<typename T>
double Evaluate(const RegressionCoefficients<T> &where) const

Evaluate the LS loss function.

Return

loss evaluated at where.

Parameters

  • where: point where to evaluate the loss function.

template<typename T>
double Difference(const RegressionCoefficients<T> &x, const RegressionCoefficients<T> &y) const

Get the difference between two sets of regression coefficients.

For the LS loss, the relative difference is an approximation to the 2-norm of the matrix-vector product ||X . (beta1 - beta2) + ( mu1 - mu2 )||_2 < |mu1 - mu2| sqrt(n) + ||X|| ||beta1 - beta2||_2

Return

the relative difference between x and y.

Parameters

  • x: a set of regression coefficients.

  • y: the other set of regression coefficients.

template<typename T>
double Difference(const RegressionCoefficients<T> &x, const RegressionCoefficients<T> &y)

Get the difference between two sets of regression coefficients.

For the LS loss, the relative difference is an approximation to the 2-norm of the matrix-vector product ||X . (beta1 - beta2) + ( mu1 - mu2 )||_2 < |mu1 - mu2| sqrt(n) + ||X|| ||beta1 - beta2||_2

Return

the relative difference between x and y.

Parameters

  • x: a set of regression coefficients.

  • y: the other set of regression coefficients.

template<typename T>
GradientType<T> Gradient(const RegressionCoefficients<T> &where) const

Evaluate the gradient of the weighted LS loss at the given coefficient value.

The gradient of the LS loss is given by intercept = -1/n * sum_{i = 1}^n w_i (y_i - {} - x_i’ . ) beta = -1/n sum_{i = 1}^n w_i (y_i - {} - x_i’ . ) x_i

Parameters

  • coefs: the point where the gradient should be evaluated.

  • gradient_intercept: a pointer where the gradient of the intercept should be stored at.

  • gradient_beta: a pointer where the gradient of the slope should be stored at.

const PredictorResponseData &data() const

Access the data used by this weighted LS loss function.

Return

a constant reference to this loss’ data.

bool IncludeIntercept() const

Check if the intercept term is included in this weighted LS loss function.

Return

true if the intercept term is included, false otherwise.

struct is_ls_regression_loss_tag

Tag this loss function as an LS-type loss.

nsoptim::WeightedLsRegressionLoss

The classical weighted least-squares (LS) loss function for regression, as defined by

with being a vector of non-negative weights of length n.

class WeightedLsRegressionLoss : public nsoptim::LossFunction<PredictorResponseData>, public nsoptim::ConvexFunction<WeightedLsRegressionLoss>

The weighted least-squares loss for regression.

Public Types

using GradientType = RegressionCoefficients<arma::vec>

Type of the gradient if evaluated with coefficients of any type.

using WeightsType = arma::vec

Type of the weights vector.

Public Functions

WeightedLsRegressionLoss(std::shared_ptr<const PredictorResponseData> data, std::shared_ptr<const arma::vec> weights, const bool include_intercept = true)

Initialize a weighted LS regression loss.

Parameters

  • data: A shared pointer to the constant predictor-response data.

  • weighs: A shared pointer to the constant vector of observataion weights. Must be the same length as the number of observations in data.

  • include_intercept: Include an intercept term in the loss? Default is yes.

WeightedLsRegressionLoss(std::shared_ptr<const PredictorResponseData> data, const arma::vec &weights, const bool include_intercept = true)

Initialize a weighted LS regression loss.

Parameters

  • data: A shared pointer to the constant predictor-response data.

  • weighs: A vector of observation weights. Must be the same length as the number of observations in data.

  • include_intercept: Include an intercept term in the loss? Default is yes.

WeightedLsRegressionLoss(const WeightedLsRegressionLoss &other)

Default copy constructor. The new object will share pointers to the data and the weights!

WeightedLsRegressionLoss(WeightedLsRegressionLoss &&other)

Default move constructor. The new object will share pointers to the data and the weights!

WeightedLsRegressionLoss &operator=(const WeightedLsRegressionLoss &other)

Default copy assignment operator. This object will share pointers to the data and the weights with other.

WeightedLsRegressionLoss &operator=(WeightedLsRegressionLoss &&other)

Default move assignment operator. This object will inherit the shared pointers to the data and the weights.

template<typename T, typename = typename std::enable_if<std::is_same<T, RegressionCoefficients<typename T::SlopeCoefficient>>::value, void>::type>
T ZeroCoefficients() const

Get the zero coefficients for this loss type.

Return

Zero coefficients of type T.

template<typename VectorType>
double operator()(const RegressionCoefficients<VectorType> &where) const

Evaluate the weighted LS loss function.

Return

loss evaluated at where.

Parameters

  • where: Point where to evaluate the loss function.

template<typename VectorType>
double Evaluate(const RegressionCoefficients<VectorType> &where) const

Evaluate the weighted LS loss function.

Return

loss evaluated at where.

Parameters

  • where: Point where to evaluate the loss function.

template<typename T>
double Difference(const RegressionCoefficients<T> &x, const RegressionCoefficients<T> &y) const

Get the difference between two sets of regression coefficients.

For the weighted LS loss, the difference is an approximation to the 2-norm of the matrix-vector product

Return

the difference between x and y.

Parameters

  • x: Regression coefficients.

  • y: Regression coefficients.

template<typename T>
double Difference(const RegressionCoefficients<T> &x, const RegressionCoefficients<T> &y)

Get the difference between two sets of regression coefficients.

For the weighted LS loss, the difference is an approximation to the 2-norm of the matrix-vector product

Return

the difference between x and y.

Parameters

  • x: Regression coefficients.

  • y: Regression coefficients.

template<typename T>
GradientType<T> Gradient(const RegressionCoefficients<T> &coefs) const

Evaluate the gradient of the weighted LS loss at the given coefficient value.

Parameters

  • coefs: Point where the gradient should be evaluated.

const PredictorResponseData &data() const

Access the data used by this weighted LS loss function.

Return

a constant reference to this loss’ data.

WeightsType weights() const

Access the un-normalized weights used by this weighted LS loss function.

Return

a vector of weights.

const WeightsType &sqrt_weights() const

Access the normalized sqrt-weights used by this weighted LS loss function. Normalized means that sqrt_weights[i] = sqrt(weights[i] / mean(weights)).

Return

a constant reference to this loss’ sqrt-weights.

double mean_weight() const

Get the average of the weights, i.e., the normalizing constant used for sqrt_weights().

Return

average of the raw weights vector.

bool IncludeIntercept() const

Check if the intercept term is included in this weighted LS loss function.

Return

true if the intercept term is included, false otherwise.

struct is_ls_regression_loss_tag

Tag this loss function as an LS-type loss.