Penalty Functions¶

Non-Adaptive Elastic Net penalties¶

Non-adaptive elastic net (EN) penalties are defined as

where $\text{[math]}$ and $\text{[math]}$ are two hyper-parameters. Because the boundary values of $\text{[math]}$ allow for specific optimizations, the library provides dedicated classes for these special cases to enable compile-time optimizations.

`nsoptim::EnPenalty`¶

The most generic EN penalty.

class EnPenalty : public nsoptim::PenaltyFunction, public nsoptim::ConvexFunction<EnPenalty>¶

The EN penalty function with hyper-parameters alpha and lambda.

Public Types

using GradientType = typename T::SlopeCoefficient¶: Type of the gradient if evaluated with coefficients of type T.

Public Functions

EnPenalty()¶: Initialize an EN penalty, setting alpha to 1 and lambda to 0.

EnPenalty(const double alpha, const double lambda)¶

Initialize an EN penalty.

Parameters

alpha: Value for the alpha hyper-parameter.
lambda: Value for the lambda hyper-parameter.

EnPenalty(const EnPenalty &other)¶: Default copy constructor.

EnPenalty &operator=(const EnPenalty &other)¶: Default copy assignment operator.

EnPenalty(EnPenalty &&other)¶: Default move operator.

EnPenalty &operator=(EnPenalty &&other)¶: Default move assignment.

operator AdaptiveEnPenalty() const¶: Cast the EN penalty to an adaptive EN penalty (with penalty loadings all equal to 1).

void alpha(const double alpha)¶

Set the alpha hyper-parameter.

Parameters

alpha: New alpha value.

double alpha() const¶: Get the value of the alpha hyper-parameter.

void lambda(const double lambda)¶

Set the lambda hyper-parameter.

Parameters

lambda: New lambda value.

double lambda() const¶: Get the value of the lambda hyper-parameter.

template<typename T> double operator()(const RegressionCoefficients<T> &where) const¶

Evaluate the elastic net penalty at where.

Return

penalty evaluated at where.

Parameters

where: Point where to evaluate the penalty function.

template<typename T> double Evaluate(const RegressionCoefficients<T> &where) const¶

Evaluate the elastic net penalty at where.

Return

penalty evaluated at where.

Parameters

where: point where to evaluate the penalty function.

template<typename T> T Gradient(const RegressionCoefficients<T> &where) const¶

Evaluate the subgradient of the EN penalty at the given coefficient value.

Elements of the slope that are 0 will be set to 0 in the gradient.

Parameters

where: Coefficients where the subgradient should be evaluated.

struct is_en_penalty_tag¶: Declare this penalty function as an EN penalty.

`nsoptim::LassoPenalty`¶

An EN penalty with $\text{[math]}$ fixed at 1. Can be cast to a generic EN penalty if desired.

class LassoPenalty : public nsoptim::PenaltyFunction, public nsoptim::ConvexFunction<LassoPenalty>¶

The LASSO penalty function with hyper-parameter lambda.

Public Types

using GradientType = typename T::SlopeCoefficient¶: Type of the gradient if evaluated with coefficients of type T.

Public Functions

LassoPenalty()¶: Initialize a LASSO penalty with lambda set to 0.

LassoPenalty(const double lambda)¶

Initialize a LASSO penalty.

Parameters

lambda: Value for the lambda hyper-parameter.

LassoPenalty(const LassoPenalty &other)¶: Default copy constructor.

LassoPenalty &operator=(const LassoPenalty &other)¶: Default copy assignment.

LassoPenalty(LassoPenalty &&other)¶: Default move constructor.

LassoPenalty &operator=(LassoPenalty &&other)¶: Default move assignment.

double alpha() const¶: Get the value of the alpha hyper-parameter. This is always 1!

void lambda(const double lambda)¶

Set the lambda hyper-parameter.

Parameters

lambda: New lambda value.

double lambda() const¶: Get the value of the lambda hyper-parameter.

operator AdaptiveEnPenalty() const¶: Cast the LASSO penalty to an adaptive EN penalty, with penalty loadings all equal to 1 and alpha set to 1.

operator AdaptiveLassoPenalty() const¶: Cast the LASSO penalty to an adaptive LASSO penalty, with penalty loadings all equal to 1.

operator EnPenalty() const¶: Cast the LASSO penalty to an EN penalty, with alpha set to 1.

template<typename T> double operator()(const RegressionCoefficients<T> &where) const¶

Evaluate the LASSO penalty at where.

Return

Penalty evaluated at where.

Parameters

where: Point where to evaluate the penalty function.

template<typename T> double Evaluate(const RegressionCoefficients<T> &where) const¶

Evaluate the LASSO penalty at where.

Return

Penalty evaluated at where.

Parameters

where: Point where to evaluate the penalty function.

template<typename T> T Gradient(const RegressionCoefficients<T> &where) const¶

Evaluate the subgradient of the LASSO penalty at the given coefficient value.

Elements of the slope that are 0 will be set to 0 in the gradient.

Parameters

where: Coefficients where the subgradient should be evaluated.

struct is_en_penalty_tag¶: Declare this penalty function as an EN penalty.

`nsoptim::RidgePenalty`¶

An EN penalty with $\text{[math]}$ fixed at 0. Can be cast to a generic EN penalty if desired.

class RidgePenalty : public nsoptim::PenaltyFunction, public nsoptim::ConvexFunction<RidgePenalty>¶

The LASSO penalty function with hyper-parameter lambda.

Public Types

using GradientType = typename T::SlopeCoefficient¶: Type of the gradient if evaluated with coefficients of type T.

Public Functions

RidgePenalty()¶: Initialize a Ridge penalty with lambda set to 0.

RidgePenalty(const double lambda)¶

Initialize a Ridge penalty.

Parameters

lambda: Value for the lambda hyper-parameter.

RidgePenalty(const RidgePenalty &other)¶: Default copy constructor.

RidgePenalty &operator=(const RidgePenalty &other)¶: Default copy assignment.

RidgePenalty(RidgePenalty &&other)¶: Default move constructor.

RidgePenalty &operator=(RidgePenalty &&other)¶: Default move assignment.

double alpha() const¶: Get the value of the alpha hyper-parameter. This is always 0s!

void lambda(const double lambda)¶

Set the lambda hyper-parameter.

Parameters

lambda: New lambda value.

double lambda() const¶: Get the value of the lambda hyper-parameter.

operator AdaptiveEnPenalty() const¶: Cast the Ridge penalty to an adaptive EN penalty, with penalty loadings all equal to 1 and alpha set to 0.

operator EnPenalty() const¶: Cast the Ridge penalty to an EN penalty, with alpha set to 0.

template<typename T> double operator()(const RegressionCoefficients<T> &where) const¶

Evaluate the Ridge penalty at where.

Return

Penalty evaluated at where.

Parameters

where: Point where to evaluate the penalty function.

template<typename T> double Evaluate(const RegressionCoefficients<T> &where) const¶

Evaluate the Ridge penalty at where.

Return

Penalty evaluated at where.

Parameters

where: Point where to evaluate the penalty function.

template<typename T> T Gradient(const RegressionCoefficients<T> &where) const¶

Evaluate the gradient of the Ridge penalty at the given coefficient value.

Parameters

where: Coefficients where the gradient should be evaluated.

struct is_en_penalty_tag¶: Declare this penalty function as an EN penalty.

Adaptive Elastic Net penalties¶

Adaptive EN penalties are of the form

where $\text{[math]}$ is the vector of positive penalty loadings. The other parameters are the same as for regular EN penalties.

It is important to note that the penalty loadings are only applied to the L1 part of the penalty! Therefore, there is no “adaptive Rdige penalty”.

`nsoptim::AdaptiveEnPenalty`¶

The most generic adaptive EN penalty.

class AdaptiveEnPenalty : public nsoptim::PenaltyFunction, public nsoptim::ConvexFunction<AdaptiveEnPenalty>¶

The adaptive elastic net penalty function with hyper-parameters alpha, lambda, and non-negative penalty loadings.

Public Types

using GradientType = typename T::SlopeCoefficient¶: Type of the gradient if evaluated with coefficients of T.

Public Functions

AdaptiveEnPenalty(std::shared_ptr<const arma::vec> loadings, const double alpha, const double lambda = 0)¶

Initialize an adaptive EN penalty.

Parameters

loadings: A shared pointer to a constant vector of penalty loadings.
alpha: Value of the alpha hyper-parameter.
lambda: Value of the lambda hyper-parameter. Default is 0.

AdaptiveEnPenalty(const AdaptiveEnPenalty &other)¶: Default copy constructor. The copied penalty will share the pointer to the penalty loadings!

AdaptiveEnPenalty &operator=(const AdaptiveEnPenalty &other)¶: Default copy assignment operator. The copied penalty will share the pointer to the penalty loadings!

AdaptiveEnPenalty(AdaptiveEnPenalty &&other)¶: Default move operator. The copied penalty will share the pointer to the penalty loadings!

AdaptiveEnPenalty &operator=(AdaptiveEnPenalty &&other)¶: Default move assignment. The copied penalty will share the pointer to the penalty loadings!

void lambda(const double lambda)¶

Set the lambda hyper-parameter.

Parameters

lambda: New lambda value.

double lambda() const¶: Get the value of the lambda hyper-parameter.

void alpha(const double alpha)¶

Set the alpha hyper-parameter.

Parameters

alpha: New alpha value.

double alpha() const¶: Get the value of the alpha hyper-parameter.

const arma::vec &loadings() const¶: Get a constant reference to the vector of penalty loadings.

template<typename T> double operator()(const RegressionCoefficients<T> &where) const¶

Evaluate the adaptive EN penalty at where.

Return

penalty evaluated at where.

Parameters

where: Point where to evaluate the penalty function.

template<typename VectorType> double Evaluate(const RegressionCoefficients<VectorType> &where) const¶

Evaluate the adaptive EN penalty at where.

Return

penalty evaluated at where.

Parameters

where: Point where to evaluate the penalty function.

template<typename T> T Gradient(const RegressionCoefficients<T> &where) const¶

Evaluate the subgradient of the adaptive EN penalty at the given coefficient value.

Elements of the slope that are 0 will be set to 0 in the gradient.

Parameters

where: Coefficients where the subgradient should be evaluated.

struct is_en_penalty_tag¶: Declare this penalty function as an EN penalty.

`nsoptim::AdaptiveLassoPenalty`¶

An adaptive EN penalty with $\text{[math]}$ fixed at 1. Can be cast to a generic adaptive EN penalty if desired.

class AdaptiveLassoPenalty : public nsoptim::PenaltyFunction, public nsoptim::ConvexFunction<AdaptiveLassoPenalty>¶

The adaptive lasso penalty with hyper-parmaeter lambda and non-negative penalty loadings.

Public Types

using GradientType = typename T::SlopeCoefficient¶: Type of the gradient if evaluated with coefficients of type T.

Public Functions

AdaptiveLassoPenalty(std::shared_ptr<const arma::vec> loadings, const double lambda = 0)¶

Initialize an adaptive LASSO penalty.

Parameters

loadings: A shared pointer to a constant vector of penalty loadings.
lambda: Value of the lambda hyper-parameter. Default is 0.

AdaptiveLassoPenalty(const AdaptiveLassoPenalty &other)¶: Default copy constructor. The copied penalty will share the pointer to the penalty loadings!

AdaptiveLassoPenalty &operator=(const AdaptiveLassoPenalty &other)¶: Default copy assignment operator. The copied penalty will share the pointer to the penalty loadings!

AdaptiveLassoPenalty(AdaptiveLassoPenalty &&other)¶: Default move operator. The copied penalty will share the pointer to the penalty loadings!

AdaptiveLassoPenalty &operator=(AdaptiveLassoPenalty &&other)¶: Default move assignment. The copied penalty will share the pointer to the penalty loadings!

void alpha(const double) const¶: Set the alpha hyper-parameter. For adaptive LASSO penalties, this has no effect.

double alpha() const¶: Get the value of the alpha hyper-parameter, which is always 1.

void lambda(const double lambda)¶

Set the lambda hyper-parameter.

Parameters

lambda: New lambda value.

double lambda() const¶: Get the value of the lambda hyper-parameter.

const arma::vec &loadings() const¶: Get a constant reference to the vector of penalty loadings.

operator AdaptiveEnPenalty() const¶: Cast the adaptive LASSO penalty to an adaptive EN penalty, with alpha set to 1.

template<typename T> double operator()(const RegressionCoefficients<T> &where) const¶

Evaluate the adaptive LASSO penalty at where.

Return

penalty evaluated at where.

Parameters

where: Point where to evaluate the penalty function.

template<typename T> double Evaluate(const RegressionCoefficients<T> &where) const¶

Evaluate the adaptive LASSO penalty at where.

Return

penalty evaluated at where.

Parameters

where: Point where to evaluate the penalty function.

template<typename T> T Gradient(const RegressionCoefficients<T> &where) const¶

Evaluate the subgradient of the adaptive LASSO penalty at the given coefficient value.

Elements of the slope that are 0 will be set to 0 in the gradient.

Parameters

where: Coefficients where the subgradient should be evaluated.

struct is_en_penalty_tag¶: Declare this penalty function as an EN penalty.

nsoptim

Navigation

Related Topics

Penalty Functions¶

Non-Adaptive Elastic Net penalties¶

`nsoptim::EnPenalty`¶

`nsoptim::LassoPenalty`¶

`nsoptim::RidgePenalty`¶

Adaptive Elastic Net penalties¶

`nsoptim::AdaptiveEnPenalty`¶

`nsoptim::AdaptiveLassoPenalty`¶

Penalty Functions¶

Non-Adaptive Elastic Net penalties¶

nsoptim::EnPenalty¶

nsoptim::LassoPenalty¶

nsoptim::RidgePenalty¶

Adaptive Elastic Net penalties¶

nsoptim::AdaptiveEnPenalty¶

nsoptim::AdaptiveLassoPenalty¶

`nsoptim::EnPenalty`¶

`nsoptim::LassoPenalty`¶

`nsoptim::RidgePenalty`¶

`nsoptim::AdaptiveEnPenalty`¶

`nsoptim::AdaptiveLassoPenalty`¶