Reproducing kernel Hilbert spaces for penalized regression

A tutorial

Alvaro Nosedal-Sanchez, Curtis Storlie, Thomas C M Lee, Ronald Christensen

Research output: Contribution to journalArticle

8 Citations (Scopus)

Abstract

Penalized regression procedures have become very popular ways to estimate complicated functions. The smoothing spline, for example, is the solution of a minimization problem in a functional space. If such a minimization problem is posed on a reproducing kernel Hilbert space (RKHS), the solution is guaranteed to exist, is unique, and has a very simple form. There are excellent books and articles about RKHS and their applications in statistics; however, this existing literature is very dense. This article provides a friendly reference for a reader approaching this subject for the first time. It begins with a simple problem, a system of linear equations, and then gives an intuitive motivation for reproducing kernels. Armed with the intuition gained from our first examples, we take the reader from vector spaces to Ba-nach spaces and to RKHS. Finally, we present some statistical estimation problems that can be solved using the mathematical machinery discussed. After reading this tutorial, the reader will be ready to study more advanced texts and articles about the subject, such as those by Wahba or Gu. Online supplements are available for this article.

Original languageEnglish (US)
Pages (from-to)50-60
Number of pages11
JournalAmerican Statistician
Volume66
Issue number1
DOIs
StatePublished - 2012
Externally publishedYes

Fingerprint

Penalized Regression
Reproducing Kernel Hilbert Space
Minimization Problem
Statistical Estimation
Smoothing Splines
Reproducing Kernel
System of Linear Equations
Vector space
Intuitive
Statistics
Estimate
Hilbert space
Kernel

Keywords

  • Projection principle
  • Regularization
  • Representation theorem
  • Ridge regression
  • Smoothing splines

ASJC Scopus subject areas

  • Mathematics(all)
  • Statistics and Probability
  • Statistics, Probability and Uncertainty

Cite this

Reproducing kernel Hilbert spaces for penalized regression : A tutorial. / Nosedal-Sanchez, Alvaro; Storlie, Curtis; Lee, Thomas C M; Christensen, Ronald.

In: American Statistician, Vol. 66, No. 1, 2012, p. 50-60.

Research output: Contribution to journalArticle

Nosedal-Sanchez, Alvaro ; Storlie, Curtis ; Lee, Thomas C M ; Christensen, Ronald. / Reproducing kernel Hilbert spaces for penalized regression : A tutorial. In: American Statistician. 2012 ; Vol. 66, No. 1. pp. 50-60.
@article{b206e2b0347f4768b5aaaa839d2c7176,
title = "Reproducing kernel Hilbert spaces for penalized regression: A tutorial",
abstract = "Penalized regression procedures have become very popular ways to estimate complicated functions. The smoothing spline, for example, is the solution of a minimization problem in a functional space. If such a minimization problem is posed on a reproducing kernel Hilbert space (RKHS), the solution is guaranteed to exist, is unique, and has a very simple form. There are excellent books and articles about RKHS and their applications in statistics; however, this existing literature is very dense. This article provides a friendly reference for a reader approaching this subject for the first time. It begins with a simple problem, a system of linear equations, and then gives an intuitive motivation for reproducing kernels. Armed with the intuition gained from our first examples, we take the reader from vector spaces to Ba-nach spaces and to RKHS. Finally, we present some statistical estimation problems that can be solved using the mathematical machinery discussed. After reading this tutorial, the reader will be ready to study more advanced texts and articles about the subject, such as those by Wahba or Gu. Online supplements are available for this article.",
keywords = "Projection principle, Regularization, Representation theorem, Ridge regression, Smoothing splines",
author = "Alvaro Nosedal-Sanchez and Curtis Storlie and Lee, {Thomas C M} and Ronald Christensen",
year = "2012",
doi = "10.1080/00031305.2012.678196",
language = "English (US)",
volume = "66",
pages = "50--60",
journal = "American Statistician",
issn = "0003-1305",
publisher = "American Statistical Association",
number = "1",

}

TY - JOUR

T1 - Reproducing kernel Hilbert spaces for penalized regression

T2 - A tutorial

AU - Nosedal-Sanchez, Alvaro

AU - Storlie, Curtis

AU - Lee, Thomas C M

AU - Christensen, Ronald

PY - 2012

Y1 - 2012

N2 - Penalized regression procedures have become very popular ways to estimate complicated functions. The smoothing spline, for example, is the solution of a minimization problem in a functional space. If such a minimization problem is posed on a reproducing kernel Hilbert space (RKHS), the solution is guaranteed to exist, is unique, and has a very simple form. There are excellent books and articles about RKHS and their applications in statistics; however, this existing literature is very dense. This article provides a friendly reference for a reader approaching this subject for the first time. It begins with a simple problem, a system of linear equations, and then gives an intuitive motivation for reproducing kernels. Armed with the intuition gained from our first examples, we take the reader from vector spaces to Ba-nach spaces and to RKHS. Finally, we present some statistical estimation problems that can be solved using the mathematical machinery discussed. After reading this tutorial, the reader will be ready to study more advanced texts and articles about the subject, such as those by Wahba or Gu. Online supplements are available for this article.

AB - Penalized regression procedures have become very popular ways to estimate complicated functions. The smoothing spline, for example, is the solution of a minimization problem in a functional space. If such a minimization problem is posed on a reproducing kernel Hilbert space (RKHS), the solution is guaranteed to exist, is unique, and has a very simple form. There are excellent books and articles about RKHS and their applications in statistics; however, this existing literature is very dense. This article provides a friendly reference for a reader approaching this subject for the first time. It begins with a simple problem, a system of linear equations, and then gives an intuitive motivation for reproducing kernels. Armed with the intuition gained from our first examples, we take the reader from vector spaces to Ba-nach spaces and to RKHS. Finally, we present some statistical estimation problems that can be solved using the mathematical machinery discussed. After reading this tutorial, the reader will be ready to study more advanced texts and articles about the subject, such as those by Wahba or Gu. Online supplements are available for this article.

KW - Projection principle

KW - Regularization

KW - Representation theorem

KW - Ridge regression

KW - Smoothing splines

UR - http://www.scopus.com/inward/record.url?scp=84862649398&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84862649398&partnerID=8YFLogxK

U2 - 10.1080/00031305.2012.678196

DO - 10.1080/00031305.2012.678196

M3 - Article

VL - 66

SP - 50

EP - 60

JO - American Statistician

JF - American Statistician

SN - 0003-1305

IS - 1

ER -