View source for Rectifier (neural networks)

[[Image:Rectifier and softplus functions.svg|thumb|Plot of the rectifier (blue) and softplus (green) functions near {{nobr|''x'' {{=}} 0}}]]{{Short description|In artificial neural networks, an activation function defined as the positive part of its argument.}}
In the context of [[artificial neural network]]s, the '''rectifier''' is an [[activation function]] defined as the positive part of its argument:

: <math>f(x) = x^+ = \max(0, x),</math>

where ''x'' is the input to a neuron. This is also known as a [[ramp function]] and is analogous to [[half-wave rectification]] in electrical engineering. 

This [[activation function]] was first introduced to a dynamical network by Hahnloser et al. in 2000 with strong [[biological]] motivations and mathematical justifications.<ref name="Hahnloser2000">{{cite journal |first=R. |last=Hahnloser |first2=R. |last2=Sarpeshkar |first3=M. A. |last3=Mahowald |first4=R. J. |last4=Douglas |first5=H. S. |last5=Seung |title=Digital selection and analogue amplification coexist in a cortex-inspired silicon circuit |journal=[[Nature (journal)|Nature]] |volume=405 |issue= |year=2000 |pages=947–951 |doi=10.1038/35016072 }}</ref><ref name="Hahnloser2001">{{cite conference |authors=R. Hahnloser, H. S. Seung |year=2001 |title=Permitted and Forbidden Sets in Symmetric Threshold-Linear Networks|conference=NIPS 2001}}</ref> It has been demonstrated for the first time in 2011 to enable better training of deeper networks,<ref name="glorot2011">{{cite conference |authors=Xavier Glorot, Antoine Bordes and [[Yoshua Bengio]] |first= |date= |year=2011 |title=Deep sparse rectifier neural networks |url=http://jmlr.org/proceedings/papers/v15/glorot11a/glorot11a.pdf |conference=AISTATS |volume= |pages= |quote=Rectifier and softplus activation functions. The second one is a smooth version of the first. |via=}}</ref> compared to the widely used activation functions prior to 2011, e.g., the [[Logistic function|logistic sigmoid]] (which is inspired by [[probability theory]]; see [[logistic regression]]) and its more practical<ref>{{cite encyclopedia |authors=[[Yann LeCun]], [[Leon Bottou]], Genevieve B. Orr and [[Klaus-Robert Müller]] |year=1998 |url=http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf |title=Efficient BackProp |editors=G. Orr and K. Müller |encyclopedia=Neural Networks: Tricks of the Trade |publisher=Springer}}</ref> counterpart, the [[hyperbolic tangent]]. The rectifier is, {{as of|2017|lc=y}}, the most popular activation function for [[deep learning|deep neural networks]].<ref>{{cite journal |first1=Yann |last1=LeCun |first2=Yoshua |last2=Bengio |first3=Geoffrey |last3=Hinton |title=Deep learning |journal=Nature |volume=521 |issue=7553 |year=2015 |pages=436–444 |doi=10.1038/nature14539 |pmid=26017442|bibcode=2015Natur.521..436L }}</ref><ref>{{cite arXiv |last1=Ramachandran |first1=Prajit |last2=Barret |first2=Zoph |last3=Quoc |first3=V. Le |date=October 16, 2017 |title=Searching for Activation Functions |eprint=1710.05941 |class=cs.NE}}</ref>

A unit employing the rectifier is also called a '''rectified linear unit''' ('''ReLU''').<ref name="nair2010"/>

Rectified linear units find applications in [[computer vision]]<ref name="glorot2011"/> and [[speech recognition]]<ref name="tothl2013">{{cite conference |authors=László Tóth |year=2013 |title=Phone Recognition with Deep Sparse Rectifier Neural Networks |conference=[[International Conference on Acoustics, Speech and Signal Processing|ICASSP]] |url=http://www.inf.u-szeged.hu/~tothl/pubs/ICASSP2013.pdf}}</ref><ref name="maas2014">Andrew L. Maas, Awni Y. Hannun, Andrew Y. Ng (2014). [https://ai.stanford.edu/~amaas/papers/relu_hybrid_icml2013_final.pdf Rectifier Nonlinearities Improve Neural Network Acoustic Models].</ref> using [[Deep learning|deep neural nets]].

== Softplus ==
A smooth approximation to the rectifier is the [[analytic function]]
:<math>f(x) = \log(1 + e^x),</math>
which is called the '''softplus'''<ref>{{Cite journal |last=Dugas |first=Charles |last2=Bengio |first2=Yoshua |last3=Bélisle |first3=François |last4=Nadeau |first4=Claude |last5=Garcia |first5=René |date=2000-01-01 |title=Incorporating second-order functional knowledge for better option pricing |url=http://papers.nips.cc/paper/1920-incorporating-second-order-functional-knowledge-for-better-option-pricing.pdf |journal=Proceedings of the 13th International Conference on Neural Information Processing Systems (NIPS'00) |publisher=MIT Press |volume= |pages=451–457 |quote=Since the sigmoid ''h'' has a positive first derivative, its primitive, which we call softplus, is convex. |via=}}</ref><ref name="glorot2011" /> or '''SmoothReLU''' function.<ref>{{Cite web |url=https://software.intel.com/sites/products/documentation/doclib/daal/daal-user-and-reference-guides/daal_prog_guide/GUID-FAC73B9B-A597-4F7D-A5C4-46707E4A92A0.htm |title=Smooth Rectifier Linear Unit (SmoothReLU) Forward Layer |date=2017 |website=Developer Guide for Intel Data Analytics Acceleration Library |language=en-US |access-date=2018-12-04 }}</ref> The derivative of softplus is <math>f'(x) = \frac{e^x}{1 + e^x} = \frac{1}{1 + e^{-x}}</math>, the [[logistic function]]. The logistic sigmoid function is a smooth approximation of the derivative of the rectifier, the [[Heaviside step function]].

The multivariable generalization of single-variable softplus is the [[LogSumExp]] with the first argument set to zero:
: <math>\operatorname{LSE_0}^+(x_1, ..., x_n) := \operatorname{LSE}(0, x_1, ..., x_n) = \log\left(1 + e^{x_1} + \cdots + e^{x_n} \right).</math>
The LogSumExp function itself is
: <math>\operatorname{LSE}(x_1, \dots, x_n) = \log\left(e^{x_1} + \cdots + e^{x_n}\right),</math>
and its gradient is the [[softmax function|softmax]]; the softmax with the first argument set to zero is the multivariable generalization of the logistic function. Both LogSumExp and softmax are used in machine learning.

== Variants ==

===Noisy ReLUs===
Rectified linear units can be extended to include [[Gaussian noise]], making them noisy ReLUs, giving<ref name="nair2010">{{cite conference |authors=Vinod Nair and [[Geoffrey Hinton]] |first= |date= |year=2010 |title=Rectified Linear Units Improve Restricted Boltzmann Machines |url=https://www.cs.toronto.edu/~hinton/absps/reluICML.pdf |conference=[[International Conference on Machine Learning|ICML]] |volume= |pages= |via=}}</ref>
: <math>f(x) = \max(0, x + Y)</math>, with <math>Y \sim \mathcal{N}(0, \sigma(x)).</math>

Noisy ReLUs have been used with some success in [[restricted Boltzmann machine]]s for computer-vision tasks.<ref name="nair2010"/>

=== Leaky ReLUs ===
Leaky ReLUs allow a small, positive gradient when the unit is not active.<ref name="maas2014"/>
:<math>f(x) = \begin{cases}
    x & \text{if } x > 0, \\
    0.01x & \text{otherwise}.
\end{cases}</math>

==== Parametric ReLUs ====
Parametric ReLUs (PReLUs) take this idea further by making the coefficient of leakage into a parameter that is learned along with the other neural-network parameters.<ref name="prelu">{{cite arxiv  |eprint=1502.01852 |last1=He |first1=Kaiming |title=Delving Deep into Rectifiers: Surpassing Human-Level Performance on Image ''Net'' Classification |last2=Zhang |first2=Xiangyu |last3=Ren |first3=Shaoqing |last4=Sun |first4=Jian |class=cs.CV |year=2015}}</ref>

:<math>f(x) = \begin{cases}
    x & \text{if } x > 0, \\
    a x & \text{otherwise}.
\end{cases}</math>
Note that for <math>a \leq 1</math>, this is equivalent to
: <math>f(x) = \max(x, ax)</math>
and thus has a relation to "maxout" networks.<ref name="prelu"/>

=== ELUs ===
Exponential linear units try to make the mean activations closer to zero, which speeds up learning. It has been shown that ELUs can obtain higher classification accuracy than ReLUs.<ref>{{Cite arxiv  |eprint=1511.07289 |last1=Clevert |first1=Djork-Arné |title=Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs) |last2=Unterthiner |first2=Thomas |last3=Hochreiter |first3=Sepp |class=cs.LG |year=2015}}</ref>

: <math>f(x) = \begin{cases} 
 x & \text{if } x > 0, \\ 
 a(e^x-1) & \text{otherwise},
\end{cases}</math>

where <math>a</math> is a [[Hyperparameter (machine learning)|hyper-parameter]] to be tuned, and <math>a \geq 0</math> is a constraint.

== Advantages ==

* Biological plausibility: One-sided, compared to the [[Antisymmetric relation|antisymmetry]] of [[tanh]].
* Sparse activation: For example, in a randomly initialized network, only about 50% of hidden units are activated (having a non-zero output).
* Better gradient propagation: Fewer [[Vanishing gradient problem|vanishing gradient]] problems compared to sigmoidal activation functions that saturate in both directions.<ref name="glorot2011" />
* Efficient computation: Only comparison, addition and multiplication.
* Scale-invariant: <math>\max(0, ax) = a \max(0, x) \text{ for } a \geq 0</math>.

Rectifying activation functions were used to separate specific excitation and unspecific inhibition in the neural abstraction pyramid, which was trained in a supervised way to learn several computer vision tasks.<ref name=NeuralAbstractionPyramid>{{cite book |last=Behnke |first=Sven |year=2003 |title=Hierarchical Neural Networks for Image Interpretation |url= https://www.researchgate.net/publication/220688219_Hierarchical_Neural_Networks_for_Image_Interpretation |series=Lecture Notes in Computer Science |volume=2766 |publisher=Springer |doi= 10.1007/b11963}}</ref> In 2011,<ref name="glorot2011"/> the use of the rectifier as a non-linearity has been shown to enable training deep [[Supervised learning|supervised]] neural networks without requiring [[Unsupervised learning|unsupervised]] pre-training. Rectified linear units, compared to [[sigmoid function]] or similar activation functions, allow faster and effective training of deep neural architectures on large and complex datasets.

== Potential problems ==
* Non-differentiable at zero; however, it is differentiable anywhere else, and the value of the derivative at zero can be arbitrarily chosen to be 0 or 1.
* Not zero-centered.
* Unbounded.
* Dying ReLU problem: ReLU neurons can sometimes be pushed into states in which they become inactive for essentially all inputs. In this state, no gradients flow backward through the neuron, and so the neuron becomes stuck in a perpetually inactive state and "dies". This is a form of the [[vanishing gradient problem]]. In some cases, large numbers of neurons in a network can become stuck in dead states, effectively decreasing the model capacity. This problem typically arises when the learning rate is set too high. It may be mitigated by using leaky ReLUs instead, which assign a small positive slope for ''x''&nbsp;<&nbsp;0.

==See also==
*[[Softmax function]]
*[[Sigmoid function]]
*[[Tobit model]]

==References==
{{Reflist|30em}}

[[Category:Artificial neural networks]]