Mathematics > Functional Analysis
[Submitted on 15 Sep 2017 (v1), revised 21 Sep 2017 (this version, v2), latest version 22 May 2018 (v4)]
Title:Optimal approximation of piecewise smooth functions using deep ReLU neural networks
View PDFAbstract:We study the necessary and sufficient complexity of ReLU neural networks---in terms of depth and number of weights---which is required for approximating classifier functions in an $L^2$-sense.
As a model class, we consider the set $\mathcal{E}^\beta (\mathbb R^d)$ of possibly discontinuous piecewise $C^\beta$ functions $f : [-1/2, 1/2]^d \to \mathbb R$, where the different "smooth regions" of $f$ are separated by $C^\beta$ hypersurfaces. For given dimension $d \geq 2$, regularity $\beta > 0$, and accuracy $\varepsilon > 0$, we construct artificial neural networks with ReLU activation function that approximate functions from $\mathcal{E}^\beta(\mathbb R^d)$ up to an $L^2$ error of $\varepsilon$.
The constructed networks have a fixed number of layers, depending only on $d$ and $\beta$ and they have $\mathcal{O}(\varepsilon^{-2(d-1)/\beta})$ many non-zero weights, which we prove to be optimal. For the proof of optimality, we establish a lower bound on the description complexity of the class $\mathcal{E}^\beta (\mathbb R^d)$. By showing that a family of approximating neural networks gives rise to an encoder for $\mathcal{E}^\beta (\mathbb R^d)$, we then prove that one cannot approximate a general function $f \in \mathcal{E}^\beta (\mathbb R^d)$ using neural networks that are less complex than those produced by our construction.
In addition to the optimality in terms of the number of weights, we show that in order to achieve this optimal approximation rate, one needs ReLU networks of a certain minimal depth. Precisely, for piecewise $C^\beta(\mathbb R^d)$ functions, this minimal depth is given---up to a multiplicative constant---by $\beta/d$. Up to a log factor, our constructed networks match this bound. This partly explains the benefits of depth for ReLU networks by showing that deep networks are necessary to achieve efficient approximation of (piecewise) smooth functions.
Submission history
From: Philipp Petersen [view email][v1] Fri, 15 Sep 2017 16:14:39 UTC (87 KB)
[v2] Thu, 21 Sep 2017 14:42:27 UTC (88 KB)
[v3] Fri, 5 Jan 2018 14:35:54 UTC (96 KB)
[v4] Tue, 22 May 2018 11:24:29 UTC (109 KB)
Current browse context:
math.FA
References & Citations
Bibliographic and Citation Tools
Bibliographic Explorer (What is the Explorer?)
Litmaps (What is Litmaps?)
scite Smart Citations (What are Smart Citations?)
Code, Data and Media Associated with this Article
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub (What is DagsHub?)
Gotit.pub (What is GotitPub?)
Papers with Code (What is Papers with Code?)
ScienceCast (What is ScienceCast?)
Demos
Recommenders and Search Tools
Influence Flower (What are Influence Flowers?)
Connected Papers (What is Connected Papers?)
CORE Recommender (What is CORE?)
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.