Two-hidden-layer Feedforward Neural Networks are Universal Approximators: A Constructive Approach

TWO-HIDDEN-LAYER FEEDFORWARD NETWORKS ARE UNIVERSAL APPROXIMATORS:ACONSTRUCTIVE APPROACH A PREPRINT Rocio Gonzalez-Diaz Dept. of Applied Mathematics I University of Seville [email protected] Miguel A. Gutiérrez-Naranjo Dept. of Computer Science and Artificial Intelligence University of Seville [email protected] Eduardo Paluzo-Hidalgo Dept. of Applied Mathematics I University of Seville [email protected] July 29, 2019 ABSTRACT It is well known that Artificial Neural Networks are universal approximators. The classical result proves that, given a continuous function on a compact set on an n -dimensional space, then there exists a one-hidden-layer feedforward network which approximates the function. Such result proves the existence, but it does not provide a method for finding it. In this paper, a constructive approach to the proof of this property is given for the case of two-hidden-layer feedforward networks. This approach is based on an approximation of continuous functions by simplicial maps. Once a triangulation of the space is given, a concrete architecture and set of weights can be obtained. The quality of the approximation depends on the refinement of the covering of the space by simplicial complexes. Keywords Universal approximation theorem · Simplicial approximation theorem · Multilayer feedforward network · Simplicial Complexes 1 Introduction One of the first results in the development of neural networks are the universal approximation theorems [ 1 , 2 ]. These classical results show that a multilayer feedforward network with only one hidden layer and non-polynomial activation function (like the sigmoid) can approximate every continuous function over a compact set on Rn . It is well-known that these results have two important drawbacks for their practical use: On the one hand, the width of the hidden layer grows exponentially and, on the other hand, the proof is not constructive, and it does not provide an algorithm for building such network. Bearing these results in mind, many researchers are paying attention to theoretical aspects of the current success of new neural network architectures and asking for bounds for the depth and width of such networks and the possibility of acting as universal approximators (see, e.g., [3, 4, 5] among many others). It is out of any doubt that the use of many hidden layers is a big contribution to the success of deep learning architectures [ 6 ], but instead of exploring the power of depth, recently several studies have made interesting contributions on the power of width [ 7 , 8 , 9 ]. To sum up, the authors show that there exists continuous functions on compact sets which cannot be approximated by a neural network if the width of the layers is not larger than a bound, regardless the depth of the network. All authors are listed by alphabetical order. APREPRINT - JULY 29, 2019 In this paper, we provide an effective method for finding the weights of a multilayer feedforward network with two hidden layers which approximates any continuous mappings between two triangulable sets on metric spaces. Let us remark that the method is constructive and it only depends on the desired level of approximation to the original continuous mapping. Our approach is based on a strong result from computational topology. Roughly speaking, our result is rooted in two observations: On the one hand, triangulable sets can be modelled by simplicial complexes, and a continuous mapping between two triangulable spaces can be approximated by a mapping between simplicial complexes. On the second hand, a mapping between simplicial complexes can be modelled as a two-hidden-layer feedforward network. Let us remark that the classical result is valid for all compact set on Rn and our theorem is valid for triangulable sets of Rn , which is a smaller collection of sets. Nevertheless, as it will be detailed below, compact and triangulable sets differ in a quite technical topological property, and therefore, our result have a practical application in many current problems. The paper is organized as follows: in Section 2 the preliminary notions about multilayer feedforward networks and simplicial complexes are provided. Then, in Section 3, a concrete architecture that acts equivalently to a given simplicial map is given. In Sections 4 and 5, we extend the already known simplicial approximation theorem and universal approximation theorem, respectively. Finally, conclusions are supplied in 6. 2 Background In this section, some notions about artificial neural networks and simplicial complexes are recalled. 2.1 Multilayer feedforward networks Artificial neural networks are inspired in biological networks of alive neurons in a brain. The number of different architectures, algorithms, and areas of application has recently grown in many directions. In general, a neural network can be formalized as a mapping Nω,Θ:Rn→Rm which depends on a set of weights ω and a set of parameters Θ which involves the description of activation functions, layers, synapses between neurons, and whatever other consideration in its architecture. A good introduction to artificial neural networks can be found in [10]. In this paper, we focus on one of the simplest classes of artificial neural networks: multilayer feedforward networks which consist of three or more fully connected layers of nodes (neurons): an input layer, an output layer, and one or more hidden layers. Each node in one layer has an activation function and it is connected with every node in the following layer. The next definition formalizes this idea. Definition 1. A multilayer feedforward network defined on a real-valued n -dimensional space is a mapping N:Rn→ Rmsuch that, for each x∈Rn,N(x)is computed as the composition of functions N(x) = fk+1 ◦fk◦ · · · ◦ f1(x) where k∈Z is the number of hidden layers, k≥1 , fi:Rdi−1→Rdi , 1≤i≤k+1 , is defined as fi(y) = φi(W(i);y; bi) being W(i)=Wdi di−1 a real-valued di−1×di matrix, bi∈Rdi the bias term, and φi a bounded, continuous, and non-constant function (called activation function). Notice that d0=n , dk+1 =m , di∈Z , 1≤i≤k , is called the width of the i-th hidden layer and max{d0, . . . , dk+1}is the width of the multilayer feedforward network. Next, we re-write one of the most important theoretical results of multilayer feedforward networks adapted to our notation. Theorem 1 (Universal Approximation Theorem, [ 2 ]) . Let A be any compact subset of Rn . The space of real-valued continuous functions on A is denoted by C(A) . Then, given any  > 0 and any function g∈C(A) , there exists a one-hidden-layer feedforward network N:Rn→R defined as N(x) = f2◦f1(x) with f1(y) = φ1(W(1);y;b1) and f2(y) = W(2)y, such that Nis an approximation of the function g, that is, |N(x)−g(x)|<  for all x∈A. The known proofs given to this theorem are non constructive (see [ 11 , 2 , 1 ]), i.e., given g and  , the theorem claims that a neural network N exists fulfilling the conditions, but there is no an algorithm to build it in the general case. In this paper, we will provide a constructive approach to this theorem but using a multilayer feedforward network with two hidden layers, instead of one. The most important restriction is that our result is valid for triangulable sets, instead of compact ones, but, as pointed out above, it covers most of the real-world problems. 2 APREPRINT - JULY 29, 2019 2.2 Simplicial Complexes The main result in this paper is based on a strong result in computational topology known as the Simplicial Approximation Theorem. Next, we recall such theorem and some basics on the theory of simplicial complexes. We have followed the notation from [12, 13]. Simplicial complexes are a data structure widely used to represent topological spaces. They are a versatile mathematical objects to represent a decomposition of a topological space into simple pieces. For a further comprehension of the field [ 14 , 15 , 12 , 16 , 13 ] can be consulted. Next, we recall the formal definition of the small pieces, called simplices, that are used to construct our main data structure: simplicial complexes. Definition 2. Let v0, v1, . . . , vi be i+ 1 affinely independent points in Rn with (i≤n) . An i -simplex, σ= (v0, v1, . . . , vi), is the convex set nx∈Rnx= i X j=0 λjvjwith λj≥0and i X j=0 λj= 1o. The points v0, v1, . . . , vi are the vertices of σ . Let us denote by dim σ the dimension of σ which in this case is i . We say that σ0is a face of σif σ0is an i0-simplex, with i0≤i, and whose vertices are also vertices of σ. When several simplices are joint together, a more complex structure called simplicial complex is built. The following definition exhibits the way simplices can be glued together to obtain such a simplicial complex. Definition 3. A simplicial complex Kis a finite collection of simplices such that: 1. σ∈Kand σ0⊆σimplies σ0∈K; 2. σ,µ∈Kimplies σ∩µis either empty or a face of both. The underlying space of K , denoted by |K| is the union of its simplices together with the topology inherited from the ambient Euclidean space where the simplices are placed. A subcomplex L of K is a simplicial complex such that L⊆K . The subcomplex of K consisting of all simplices of K of dimension j or less is called the j -skeleton of K and it is denoted by K(j). Let us remark that the 0-skeleton of Kis its vertex set. Definition 4. Let K be a simplicial complex. A simplex σ∈K is maximal if it is not a face of any other simplex in K . A simplex σ0is adjacent to σif σand σ0share a face in K. Next, we recall the idea of star of a simplex σ in a simplicial complex K . Intuitively, it is the set of simplices in K which have σas a face. Definition 5. Let Kbe a simplicial complex and σa face of K. The star of σin K, denoted by st(σ), is defined as: st(σ) = µ∈K∃ξ∈Ksuch that σ⊆ξand µ⊆ξ. Simplicial complexes are combinatorial data structures used to “model" topological spaces. A way to obtain a refined model from an existing one is to subdivide it in small pieces such that the result is topologically equivalent to the former. Definition 6. Let Kand K0be simplicial complexes. It is said that K0is a subdivision of Kif: 1. |K|=|K0|; 2. σ0∈K0implies that there exists σ∈Ksuch that σ⊆σ0. The barycentric subdivision is a concrete example of subdivision of simplicial complexes. Let us recall that the barycenter of an i-simplex σ= (v0. . . vi)is b(σ) = i X j=0 1 j+ 1vj. By using this definition, the idea of barycentric subdivision of a simplicial complex arises in a natural way. Definition 7. Let K be a simplicial complex. The barycentric subdivision of the 0 -skeleton of K is defined as the set of vertices of K , that is, Sd K(0) =K(0) . Assuming we have Sd K(i−1) , which denotes the barycentric subdivision of the (i−1) -skeleton of K , Sd K(i) is built by adding the barycenter of every i -simplex as a new vertex and connecting it to the simplices that subdivide the boundary of such i -simplex.The iterated application of barycentric subdivisions is denoted by SdtKwhere tis the number of iterations. 3 APREPRINT - JULY 29, 2019 Figure 1: Example of simplicial complex. (d) is a 0 -simplex, (o, p) is a 1 -simplex , (a, b, c) is a 2 -simplex, and (e, f, g, h) is a 2 -simplex. (a, b) is a face of the simplex (a, b, c) . The simplices (e, f, g, h) , (a, b, c) , (i, h, k) , (i, j, k) , (o, p),(o, q),(p, q), and (d)are the maximal simplices of the simplicial complex. Figure 2: On the left the 2-simplex (a, b, c)is shown, and on the right its first barycentric subdivision is plotted. Let us see now how the “size" of the simplices of a simplicial complex can be measured. Definition 8. Let Kbe a simplicial complex. The diameter of a simplex σin Kis defined as δ(σ) = max{||x−y|| | x, y ∈σ} and the mesh of Kis defined as m(K) = max{δ(σ)|σ∈K} Theorem 2 ([ 13 , p. 86]) . Given a simplicial complex K , and  > 0 , there exists an integer t > 0 such that m(SdtK)≤. Next, we recall one of the main tools used in this paper. A simplicial complex can be seen as mathematical model of a region of an n -dimensional space. By using subdivisions, more and more tuned simplicial complexes can be obtained. Let us now think about maps between simplicial complexes. These maps can be considered as extensions of simpler maps defined from vertices to vertices of two simplicial complexes. Interestingly, such maps can be considered as approximations of continuous functions defined on the underlying topological spaces that the simplicial complexes are modeling. Let us formalize these notions. 4 APREPRINT - JULY 29, 2019 Definition 9. Let K and L be two simplicial complexes. A vertex map is a function ϕ:K(0) →L(0) with the property that the vertices of every simplex in Kmap to vertices of a simplex in L. A vertex map ϕcan be extended to a continuous map ϕc:|K|→|L|in the following way. Definition 10. Let K and L be two simplicial complexes and let ϕ:K(0) →L(0) be a vertex map. The simplicial map ϕc induced by ϕ is defined as follows. Let x∈ |K| . Then there exists an i -simplex σ= (v0, . . . , vi)) in K and numbers λj≥0such that i X j=0 λj= 1 and x= i X j=0 λjvj. Then ϕc(x) = i X j=0 λjϕ(vj). Any vertex map ϕ induces a simplicial map ϕc , but, if we want that such map is an approximation between the underlying spaces of both simplicial complexes, a restriction about the star of each simplex needs to be added. Definition 11. Let K and L be simplicial complexes and let g:|K|→|L| be a continuous map. A simplicial map ϕc:|K|→|L|induced by a vertex map ϕ:K(0) →L(0) is a simplicial approximation if g(|st(v)|)⊂ | st(ϕ(v)) | for each vertex vof K. Furthermore, there exists strong results on simplicial complexes. Concretely, we focus our attention in the Simplicial Approximation Theorem which ensures the existence of simplicial maps that approximate continuous maps arbitrarily closely. Theorem 3 (Simplicial Approximation Theorem [ 12 , p. 56]) . If g:|K|→|L| is continuous, then there is a sufficiently large integer t > 0such that ϕc:|SdtK|→|L|is a simplicial approximation of g. Theorems 1 and 3 are key results in their respective fields. Our aim is to take Theorem 3 as a pillar for obtaining a result close to Theorem 1. Roughly speaking, the idea is to consider a simplicial approximation between two simplicial complexes. On the one hand, such simplicial approach is characterized via a vertex map which can be expressed as a neural network. On the other hand, the simplicial approximation can be chosen in such way that it approximates the continuous map between the underlying spaces of the simplicial complexes. Since the parameters of the neural network can be effectively obtained from the vertex map, this method provides a constructive way to find a neural network which approximates a continuous function on the underlying spaces of two simplicial complexes. Following that aim, let us formalize the relationship between topological spaces and simplicial complexes. Definition 12. A triangulation of a topological space X consists of a simplicial complex K and a homeomorphism τ:X→ |K|. The spaces that can be triangulated by simplicial complexes (see [ 16 , Theorem A.7, p. 525]) are every compact, locally contractible that can be embedded in Rn for some n . Let us remark that the Universal Approximation Theorem (Theorem 1) is valid for any compact subset of Rn , regardless if they are locally contractible or not. Not all compact sets in a metric space are locally contractible (see [ 17 , Chapter 17.7, p. 426]). Nevertheless, non-locally contractible spaces are very odd in Rn and this technical topological property has no practical application in real-world problems. Therefore, we can say that the results proved on triangulable spaces are true on a large amount of current neural network problems. Next, we can extend the definition of a mesh of a simplicial complex in the following way. Definition 13. Let X be a triangulable topological space and (K, τ) a triangulation of X . The mesh of X induced by (K, τ)is defined as ˜m(K,τ)(X) = max{˜ δ(σ)|σ∈K} where ˜ δ(σ) = max ||x−y|| such that x=τ−1(a), y =τ−1(b)with a, b ∈σ is the extended diameter of a simplex. 5 APREPRINT - JULY 29, 2019 3 Multilayer Feedforward Networks and Simplicial Maps In this section, we will show that such simplicial maps can be modelled via multilayer feedforward networks in a straightforward way. In the following theorem, we will compute a two-hidden-layer feedforward network to a simplicial map ϕc:|K|→|L| where all the simplices of K and L are maximal simplices with maximal dimension. This is not an important constraint in our case, since our final aim is to design a multilayer feedforward network that approximates a continuous function between triangulable spaces. Theorem 4. Given a simplicial map ϕc:|K|→|L| where K and L are composed by maximal simplices with maximal dimension. A two-hidden layer feedforward network Nϕ such that ϕc(x) = Nϕ(x) for all x∈ |K| can be explicitly defined. Proof. Let us assume |K| ⊂ Rn and |L| ⊂ Rm . Besides, let us consider that K is composed by k maximal n - simplices {σ1,...σk} where σi=vi 0, . . . , vi n for all i , and L by ` maximal m -simplices {µ1, . . . , µ`} where µi=uj 0, . . . , uj m for all j . Let us consider a multilayer feedforward network Nϕ with the following architecture: (1) An input layer composed by d0=n neurons; (2) a first hidden layer composed by d1=k·(n+ 1) neurons where ; (3) a second hidden layer composed by d2=`·(m+ 1) neurons; and (4) an output layer with d3=m neurons. Then, Nϕ=f3◦f2◦f1 being fi(y) = φi(W(i);y;bi) , i= 1,2,3 . The idea is to codify the simplicial complexes involved in the mapping in the hidden layers of the multilayer feedforward network. Firstly, a point x in Rn is transformed into a k·(n+ 1) vector. This vector can be seen as the juxtaposition of k vectors of dimension n+ 1 , one for each of the k simplices in K . Each vector of dimension k represents the barycentric coordinates of x with respect to the corresponding simplex. The matrix of the weights to obtain the output of the first hidden layer, W(1) , and the corresponding bias term b1can be obtained from the equations for getting the barycentric coordinates as follows: W(1) =   W(1) 1 . . . W(1) k   ∈ Mk(n+1)×n, where W(1) i∈ M(n+1)×nis: vi 0· · · vi n 1· · · 1−1 =W(1) iBi, and Bi∈Rn+1 is: b1=   B1 . . . Bk   ; The matrix of weights between the first and the second hidden layer encodes the vertex map between vertices. Since K is encoded in the first hidden layer with k·(n+ 1) neurons and L is represented via `·(m+ 1) neurons, the matrix is composed by 0’s and 1’s. The value 1 represents that the corresponding vertices are related via the vertex map, and 0 represents that they do not. Then, this matrix, W(2), can be defined as follows: W(2) =W(2) s1,s2∈ M`(m+1)×k(n+1), being s1=j(r+ 1) and s2=i(t+ 1) for i= 1, . . . , k;j= 1, . . . , `;t= 0, . . . , n; and r= 0, . . . , m. Finally, W(2) s1,s2=1if ϕ(vi t) = wj r, 0otherwise; and b2= 0. The output of the second hidden layer can be seen as the juxtaposition of ` vectors of dimension m+ 1 , one vector for each simplex in the simplicial complex L . Each of the vectors represents the barycentric coordinates of ϕc(x) with respect to the corresponding simplex. For each simplex in L , the corresponding barycentric coordinates are a vector of m+ 1 components whose sum is 1, but all coordinates are greater than or equal to zero if ϕc(x) belongs to that simplex. Only those coordinates are considered, and, in the next step, φ3(W(3);y;b3) transforms this vector and the outputs the Cartesian coordinates of ϕc(x). Then: 6 APREPRINT - JULY 29, 2019 W(3) =W(3) 1· · · W(3) `∈ Mm×`(m+1), being W(3) j=uj 0· · · uj m;b3= 0. And, finally, the following functions are defined. For the first two layers: φi(W(i);y;bi) = W(i)y+bifor i= 1,2. In the case of the third layer, φ3is defined as follows: φ3(W(3);y;b3) = P` j=1 zjψ(yj) P` j=1 yj with zj=W(3) j·yj , and y=   y1 . . . y`   ∈ M`·(m+1) where yj∈Rm+1 and ψ(yj)=1 if all the coordinates of yj are greater than or equal to 0, and ψ(yj)=0otherwise. This proposition establishes that two-hidden-layer feedforward network can act equivalently to simplicial maps. Besides, the architecture and the specific computation of the parameters is shown. 4 Simplicial Approximation Theorem extension In this section, we provide an extension of the simplicial approximation theorem, and an explicit algorithm to compute a simplicial approximation as close as we want to a given continuous function g:|K|→|L| between the underlying spaces of, respectively, two simplicial complexes K and L . The first observation is that the Simplicial Approximation Theorem (Theorem 3) refers to any continuous mapping. Continuity is a property of functions in topological spaces, not necessarily metric spaces. The next result introduces the concept of metric into the simplicial approximations. Proposition 1. Given  > 0 and a continuous function g:|K|→|L| between the underlying spaces of two simplicial complexes K and L , there exists t1, t2>0 such that ϕc:|Sdt1K|→|Sdt2L| is a simplicial approximation of g and ||g−ϕc|| ≤ . Proof. By Theorem 2, there exists t2 such that m(Sdt2L)≤ . Then, by Theorem 3, there exist t1 such that ϕc:|Sdt1K| → | Sdt2L|is a simplicial approximation of g: |K| |L| K L Sdt1KSdt2L |Sdt1K| | Sdt2L| g Sd Sd ϕc Besides, ||g−ϕc|| ≤ because m(Sdt2L)≤. Algorithm 1 is inspired in the proof of the Simplicial Approximation Theorem in [ 12 , p.56] and computes a vertex map ϕ: (SdtK)(0) →(SdL)(0) from which the simplicial approximation ϕc:|SdtK|→|L| of a given function g:|K|→|L|between the underlying spaces of, respectively, two simplicial complexes Kand Lcan be defined. 7 APREPRINT - JULY 29, 2019 Algorithm 1: Computing simplicial approximations Input: A continuous function g:|K|→|L|between the underlying spaces of two simplicial complexes Kand L, and an integer twhere Sdtksatisfies the star condition:g(|st(v)|)⊆ |st(w)|for all v∈(Sdtk)(0) and for some w∈L(0). Output: A simplicial approximation ϕcof g. foreach point v∈(SdtK)(0) do Let ϕ(v) = wthat exists by the star condition. Theorem 5. Given a continuous function g:|K| → |L| and  > 0 , a two-hidden-layer feedforward network N such that ||N − g|| ≤ can be explicitly defined. Proof. By proposition 1, there exists a simplicial approximation ϕc of g such that ||g−ϕc|| ≤  . Then, by Theorem 4 there exists Nϕsuch that ϕc=Nϕ. 5 Universal Approximation Theorem Extension The results of the previous section can be extended to triangulable spaces. In this case, the approach is constructive if the homeomorphisms of the triangulations are known. In the previous sections, we have proved that a continuous function between triangulable sets can be approximated by using the simplicial approximating theorem. In this section, using the new version of the simplicial approximation theorem (Proposition 3) we can give a constructive version of the Universal Approximation Theorem that approximates any continuous function (under some conditions) arbitrarily close. Proposition 2. Let (K, τ) be a triangulation of a space X . For all  > 0 there exists n and γ > 0 such that if m(SdnK)≤γthen ˜m(SdnK,τK)(X)≤. Proof. Consider x, y0∈ |K| . If x and y0 belong to σ0∈K . Then, ||x−y0|| ≤ δ(σ0) and ||τ(x)−τ(y0)|| ≤ ˜ δ(σ0) . Otherwise, we repeat the reasoning with SdK . Now, x and y0 can belong to the same simplex in SdK or not. If they belong to the same simplex, write y1=y0 . If not, take a new point y1 such that x, y1∈σ1 and σ1∈Sd σ0 . Therefore, ||x−y1|| ≤ δ(σ1)≤δ(σ0) . Besides, ||τ(x)−τ(y1)|| ≤ ˜ δ(σ1)≤˜ δ(σ0) . This can be iterated: ||x−yt|| ≤ δ(σn)≤ · · · ≤ δ(σ1)≤δ(σ0) and ||τ(x)−τ(yt)|| ≤ ˜ δ(σt)≤ · · · ≤ ˜ δ(σ1)≤˜ δ(σ0) . By this, we have defined a sequence {yi}t i=0 that converges to x . Therefore, given  > 0 , there exists n such that ||τ(x)−τ(yn)|| ≤  . Let us suppose, without loss of generality, that ˜ δ(σn) = ˜m(SdnK,τ)(X). Then, we can consider γ=m(SdnK). Corollary 1. Given  > 0and a triangulation (K, τ)of X, there exists tsuch that ˜m(SdtK)≤. Proof. By Theorem 2 there exists t0 such that m(Sdt0 K)≤γ . Then, by Proposition 2, there exists t such that ˜m(SdtK,X)≤. Finally, we reach the main result of this section. Given a continuous map g between two triangulable spaces X and Y , we can obtain two simplicial complexes K and L associated to them, and a simplicial approximation ϕc between them which approximates g. Proposition 3. Let X and Y be two triangulable topological spaces, g:X→Y a continuous map, and  > 0 . Then, there exists two triangulations (K, τK) and (L, τL) of X and Y , respectively, and a simplicial approximation ϕc: |Sdt1K|→|Sdt2L|such that ||g−τ−1 L◦ϕc◦τK|| ≤ . Proof. By Corollary 1, there exists t2such that ˜m(Sdt2˜ L,τL)(Y)≤ . Next, by Theorem 3, there exists t1 and ˜ϕ such that ˜ϕ: Sdt1˜ K→Sdt2˜ L is a simplicial approximation of τ˜ L◦g◦τ−1 ˜ K . Taking into account that |Sdt1˜ K|=|˜ K| and |Sdt2˜ L|=|˜ L| , we can define the 8 APREPRINT - JULY 29, 2019 simplicial map ˜ϕc:|˜ K|→|˜ L| induced by ˜ϕ . Finally, we can define ϕc=τ−1 ˜ L◦˜ϕc◦τ˜ K , K= Sdt1˜ K , and L= Sdt2˜ L . Since ˜m(L,τL)(Y)≤then ||g−τ−1 L◦ϕc◦τK|| ≤  X Y |˜ K| |˜ L| ˜ K˜ L Sdt1˜ K=K Sdt2˜ L=L τ˜ K g τ˜ L ˜ϕc Sd Sd ˜ϕ Theorem 6. Given a continuous function g:X→Y and two triangulations (K, τK) and (L, τL) of X and Y , respectively. A two-hidden-layer feedforward network N that approximates g arbitrarily close can be explicitly defined. Proof. By Proposition 3, there exists a simplicial approximation ϕc of g that approximates g arbitrarily close. Besides, by Theorem 4, there exists Nsuch that N=ϕcin all the domain. Therefore, Napproximates garbitrarily close. 6 Conclusions In this paper, we provide an effective method for building a multilayer feedforward network which approximates a continuous function between triangulable spaces. The main contribution of the paper is that the weights can be exactly and effectively computed without any training process. On the disadvantages, two can be considered from a technical point of view: Firstly, the homeomorfisms between the triangulable spaces and the simplicial complexes can be hard to find, and secondly, the classic theorem for approximation of neural networks is valid for compact sets, and our result is only valid for triangulable sets. Nonetheless, as pointed above, most of the real-world problems are covered by our result and, therefore the approximation to the continuous function via neural network can be effectively built. References [1] George Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics of Control Signal and Systems, 2(4):303–314, 1989. [2] Kurt Hornik. Approximation capabilities of multilayer feedforward networks. Neural Networks, 4(2):251–257, March 1991. [3] Shiyu Liang and R. Srikant. Why deep neural networks? CoRR, abs/1610.04161, 2016. [4] Itay Safran and Ohad Shamir. Depth-width tradeoffs in approximating natural functions with neural networks. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 2979–2987, International Convention Centre, Sydney, Australia, 06–11 Aug 2017. PMLR. [5] Matus Telgarsky. Benefits of depth in neural networks. CoRR, abs/1602.04485, 2016. [6] Shizhao Sun, Wei Chen, Liwei Wang, Xiaoguang Liu, and Tie-Yan Liu. On the depth of deep neural networks: A theoretical view. In Dale Schuurmans and Michael P. Wellman, editors, Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, February 12-17, 2016, Phoenix, Arizona, USA., pages 2066–2072. AAAI Press, 2016. [7] Boris Hanin and Mark Sellke. Approximating continuous functions by relu nets of minimal width, 2017. [8] Zhou Lu, Hongming Pu, Feicheng Wang, Zhiqiang Hu, and Liwei Wang. The expressive power of neural networks: A view from the width. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pages 6232–6240, 2017. 9