(************** Content-type: application/mathematica ************** CreatedBy='Mathematica 5.0' Mathematica-Compatible Notebook This notebook can be used with any Mathematica-compatible application, such as Mathematica, MathReader or Publicon. The data for the notebook starts with the line containing stars above. To get the notebook into a Mathematica-compatible application, do one of the following: * Save the data starting with the line of stars above into a file with a name ending in .nb, then open the file inside the application; * Copy the data starting with the line of stars above to the clipboard, then use the Paste menu command inside the application. Data for notebooks contains only printable 7-bit ASCII and can be sent directly in email or through ftp in text mode. Newlines can be CR, LF or CRLF (Unix, Macintosh or MS-DOS style). NOTE: If you modify the data for this notebook not in a Mathematica- compatible application, you must delete the line below containing the word CacheID, otherwise Mathematica-compatible applications may try to use invalid cache data. For more information on notebooks and Mathematica-compatible applications, contact Wolfram Research: web: http://www.wolfram.com email: info@wolfram.com phone: +1-217-398-0700 (U.S.) Notebook reader applications are available free of charge from Wolfram Research. *******************************************************************) (*CacheID: 232*) (*NotebookFileLineBreakTest NotebookFileLineBreakTest*) (*NotebookOptionsPosition[ 74052, 2339]*) (*NotebookOutlinePosition[ 79897, 2497]*) (* CellTagsIndexPosition[ 78553, 2459]*) (*WindowFrame->Normal*) Notebook[{ Cell[CellGroupData[{ Cell["Notes", "Section 1"], Cell[CellGroupData[{ Cell["Editorial Changes", "Subsection"], Cell["American spelling changed to British spelling.", "Text"], Cell[TextData[{ Cell[BoxData[ FormBox[ ButtonBox["TYPO", ButtonData:>"Ed:Change3", Active->True, ButtonStyle->"Hyperlink"], TextForm]]], "\"", Cell[BoxData[ \(TraditionalForm\`1/n\)]], "\" changed to \"", Cell[BoxData[ \(TraditionalForm\`1\/n\)]], "\"" }], "Text"], Cell[TextData[{ Cell[BoxData[ FormBox[ ButtonBox["TYPO", ButtonData:>"Ed:Change1", Active->True, ButtonStyle->"Hyperlink"], TextForm]]], "\"(b) The basic two-layer network.\" changed to \"(a) The basic two-layer \ network.\"" }], "Text"], Cell[TextData[{ Cell[BoxData[ FormBox[ ButtonBox["TYPO", ButtonData:>"Ed:Change2", Active->True, ButtonStyle->"Hyperlink"], TextForm]]], "\"equations (8) and (9)\" changed to \"equation (8) and equation (9)\"\n", StyleBox["This isn't really a typo", FontColor->RGBColor[1, 0, 0]] }], "Text"] }, Closed]] }, Closed]], Cell[CellGroupData[{ Cell["\<\ A discrete firing event analysis of the adaptive cluster expansion network\ \>", "Title"], Cell["\<\ S P Luttrell Defence Research Agency, St Andrews Road, Malvern, Worcs WR14 3PS, UK E-mail: luttrell@signal.dra.hmg.gb.\ \>", "Author"], Cell["\<\ This paper appeared in Network: Computation in Neural Systems, 1997, vol. 7, \ pp. 285-290. Received 31 January 1996\ \>", "Text"], Cell["\<\ This paper was presented at the Workshop on Information Theory and the Brain, \ held at the University of Stirling, UK, on 4-5 September 1995.\ \>", "Text"], Cell["\[Copyright] 1996 British Crown Copyright/DRA", "Text"], Cell[TextData[{ StyleBox["Abstract.", FontWeight->"Bold"], " This paper describes how a hierarchical network for encoding sensor data \ (the adaptive cluster expansion network) can be constructed by linking \ together a number of elementary modules, each of which is a simple two-layer \ encoder/decoder network. To achieve this goal, a Bayesian analysis is applied \ to the discrete neural firing events that occur within each layer of the \ network." }], "Abstract"], Cell[CellGroupData[{ Cell[TextData[{ CounterBox["Section"], ". Introduction" }], "Section 1"], Cell[TextData[{ "The overall goal of this paper is to relate two pre-existing ideas: (i) \ the discrete neural firing event analysis of a two-layer self-organising \ network [", ButtonBox["14", ButtonData:>"Ref:Luttrell1996a", ButtonStyle->"Hyperlink"], ", ", ButtonBox["15", ButtonData:>"Ref:Luttrell1996b", ButtonStyle->"Hyperlink"], "], and (ii) the adaptive cluster expansion (ACE) approach to stepwise \ encoding of high-dimensional vectors [", ButtonBox["6", ButtonData:>"Ref:Luttrell1989a", ButtonStyle->"Hyperlink"], ", ", ButtonBox["7", ButtonData:>"Ref:Luttrell1989b", ButtonStyle->"Hyperlink"], "]." }], "Text"], Cell[TextData[Cell[TextData[{ " ", ButtonBox["OPEN", ButtonData:>{ URL[ "http://www.luttrell.org.uk/papers/itbrain/fig1.gif"], None}, Active->True, ButtonStyle->"Hyperlink"], " " }]]], "NumberedFigure", TextAlignment->Center, CellTags->"Fig:2Layer"], Cell["\<\ (a) The basic two-layer network with a feedforward recognition model and a \ feedback generative model. (b) How the recognition model is approximated in \ practice as a superposition of contributions from the firing events in the \ output layer.\ \>", "Caption"], Cell[TextData[{ ButtonBox["Figure", ButtonData:>"Fig:2Layer", ButtonStyle->"Hyperlink"], " ", CounterBox["NumberedFigure", "Fig:2Layer"], "(a) shows the basic structure of a two-layer self-organising network, in \ which there is both a feedforward recognition model and a feedback generative \ model (the terminology is borrowed from [", ButtonBox["1", ButtonData:>"Ref:BeckerHinton1992", ButtonStyle->"Hyperlink"], ", ", ButtonBox["2", ButtonData:>"Ref:DayanHintonNealZemel1995", ButtonStyle->"Hyperlink"], "]). The pattern of activity in the input layer is processed by the \ recognition model to cause neurons in the output layer to fire. For instance, \ if the recognition model were implemented using weighted connections feeding \ into sigmoid nonlinearities, then a two-layer perceptron network would \ result. However, unlike in the perceptron, the pattern of activity in the \ output layer is then processed by the generative model to try to reconstruct \ the pattern of activity in the input layer as accurately as possible [", ButtonBox["12", ButtonData:>"Ref:Luttrell1994a", ButtonStyle->"Hyperlink"], "]. The overall goal of optimising this type of network is to adjust the \ feedforward and feedback processing so that on average the reconstruction \ error is as small as possible." }], "Text"], Cell[TextData[{ "The optimisation of such two-layer networks will be described in ", ButtonBox["section", ButtonData:>"Sect:Optimisation2LayerNetwork", ButtonStyle->"Hyperlink"], " ", CounterBox["Section", "Sect:Optimisation2LayerNetwork"], ", and the extension to coupled two-layer networks will be presented in ", ButtonBox["section", ButtonData:>"Sect:OptimisationCoupled2LayerNetworks", ButtonStyle->"Hyperlink"], " ", CounterBox["Section", "Sect:OptimisationCoupled2LayerNetworks"], ". Finally, a description of how these techniques may be applied to the ACE \ approach will be given in ", ButtonBox["section", ButtonData:>"Sect:AdaptiveClusterExpansion", ButtonStyle->"Hyperlink"], " ", CounterBox["Section", "Sect:AdaptiveClusterExpansion"], "." }], "Text"] }, Closed]], Cell[CellGroupData[{ Cell[TextData[{ CounterBox["Section"], ". Optimisation of a two-layer network" }], "Section", CellTags->"Sect:Optimisation2LayerNetwork"], Cell[TextData[{ "In order to make contact with existing results it is necessary to assume \ that only the location of a single firing event in the output layer, rather \ than the full pattern of activity, is used by the generative model to \ reconstruct the pattern of activity in the input layer. Thus define the \ following notation: ", Cell[BoxData[ FormBox[ StyleBox["x", FontWeight->"Bold"], TraditionalForm]]], " = pattern of activity in the input layer (which may be derived from the \ output firing events occurring in another network), ", Cell[BoxData[ FormBox[ StyleBox["y", FontWeight->"Bold"], TraditionalForm]]], " = location of the firing event in the output layer (which is therefore a \ lattice vector), ", Cell[BoxData[ FormBox[ RowBox[{"Pr", "(", StyleBox["x", FontWeight->"Bold"], ")"}], TraditionalForm]]], " = probability that input activity ", Cell[BoxData[ FormBox[ StyleBox["x", FontWeight->"Bold"], TraditionalForm]]], " occurs, ", Cell[BoxData[ FormBox[ RowBox[{"Pr", "(", RowBox[{ StyleBox["y", FontWeight->"Bold"], "|", StyleBox["x", FontWeight->"Bold"]}], ")"}], TraditionalForm]]], " = probability that the firing event occurs at location ", Cell[BoxData[ FormBox[ StyleBox["y", FontWeight->"Bold"], TraditionalForm]]], " given that the input activity ", Cell[BoxData[ FormBox[ StyleBox["x", FontWeight->"Bold"], TraditionalForm]]], " is known (recognition model), ", Cell[BoxData[ FormBox[ RowBox[{"Pr", "(", RowBox[{ StyleBox["x", FontWeight->"Bold"], "|", StyleBox["y", FontWeight->"Bold"]}], ")"}], TraditionalForm]]], " = probability that the input activity is ", Cell[BoxData[ FormBox[ StyleBox["x", FontWeight->"Bold"], TraditionalForm]]], " given that the location ", Cell[BoxData[ FormBox[ StyleBox["y", FontWeight->"Bold"], TraditionalForm]]], " of the firing event is known (generative model). Bayes' theorem relates \ these probabilities thus:" }], "Text"], Cell[BoxData[ FormBox[ RowBox[{ RowBox[{"Pr", "(", RowBox[{ StyleBox["x", FontWeight->"Bold"], "|", StyleBox["y", FontWeight->"Bold"]}], ")"}], "=", RowBox[{ FractionBox[ RowBox[{ RowBox[{"Pr", "(", RowBox[{ StyleBox["y", FontWeight->"Bold"], "|", StyleBox["x", FontWeight->"Bold"]}], ")"}], RowBox[{"Pr", "(", StyleBox["x", FontWeight->"Bold"], ")"}]}], RowBox[{"\[Integral]", RowBox[{ SuperscriptBox[ StyleBox[ RowBox[{ StyleBox["d", FontSlant->"Plain"], StyleBox["x", FontWeight->"Bold"]}]], "\[Prime]"], RowBox[{"Pr", "(", RowBox[{ StyleBox["y", FontWeight->"Bold"], "|", StyleBox[ SuperscriptBox[ StyleBox["x", FontWeight->"Bold"], StyleBox["\[Prime]", FontWeight->"Plain"]], FontWeight->"Bold"]}], ")"}], RowBox[{"Pr", "(", StyleBox[ SuperscriptBox["x", StyleBox["\[Prime]", FontWeight->"Plain"]], FontWeight->"Bold"], ")"}]}]}]], "."}]}], TraditionalForm]], "Text"], Cell[TextData[{ "A suitable optimisation principle is to minimise the average Euclidean \ reconstruction error ", Cell[BoxData[ \(TraditionalForm\`D\)]], " defined as [", ButtonBox["12", ButtonData:>"Ref:Luttrell1994a", ButtonStyle->"Hyperlink"], ", ", ButtonBox["14", ButtonData:>"Ref:Luttrell1996a", ButtonStyle->"Hyperlink"], ", ", ButtonBox["15", ButtonData:>"Ref:Luttrell1996b", ButtonStyle->"Hyperlink"], "]" }], "Text"], Cell[BoxData[ FormBox[ RowBox[{"D", "\[Congruent]", RowBox[{"\[Integral]", RowBox[{ StyleBox[ RowBox[{ StyleBox["d", FontSlant->"Plain"], StyleBox["x", FontWeight->"Bold"]}]], " ", SuperscriptBox[ StyleBox[ RowBox[{ StyleBox["d", FontSlant->"Plain"], StyleBox["x", FontWeight->"Bold"]}]], "\[Prime]"], RowBox[{"Pr", "(", StyleBox["x", FontWeight->"Bold"], ")"}], RowBox[{ UnderscriptBox["\[Sum]", StyleBox["y", FontWeight->"Bold"]], RowBox[{ RowBox[{"Pr", "(", RowBox[{ StyleBox["y", FontWeight->"Bold"], "|", StyleBox["x", FontWeight->"Bold"]}], ")"}], RowBox[{"Pr", "(", RowBox[{ StyleBox[ SuperscriptBox["x", StyleBox["\[Prime]", FontWeight->"Plain"]], FontWeight->"Bold"], "|", StyleBox["y", FontWeight->"Bold"]}], ")"}], SuperscriptBox[ RowBox[{"\[LeftDoubleBracketingBar]", RowBox[{ StyleBox["x", FontWeight->"Bold"], "-", SuperscriptBox[ StyleBox["x", FontWeight->"Bold"], "\[Prime]"]}], "\[RightDoubleBracketingBar]"}], "2"]}]}]}]}]}], TraditionalForm]], "NumberedEquation", CellTags->"Eq:Objective"], Cell[TextData[{ "where the average is performed over all ", Cell[BoxData[ FormBox[ StyleBox["x", FontWeight->"Bold"], TraditionalForm]]], ", ", Cell[BoxData[ FormBox[ SuperscriptBox[ StyleBox["x", FontWeight->"Bold"], "\[Prime]"], TraditionalForm]]], " and ", Cell[BoxData[ FormBox[ StyleBox["y", FontWeight->"Bold"], TraditionalForm]]], " under the joint probability ", Cell[BoxData[ FormBox[ RowBox[{ RowBox[{"Pr", " ", RowBox[{"(", RowBox[{ StyleBox["x", FontWeight->"Bold"], ",", SuperscriptBox[ StyleBox["x", FontWeight->"Bold"], "\[Prime]"], ",", StyleBox["y", FontWeight->"Bold"]}], ")"}]}], "=", RowBox[{"Pr", " ", RowBox[{"(", StyleBox["x", FontWeight->"Bold"], ")"}], " ", "Pr", " ", RowBox[{"(", RowBox[{ StyleBox["y", FontWeight->"Bold"], "|", StyleBox["x", FontWeight->"Bold"]}], ")"}], " ", RowBox[{"Pr", "(", RowBox[{ StyleBox[ SuperscriptBox["x", StyleBox["\[Prime]", FontWeight->"Plain"]], FontWeight->"Bold"], "|", StyleBox["y", FontWeight->"Bold"]}], ")"}]}]}], TraditionalForm]]], ". The Euclidean distance ", Cell[BoxData[ FormBox[ SuperscriptBox[ RowBox[{"\[LeftDoubleBracketingBar]", RowBox[{ StyleBox["x", FontWeight->"Bold"], "-", SuperscriptBox[ StyleBox["x", FontWeight->"Bold"], "\[Prime]"]}], "\[RightDoubleBracketingBar]"}], "2"], TraditionalForm]]], " can be expanded out as ", Cell[BoxData[ FormBox[ RowBox[{ RowBox[{ StyleBox["x", FontWeight->"Bold"], "\[CenterDot]", StyleBox["x", FontWeight->"Bold"]}], "+", RowBox[{ SuperscriptBox[ StyleBox["x", FontWeight->"Bold"], "\[Prime]"], "\[CenterDot]", SuperscriptBox[ StyleBox["x", FontWeight->"Bold"], "\[Prime]"]}], "-", RowBox[{"2", RowBox[{ StyleBox["x", FontWeight->"Bold"], "\[CenterDot]", SuperscriptBox[ StyleBox["x", FontWeight->"Bold"], "\[Prime]"]}]}]}], TraditionalForm]]], ", and the symmetry of the expression for ", Cell[BoxData[ \(TraditionalForm\`D\)]], " can be used to integrate out ", Cell[BoxData[ FormBox[ SuperscriptBox[ StyleBox["x", FontWeight->"Bold"], "\[Prime]"], TraditionalForm]]], " to yield the result" }], "Text"], Cell[BoxData[ FormBox[ RowBox[{"D", "=", RowBox[{"\[Integral]", RowBox[{ StyleBox[ RowBox[{ StyleBox["d", FontSlant->"Plain"], StyleBox["x", FontWeight->"Bold"]}]], " ", RowBox[{"Pr", "(", StyleBox["x", FontWeight->"Bold"], ")"}], RowBox[{"D", "(", StyleBox["x", FontWeight->"Bold"], StyleBox[")", FontWeight->"Plain"]}]}]}]}], TraditionalForm]], "NumberedEquation", CellTags->"Eq:ObjectiveSimple"], Cell[TextData[{ "where ", Cell[BoxData[ FormBox[ RowBox[{"D", "(", StyleBox["x", FontWeight->"Bold"], StyleBox[")", FontWeight->"Plain"]}], TraditionalForm]]], " is defined as ", Cell[BoxData[ FormBox[ RowBox[{ RowBox[{"D", "(", StyleBox["x", FontWeight->"Bold"], StyleBox[")", FontWeight->"Plain"]}], StyleBox["\[Congruent]", FontWeight->"Plain"], RowBox[{ StyleBox["2", FontWeight->"Plain"], RowBox[{ UnderscriptBox["\[Sum]", StyleBox["y", FontWeight->"Bold"]], RowBox[{ RowBox[{"Pr", "(", RowBox[{ StyleBox["y", FontWeight->"Bold"], "|", StyleBox["x", FontWeight->"Bold"]}], ")"}], SuperscriptBox[ RowBox[{"\[LeftDoubleBracketingBar]", RowBox[{ StyleBox["x", FontWeight->"Bold"], "-", RowBox[{ SuperscriptBox[ StyleBox["x", FontWeight->"Bold"], "\[Prime]"], "(", StyleBox["y", FontWeight->"Bold"], ")"}]}], "\[RightDoubleBracketingBar]"}], "2"]}]}]}]}], TraditionalForm]]], ", and ", Cell[BoxData[ FormBox[ RowBox[{ SuperscriptBox[ StyleBox["x", FontWeight->"Bold"], "\[Prime]"], "(", StyleBox["y", FontWeight->"Bold"], ")"}], TraditionalForm]]], " is defined as ", Cell[BoxData[ FormBox[ RowBox[{ RowBox[{ SuperscriptBox[ StyleBox["x", FontWeight->"Bold"], "\[Prime]"], "(", StyleBox["y", FontWeight->"Bold"], ")"}], "\[Congruent]", RowBox[{"\[Integral]", RowBox[{ StyleBox[ RowBox[{ StyleBox["d", FontSlant->"Plain"], StyleBox["x", FontWeight->"Bold"]}]], StyleBox[" ", FontWeight->"Plain"], RowBox[{"Pr", "(", RowBox[{ StyleBox["x", FontWeight->"Bold"], "|", StyleBox["y", FontWeight->"Bold"]}], ")"}], StyleBox["x", FontWeight->"Bold"]}]}]}], TraditionalForm]]], ". Conveniently, the stationary point of ", Cell[BoxData[ \(TraditionalForm\`D\)]], " with respect to variation of ", Cell[BoxData[ FormBox[ RowBox[{ SuperscriptBox[ StyleBox["x", FontWeight->"Bold"], "\[Prime]"], "(", StyleBox["y", FontWeight->"Bold"], ")"}], TraditionalForm]]], " is located at ", Cell[BoxData[ FormBox[ RowBox[{ RowBox[{ SuperscriptBox[ StyleBox["x", FontWeight->"Bold"], "\[Prime]"], "(", StyleBox["y", FontWeight->"Bold"], ")"}], "=", RowBox[{"\[Integral]", RowBox[{ StyleBox[ RowBox[{ StyleBox["d", FontSlant->"Plain"], StyleBox["x", FontWeight->"Bold"]}]], StyleBox[" ", FontWeight->"Plain"], RowBox[{"Pr", "(", RowBox[{ StyleBox["x", FontWeight->"Bold"], "|", StyleBox["y", FontWeight->"Bold"]}], ")"}], StyleBox["x", FontWeight->"Bold"]}]}]}], TraditionalForm]]], ", so the ", Cell[BoxData[ FormBox[ RowBox[{ SuperscriptBox[ StyleBox["x", FontWeight->"Bold"], "\[Prime]"], "(", StyleBox["y", FontWeight->"Bold"], ")"}], TraditionalForm]]], " can be treated as a set of independent parameters whose values must be \ chosen by minimising ", Cell[BoxData[ \(TraditionalForm\`D\)]], ". In ", ButtonBox["equation", ButtonData:>"Eq:ObjectiveSimple", ButtonStyle->"Hyperlink"], " (", CounterBox["NumberedEquation", "Eq:ObjectiveSimple"], ") the network is described primarily in terms of the recognition model, \ unlike in [", ButtonBox["1", ButtonData:>"Ref:BeckerHinton1992", ButtonStyle->"Hyperlink"], ", ", ButtonBox["2", ButtonData:>"Ref:DayanHintonNealZemel1995", ButtonStyle->"Hyperlink"], "] where the generative model is primary. If ", Cell[BoxData[ FormBox[ RowBox[{"Pr", "(", RowBox[{ StyleBox["y", FontWeight->"Bold"], "|", StyleBox["x", FontWeight->"Bold"]}], ")"}], TraditionalForm]]], " is modelled as" }], "Text"], Cell[BoxData[ FormBox[ RowBox[{ RowBox[{"Pr", "(", RowBox[{ StyleBox["y", FontWeight->"Bold"], "|", StyleBox["x", FontWeight->"Bold"]}], ")"}], "=", FractionBox[ RowBox[{"N", "(", RowBox[{ RowBox[{ StyleBox["m", FontWeight->"Bold"], StyleBox["(", FontWeight->"Plain"], StyleBox["y", FontWeight->"Bold"], StyleBox[")", FontWeight->"Plain"]}], StyleBox[",", FontWeight->"Plain"], RowBox[{ StyleBox["\[CapitalSigma]", FontWeight->"Bold"], StyleBox["(", FontWeight->"Plain"], StyleBox["y", FontWeight->"Bold"], StyleBox[")", FontWeight->"Plain"]}]}], ")"}], RowBox[{ SubscriptBox["\[Sum]", SuperscriptBox[ StyleBox["y", FontWeight->"Bold"], "\[Prime]"]], RowBox[{"N", "(", RowBox[{ RowBox[{ StyleBox["m", FontWeight->"Bold"], StyleBox["(", FontWeight->"Plain"], StyleBox[ SuperscriptBox["y", StyleBox["\[Prime]", FontWeight->"Plain"]], FontWeight->"Bold"], StyleBox[")", FontWeight->"Plain"]}], StyleBox[",", FontWeight->"Plain"], RowBox[{ StyleBox["\[CapitalSigma]", FontWeight->"Bold"], StyleBox["(", FontWeight->"Plain"], StyleBox[ SuperscriptBox["y", StyleBox["\[Prime]", FontWeight->"Plain"]], FontWeight->"Bold"], StyleBox[")", FontWeight->"Plain"]}]}], ")"}]}]]}], TraditionalForm]], "Text"], Cell[TextData[{ "where ", Cell[BoxData[ FormBox[ RowBox[{"N", "(", RowBox[{ StyleBox["m", FontWeight->"Bold"], StyleBox[",", FontWeight->"Plain"], StyleBox["\[CapitalSigma]", FontWeight->"Bold"]}], ")"}], TraditionalForm]]], " is a Gaussian probability distribution in ", Cell[BoxData[ FormBox[ StyleBox["x", FontWeight->"Bold"], TraditionalForm]]], "-space with mean ", Cell[BoxData[ FormBox[ StyleBox["m", FontWeight->"Bold"], TraditionalForm]]], " and covariance ", Cell[BoxData[ \(TraditionalForm\`\[CapitalSigma]\)], FontWeight->"Bold"], ", then minimising ", Cell[BoxData[ \(TraditionalForm\`D\)]], " in ", ButtonBox["equation", ButtonData:>"Eq:ObjectiveSimple", ButtonStyle->"Hyperlink"], " (", CounterBox["NumberedEquation", "Eq:ObjectiveSimple"], ") with respect to the ", Cell[BoxData[ FormBox[ RowBox[{ StyleBox["m", FontWeight->"Bold"], StyleBox["(", FontWeight->"Plain"], StyleBox["y", FontWeight->"Bold"], StyleBox[")", FontWeight->"Plain"]}], TraditionalForm]]], " and ", Cell[BoxData[ FormBox[ RowBox[{ StyleBox["\[CapitalSigma]", FontWeight->"Bold"], StyleBox["(", FontWeight->"Plain"], StyleBox["y", FontWeight->"Bold"], StyleBox[")", FontWeight->"Plain"]}], TraditionalForm]]], " yields ", Cell[BoxData[ FormBox[ RowBox[{"D", "=", RowBox[{"\[Integral]", RowBox[{ StyleBox[ RowBox[{ StyleBox["d", FontSlant->"Plain"], StyleBox["x", FontWeight->"Bold"]}]], " ", RowBox[{"Pr", "(", StyleBox["x", FontWeight->"Bold"], ")"}], SuperscriptBox[ RowBox[{"\[LeftDoubleBracketingBar]", RowBox[{ StyleBox["x", FontWeight->"Bold"], "-", RowBox[{ SuperscriptBox[ StyleBox["x", FontWeight->"Bold"], "\[Prime]"], "(", RowBox[{ StyleBox["y", FontWeight->"Bold"], "(", StyleBox["x", FontWeight->"Bold"], ")"}], ")"}]}], "\[RightDoubleBracketingBar]"}], "2"]}]}]}], TraditionalForm]]], " where ", Cell[BoxData[ FormBox[ RowBox[{ RowBox[{ StyleBox["y", FontWeight->"Bold"], "(", StyleBox["x", FontWeight->"Bold"], ")"}], "\[Congruent]", RowBox[{"arg", " ", RowBox[{ SubscriptBox["min", StyleBox["y", FontWeight->"Bold"]], SuperscriptBox[ RowBox[{"\[LeftDoubleBracketingBar]", RowBox[{ StyleBox["x", FontWeight->"Bold"], "-", RowBox[{ SuperscriptBox[ StyleBox["x", FontWeight->"Bold"], "\[Prime]"], "(", StyleBox["y", FontWeight->"Bold"], ")"}]}], "\[RightDoubleBracketingBar]"}], "2"]}]}]}], TraditionalForm]]], " [", ButtonBox["12", ButtonData:>"Ref:Luttrell1994a", ButtonStyle->"Hyperlink"], "], which is the standard method of optimising the code vectors ", Cell[BoxData[ FormBox[ RowBox[{ SuperscriptBox[ StyleBox["x", FontWeight->"Bold"], "\[Prime]"], "(", StyleBox["y", FontWeight->"Bold"], ")"}], TraditionalForm]]], " in a vector quantiser using a nearest-neighbour encoding rule ", Cell[BoxData[ FormBox[ RowBox[{ StyleBox["y", FontWeight->"Bold"], "(", StyleBox["x", FontWeight->"Bold"], ")"}], TraditionalForm]]], " [", ButtonBox["5", ButtonData:>"Ref:LindeBuzoGray1980", ButtonStyle->"Hyperlink"], "]." }], "Text"], Cell["\<\ This result can be generalised somewhat by making the replacement\ \>", "Text"], Cell[BoxData[ FormBox[ RowBox[{ RowBox[{ RowBox[{"Pr", "(", RowBox[{ StyleBox["y", FontWeight->"Bold"], "|", StyleBox["x", FontWeight->"Bold"]}], ")"}], "\[LongRightArrow]", RowBox[{"Pr", "(", RowBox[{ StyleBox[ SuperscriptBox["y", StyleBox["\[Prime]", FontWeight->"Plain"]], FontWeight->"Bold"], "|", StyleBox["x", FontWeight->"Bold"]}], ")"}]}], "=", RowBox[{ UnderscriptBox["\[Sum]", StyleBox["y", FontWeight->"Bold"]], RowBox[{ RowBox[{"Pr", "(", RowBox[{ StyleBox[ SuperscriptBox["y", StyleBox["\[Prime]", FontWeight->"Plain"]], FontWeight->"Bold"], "|", StyleBox["y", FontWeight->"Bold"]}], ")"}], RowBox[{"Pr", "(", RowBox[{ StyleBox["y", FontWeight->"Bold"], "|", StyleBox["x", FontWeight->"Bold"]}], ")"}]}]}]}], TraditionalForm]], "NumberedEquation", CellTags->"Eq:Leakage"], Cell[TextData[{ "where the ", Cell[BoxData[ FormBox[ RowBox[{"Pr", "(", RowBox[{ StyleBox["y", FontWeight->"Bold"], "|", StyleBox["x", FontWeight->"Bold"]}], ")"}], TraditionalForm]]], " are now filtered through a matrix of conditional probabilities ", Cell[BoxData[ FormBox[ RowBox[{"Pr", "(", RowBox[{ StyleBox[ SuperscriptBox["y", StyleBox["\[Prime]", FontWeight->"Plain"]], FontWeight->"Bold"], "|", StyleBox["y", FontWeight->"Bold"]}], ")"}], TraditionalForm]]], ". The effect of ", Cell[BoxData[ FormBox[ RowBox[{"Pr", "(", RowBox[{ StyleBox[ SuperscriptBox["y", StyleBox["\[Prime]", FontWeight->"Plain"]], FontWeight->"Bold"], "|", StyleBox["y", FontWeight->"Bold"]}], ")"}], TraditionalForm]]], " is to cause the neuron firing probabilities to 'leak' onto other neurons, \ which dilutes the information about precisely which neurons were supposed to \ fire in the first place. When ", Cell[BoxData[ \(TraditionalForm\`D\)]], " is minimised it tries to limit the damage that is caused by this \ probability leakage. It achieves this by ensuring that neurons that are \ connected by the matrix ", Cell[BoxData[ FormBox[ RowBox[{"Pr", "(", RowBox[{ StyleBox[ SuperscriptBox["y", StyleBox["\[Prime]", FontWeight->"Plain"]], FontWeight->"Bold"], "|", StyleBox["y", FontWeight->"Bold"]}], ")"}], TraditionalForm]]], " have similar properties, so their parameters become topographically \ ordered. This type of self-organising map is similar to the well known \ Kohonen map [", ButtonBox["4", ButtonData:>"Ref:Kohonen1984", ButtonStyle->"Hyperlink"], "] with neighbourhood function ", Cell[BoxData[ FormBox[ RowBox[{"Pr", "(", RowBox[{ StyleBox[ SuperscriptBox["y", StyleBox["\[Prime]", FontWeight->"Plain"]], FontWeight->"Bold"], "|", StyleBox["y", FontWeight->"Bold"]}], ")"}], TraditionalForm]]], " [", ButtonBox["7", ButtonData:>"Ref:Luttrell1989b", ButtonStyle->"Hyperlink"], ", ", ButtonBox["8", ButtonData:>"Ref:Luttrell1990", ButtonStyle->"Hyperlink"], "]." }], "Text"], Cell[TextData[{ "The result can readily be generalised to ", Cell[BoxData[ \(TraditionalForm\`n\)]], " firing events that are statistically independent given that the input ", Cell[BoxData[ FormBox[ StyleBox["x", FontWeight->"Bold"], TraditionalForm]]], " is known [", ButtonBox["14", ButtonData:>"Ref:Luttrell1996a", ButtonStyle->"Hyperlink"], ", ", ButtonBox["15", ButtonData:>"Ref:Luttrell1996b", ButtonStyle->"Hyperlink"], "]. ", Cell[BoxData[ FormBox[ RowBox[{"D", "(", StyleBox["x", FontWeight->"Bold"], StyleBox[")", FontWeight->"Plain"]}], TraditionalForm]]], " in ", ButtonBox["equation", ButtonData:>"Eq:ObjectiveSimple", ButtonStyle->"Hyperlink"], " (", CounterBox["NumberedEquation", "Eq:ObjectiveSimple"], ") then becomes" }], "Text"], Cell[BoxData[ FormBox[ RowBox[{ RowBox[{"D", "(", StyleBox["x", FontWeight->"Bold"], StyleBox[")", FontWeight->"Plain"]}], "=", RowBox[{ StyleBox["2", FontWeight->"Plain"], RowBox[{ UnderscriptBox["\[Sum]", RowBox[{ SubscriptBox[ StyleBox["y", FontWeight->"Bold"], "1"], ",", SubscriptBox[ StyleBox["y", FontWeight->"Bold"], "2"], ",", "\[CenterEllipsis]", ",", SubscriptBox[ StyleBox["y", FontWeight->"Bold"], "n"]}]], RowBox[{ RowBox[{"Pr", "(", RowBox[{ SubscriptBox[ StyleBox["y", FontWeight->"Bold"], "1"], "|", StyleBox["x", FontWeight->"Bold"]}], ")"}], RowBox[{"Pr", "(", RowBox[{ SubscriptBox[ StyleBox["y", FontWeight->"Bold"], "2"], "|", StyleBox["x", FontWeight->"Bold"]}], ")"}], RowBox[{"\[CenterEllipsis]Pr", "(", RowBox[{ SubscriptBox[ StyleBox["y", FontWeight->"Bold"], "n"], "|", StyleBox["x", FontWeight->"Bold"]}], ")"}], RowBox[{ SuperscriptBox[ RowBox[{"\[LeftDoubleBracketingBar]", RowBox[{ StyleBox["x", FontWeight->"Bold"], "-", RowBox[{ SuperscriptBox[ StyleBox["x", FontWeight->"Bold"], "\[Prime]"], "(", RowBox[{ StyleBox[ SubscriptBox["y", StyleBox["1", FontWeight->"Plain"]], FontWeight->"Bold"], ",", StyleBox[ SubscriptBox["y", StyleBox["2", FontWeight->"Plain"]], FontWeight->"Bold"], StyleBox[",", FontWeight->"Plain"], StyleBox["\[CenterEllipsis]", FontWeight->"Plain"], StyleBox[",", FontWeight->"Plain"], StyleBox[ SubscriptBox["y", StyleBox["n", FontWeight->"Plain"]], FontWeight->"Bold"]}], ")"}]}], "\[RightDoubleBracketingBar]"}], "2"], "."}]}]}]}]}], TraditionalForm]], "NumberedEquation", CellTags->"Eq:ObjectiveN"], Cell[TextData[{ "If ", Cell[BoxData[ FormBox[ RowBox[{ SuperscriptBox[ StyleBox["x", FontWeight->"Bold"], "\[Prime]"], "(", RowBox[{ StyleBox[ SubscriptBox["y", StyleBox["1", FontWeight->"Plain"]], FontWeight->"Bold"], ",", StyleBox[ SubscriptBox["y", StyleBox["2", FontWeight->"Plain"]], FontWeight->"Bold"], StyleBox[",", FontWeight->"Plain"], StyleBox["\[CenterEllipsis]", FontWeight->"Plain"], StyleBox[",", FontWeight->"Plain"], StyleBox[ SubscriptBox["y", StyleBox["n", FontWeight->"Plain"]], FontWeight->"Bold"]}], ")"}], TraditionalForm]]], " is approximated as ", Cell[BoxData[ FormBox[ RowBox[{ RowBox[{ SuperscriptBox[ StyleBox["x", FontWeight->"Bold"], "\[Prime]"], "(", RowBox[{ StyleBox[ SubscriptBox["y", StyleBox["1", FontWeight->"Plain"]], FontWeight->"Bold"], ",", StyleBox[ SubscriptBox["y", StyleBox["2", FontWeight->"Plain"]], FontWeight->"Bold"], StyleBox[",", FontWeight->"Plain"], StyleBox["\[CenterEllipsis]", FontWeight->"Plain"], StyleBox[",", FontWeight->"Plain"], StyleBox[ SubscriptBox["y", StyleBox["n", FontWeight->"Plain"]], FontWeight->"Bold"]}], ")"}], "\[TildeTilde]", RowBox[{\(1\/n\), RowBox[{\(\[Sum]\_\(i = 1\)\%n\), RowBox[{ SuperscriptBox[ StyleBox["x", FontWeight->"Bold"], "\[Prime]"], "(", StyleBox[ SubscriptBox["y", StyleBox["i", FontWeight->"Plain"]], FontWeight->"Bold"], ")"}]}]}]}], TraditionalForm]]], ", as shown in ", ButtonBox["figure", ButtonData:>"Fig:2Layer", ButtonStyle->"Hyperlink"], " ", CounterBox["NumberedFigure", "Fig:2Layer"], "(b), then ", Cell[BoxData[ \(TraditionalForm\`D\)]], " has an easily computable upper bound ", Cell[BoxData[ FormBox[ SubscriptBox["D", StyleBox["ub", FontSlant->"Italic"]], TraditionalForm]]], " [", ButtonBox["14", ButtonData:>"Ref:Luttrell1996a", ButtonStyle->"Hyperlink"], ", ", ButtonBox["15", ButtonData:>"Ref:Luttrell1996b", ButtonStyle->"Hyperlink"], "], where ", Cell[BoxData[ FormBox[ RowBox[{ RowBox[{"D", "\[LessEqual]", SubscriptBox["D", StyleBox["ub", FontSlant->"Italic"]]}], "=", \(D\_1 + D\_2\)}], TraditionalForm]]], ", with ", Cell[BoxData[ \(TraditionalForm\`D\_1\)]], " and ", Cell[BoxData[ \(TraditionalForm\`D\_2\)]], " defined as" }], "Text"], Cell[BoxData[{ FormBox[ RowBox[{\(D\_1\), "\[Congruent]", "\[AlignmentMarker]", RowBox[{\(2\/n\), RowBox[{"\[Integral]", RowBox[{ StyleBox[ RowBox[{ StyleBox["d", FontSlant->"Plain"], StyleBox["x", FontWeight->"Bold"]}]], " ", RowBox[{"Pr", "(", StyleBox["x", FontWeight->"Bold"], ")"}], RowBox[{ UnderscriptBox["\[Sum]", StyleBox["y", FontWeight->"Bold"]], RowBox[{ RowBox[{"Pr", "(", RowBox[{ StyleBox["y", FontWeight->"Bold"], "|", StyleBox["x", FontWeight->"Bold"]}], ")"}], SuperscriptBox[ RowBox[{"\[LeftDoubleBracketingBar]", RowBox[{ StyleBox["x", FontWeight->"Bold"], "-", RowBox[{ SuperscriptBox[ StyleBox["x", FontWeight->"Bold"], "\[Prime]"], "(", StyleBox["y", FontWeight->"Bold"], ")"}]}], "\[RightDoubleBracketingBar]"}], "2"]}]}]}]}]}]}], TraditionalForm], "\n", FormBox[ RowBox[{\(D\_2\), "\[Congruent]", "\[AlignmentMarker]", RowBox[{\(\(2 \((n - 1)\)\)\/n\), RowBox[{"\[Integral]", RowBox[{ StyleBox[ RowBox[{ StyleBox["d", FontSlant->"Plain"], StyleBox["x", FontWeight->"Bold"]}]], " ", RowBox[{"Pr", "(", StyleBox["x", FontWeight->"Bold"], ")"}], RowBox[{ SuperscriptBox[ RowBox[{"\[LeftDoubleBracketingBar]", RowBox[{ StyleBox["x", FontWeight->"Bold"], "-", RowBox[{ UnderscriptBox["\[Sum]", StyleBox["y", FontWeight->"Bold"]], RowBox[{ RowBox[{"Pr", "(", RowBox[{ StyleBox["y", FontWeight->"Bold"], "|", StyleBox["x", FontWeight->"Bold"]}], ")"}], RowBox[{ SuperscriptBox[ StyleBox["x", FontWeight->"Bold"], "\[Prime]"], "(", StyleBox["y", FontWeight->"Bold"], ")"}]}]}]}], "\[RightDoubleBracketingBar]"}], "2"], "."}]}]}]}]}], TraditionalForm]}], "NumberedEquation", TextAlignment->AlignmentMarker, CellTags->"Eq:ObjectiveUpperBound"], Cell[TextData[{ "This result for ", Cell[BoxData[ \(TraditionalForm\`n\)]], " firing events should be compared with the result for one firing event \ given in ", ButtonBox["equation", ButtonData:>"Eq:ObjectiveSimple", ButtonStyle->"Hyperlink"], " (", CounterBox["NumberedEquation", "Eq:ObjectiveSimple"], "). In effect, ", ButtonBox["equation", ButtonData:>"Eq:ObjectiveSimple", ButtonStyle->"Hyperlink"], " (", CounterBox["NumberedEquation", "Eq:ObjectiveSimple"], ") has split into two terms: the ", Cell[BoxData[ \(TraditionalForm\`D\_1\)]], " term is ", Cell[BoxData[ \(TraditionalForm\`1\/n\)]], " the standard Euclidean reconstruction error from one firing event, and \ the ", Cell[BoxData[ \(TraditionalForm\`D\_2\)]], " term has the neurons acting cooperatively to form a reconstruction ", Cell[BoxData[ FormBox[ RowBox[{ UnderscriptBox["\[Sum]", StyleBox["y", FontWeight->"Bold"]], RowBox[{ RowBox[{"Pr", "(", RowBox[{ StyleBox["y", FontWeight->"Bold"], "|", StyleBox["x", FontWeight->"Bold"]}], ")"}], RowBox[{ SuperscriptBox[ StyleBox["x", FontWeight->"Bold"], "\[Prime]"], "(", StyleBox["y", FontWeight->"Bold"], ")"}]}]}], TraditionalForm]]], ". When ", Cell[BoxData[ \(TraditionalForm\`n = 1\)]], " the ", Cell[BoxData[ \(TraditionalForm\`D\_2\)]], " contribution vanishes, because if there is only one firing event then the \ neurons cannot act cooperatively. When ", Cell[BoxData[ \(TraditionalForm\`n\[LongRightArrow]\[Infinity]\)]], " the ", Cell[BoxData[ \(TraditionalForm\`D\_1\)]], " contribution vanishes, because the reconstruction is dominated by neurons \ cooperating with each other." }], "Text", CellTags->"Ed:Change3"] }, Closed]], Cell[CellGroupData[{ Cell[TextData[{ CounterBox["Section"], ". Optimisation of coupled two-layer networks" }], "Section", CellTags->"Sect:OptimisationCoupled2LayerNetworks"], Cell[TextData[Cell[TextData[{ " ", ButtonBox["OPEN", ButtonData:>{ URL[ "http://www.luttrell.org.uk/papers/itbrain/fig2.gif"], None}, Active->True, ButtonStyle->"Hyperlink"], " " }]]], "NumberedFigure", TextAlignment->Center, CellTags->"Fig:2LayerCoupled"], Cell["\<\ (a) The basic two-layer network. (b) Two two-layer networks coupled together. \ The input to the upper two-layer network (represented by the shaded rectangle \ in the middle layer) is derived from the firing events that occur in the \ output layer of the lower two-layer network (represented by the white \ rectangle in the middle layer).\ \>", "Caption", CellTags->"Ed:Change1"], Cell[TextData[{ "In ", ButtonBox["figure", ButtonData:>"Fig:2LayerCoupled", ButtonStyle->"Hyperlink"], " ", CounterBox["NumberedFigure", "Fig:2LayerCoupled"], " two two-layer networks are combined to produce a three-layer network. For \ simplicity, the optimisation principle for multiple two-layer networks will \ simply be to minimise the sum of their separate Euclidean distortions. The \ following notation will be used below to describe the operation of the ", Cell[BoxData[ \(TraditionalForm\`l\)]], "th layer in a multi-layer network: ", Cell[BoxData[ FormBox[ RowBox[{ SuperscriptBox[ StyleBox["x", FontWeight->"Bold"], \((l)\)], "=", \(input\ pattern\)}], TraditionalForm]]], ", ", Cell[BoxData[ FormBox[ RowBox[{ SuperscriptBox[ StyleBox["s", FontWeight->"Bold"], \((l)\)], "=", \(layer\ parameters\)}], TraditionalForm]]], ", ", Cell[BoxData[ FormBox[ RowBox[{ RowBox[{"(", RowBox[{ SubsuperscriptBox[ StyleBox["y", FontWeight->"Bold"], "1", \((l)\)], ",", SubsuperscriptBox[ StyleBox["y", FontWeight->"Bold"], "2", \((l)\)], ",", "\[CenterEllipsis]", ",", SubsuperscriptBox[ StyleBox["y", FontWeight->"Bold"], \(n\_l\), \((l)\)]}], ")"}], "=", \(n\_l\ output\ firing\ events\)}], TraditionalForm]]], ". Thus for an ", Cell[BoxData[ \(TraditionalForm\`\((L + 1)\)\)]], "-layer network with layers numbered ", Cell[BoxData[ \(TraditionalForm\`0, 1, \[CenterEllipsis], L\)]] }], "Text"], Cell[BoxData[ FormBox[ RowBox[{"D", "=", RowBox[{ RowBox[{\(\[Sum]\+\(l = 1\)\%L D\^\((l)\)\), "\[LessEqual]", RowBox[{\(\[Sum]\+\(l = 1\)\%L\), SubsuperscriptBox["D", StyleBox["ub", FontSlant->"Italic"], \((l)\)]}]}], "=", \(\[Sum]\+\(l = 1\)\%L\((D\_1\%\((l)\) + D\_2\%\((l)\))\)\)}]}], TraditionalForm]], "NumberedEquation",\ CellTags->"Eq:ObjectiveVert2UpperN"], Cell[TextData[{ "where ", Cell[BoxData[ FormBox[ RowBox[{\(D\^\((l)\)\), "=", RowBox[{ SubscriptBox["\[Sum]", SuperscriptBox[ StyleBox["x", FontWeight->"Bold"], \((l)\)]], RowBox[{ RowBox[{"Pr", "(", SuperscriptBox[ StyleBox["x", FontWeight->"Bold"], \((l)\)], ")"}], RowBox[{\(D\^\((l)\)\), "(", SuperscriptBox[ StyleBox["x", FontWeight->"Bold"], \((l)\)], ")"}]}]}]}], TraditionalForm]]], " (by analogy with ", ButtonBox["equation", ButtonData:>"Eq:ObjectiveSimple", ButtonStyle->"Hyperlink"], " (", CounterBox["NumberedEquation", "Eq:ObjectiveSimple"], ")) is the average Euclidean reconstruction distortion caused by the ", Cell[BoxData[ \(TraditionalForm\`l\)]], "th two-layer network. A summation ", Cell[BoxData[ FormBox[ SubscriptBox["\[Sum]", SuperscriptBox[ StyleBox["x", FontWeight->"Bold"], \((l)\)]], TraditionalForm]]], " rather than an integration ", Cell[BoxData[ FormBox[ RowBox[{"\[Integral]", SuperscriptBox[ StyleBox[ RowBox[{ StyleBox["d", FontSlant->"Plain"], StyleBox["x", FontWeight->"Bold"]}]], \((l)\)]}], TraditionalForm]]], " is used because it is now assumed that the input activity pattern ", Cell[BoxData[ FormBox[ SuperscriptBox[ StyleBox["x", FontWeight->"Bold"], \((l)\)], TraditionalForm]]], " is a vector of ", Cell[BoxData[ \(TraditionalForm\`n\_\(l - 1\)\)]], " lattice vectors describing the output firing events in layer ", Cell[BoxData[ \(TraditionalForm\`l - 1\)]] }], "Text"], Cell[BoxData[ FormBox[ RowBox[{ SuperscriptBox[ StyleBox["x", FontWeight->"Bold"], \((l)\)], "=", RowBox[{ RowBox[{"(", RowBox[{ SubsuperscriptBox[ StyleBox["y", FontWeight->"Bold"], "1", \((l - 1)\)], ",", SubsuperscriptBox[ StyleBox["y", FontWeight->"Bold"], "2", \((l - 1)\)], ",", "\[CenterEllipsis]", ",", SubsuperscriptBox[ StyleBox["y", FontWeight->"Bold"], \(n\_\(l - 1\)\), \((l - 1)\)]}], ")"}], "."}]}], TraditionalForm]], "NumberedEquation", CellTags->"Eq:ConnectLayers"], Cell[TextData[{ "The upper bound in ", ButtonBox["equation", ButtonData:>"Eq:ObjectiveVert2UpperN", ButtonStyle->"Hyperlink"], " (", CounterBox["NumberedEquation", "Eq:ObjectiveVert2UpperN"], ") may be differentiated with respect to the parameters in layer ", Cell[BoxData[ \(TraditionalForm\`l\_0 \[LessEqual] L\)]], " to yield" }], "Text"], Cell[BoxData[ FormBox[ RowBox[{ FractionBox[ RowBox[{"\[PartialD]", RowBox[{\(\[Sum]\+\(l = 1\)\%L\), SubsuperscriptBox["D", StyleBox["ub", FontSlant->"Italic"], \((l)\)]}]}], RowBox[{"\[PartialD]", SuperscriptBox[ StyleBox["s", FontWeight->"Bold"], \((l\_0)\)]}]], "=", RowBox[{ FractionBox[ RowBox[{"\[PartialD]", SubsuperscriptBox["D", StyleBox["ub", FontSlant->"Italic"], \((l\_0)\)]}], RowBox[{"\[PartialD]", SuperscriptBox[ StyleBox["s", FontWeight->"Bold"], \((l\_0)\)]}]], "+", RowBox[{\(\[Sum]\+\(l = l\_0 + 1\)\%L\), RowBox[{ UnderscriptBox["\[Sum]", SuperscriptBox[ StyleBox["x", FontWeight->"Bold"], \((l)\)]], RowBox[{ FractionBox[ RowBox[{"\[PartialD]", RowBox[{"Pr", "(", SuperscriptBox[ StyleBox["x", FontWeight->"Bold"], \((l)\)], ")"}]}], RowBox[{"\[PartialD]", SuperscriptBox[ StyleBox["s", FontWeight->"Bold"], \((l\_0)\)]}]], RowBox[{ SubsuperscriptBox["D", StyleBox["ub", FontSlant->"Italic"], \((l)\)], "(", SuperscriptBox[ StyleBox["x", FontWeight->"Bold"], \((l)\)], ")"}]}]}]}]}]}], TraditionalForm]], "NumberedEquation", CellTags->"Eq:ObjectiveVert2UpperNDeriv"], Cell[TextData[{ "where the ", Cell[BoxData[ \(TraditionalForm\`\[Sum]\+\(l = l\_0 + \ 1\)\%L\((\[CenterEllipsis])\)\)]], " term describes the side effects in layers ", Cell[BoxData[ \(TraditionalForm\`l > l\_0\)]], " of varying the parameters ", Cell[BoxData[ FormBox[ SuperscriptBox[ StyleBox["s", FontWeight->"Bold"], \((l\_0)\)], TraditionalForm]]], " in layer ", Cell[BoxData[ \(TraditionalForm\`l\_0\)]], "." }], "Text"], Cell[TextData[{ "In the case shown in ", ButtonBox["figure", ButtonData:>"Fig:2LayerCoupled", ButtonStyle->"Hyperlink"], " ", CounterBox["NumberedFigure", "Fig:2LayerCoupled"], "(b) this reduces to" }], "Text"], Cell[BoxData[{ FormBox[ RowBox[{ FractionBox[ RowBox[{"\[PartialD]", RowBox[{"(", RowBox[{ SubsuperscriptBox["D", StyleBox["ub", FontSlant->"Italic"], \((1)\)], "+", SubsuperscriptBox["D", StyleBox["ub", FontSlant->"Italic"], \((2)\)]}], ")"}]}], RowBox[{"\[PartialD]", SuperscriptBox[ StyleBox["s", FontWeight->"Bold"], \((1)\)]}]], "=", "\[AlignmentMarker]", RowBox[{ FractionBox[ RowBox[{"\[PartialD]", SubsuperscriptBox["D", StyleBox["ub", FontSlant->"Italic"], \((1)\)]}], RowBox[{"\[PartialD]", SuperscriptBox[ StyleBox["s", FontWeight->"Bold"], \((1)\)]}]], "+", RowBox[{ UnderscriptBox["\[Sum]", SuperscriptBox[ StyleBox["x", FontWeight->"Bold"], \((2)\)]], RowBox[{ FractionBox[ RowBox[{"\[PartialD]", RowBox[{"Pr", "(", SuperscriptBox[ StyleBox["x", FontWeight->"Bold"], \((2)\)], ")"}]}], RowBox[{"\[PartialD]", SuperscriptBox[ StyleBox["s", FontWeight->"Bold"], \((1)\)]}]], RowBox[{ SubsuperscriptBox["D", StyleBox["ub", FontSlant->"Italic"], \((2)\)], "(", SuperscriptBox[ StyleBox["x", FontWeight->"Bold"], \((2)\)], ")"}]}]}]}]}], TraditionalForm], "\n", FormBox[ RowBox[{ FractionBox[ RowBox[{"\[PartialD]", RowBox[{"(", RowBox[{ SubsuperscriptBox["D", StyleBox["ub", FontSlant->"Italic"], \((1)\)], "+", SubsuperscriptBox["D", StyleBox["ub", FontSlant->"Italic"], \((2)\)]}], ")"}]}], RowBox[{"\[PartialD]", SuperscriptBox[ StyleBox["s", FontWeight->"Bold"], \((2)\)]}]], "=", "\[AlignmentMarker]", FractionBox[ RowBox[{"\[PartialD]", SubsuperscriptBox["D", StyleBox["ub", FontSlant->"Italic"], \((2)\)]}], RowBox[{"\[PartialD]", SuperscriptBox[ StyleBox["s", FontWeight->"Bold"], \((2)\)]}]]}], TraditionalForm]}], "NumberedEquation", TextAlignment->AlignmentMarker, CellTags->"Eq:ObjectiveDeriv3Layer"], Cell[TextData[{ "Differentiation with respect to the parameters in the upper network gives \ rise to a straightforward term ", Cell[BoxData[ FormBox[ RowBox[{"\[AlignmentMarker]", RowBox[{ RowBox[{"\[PartialD]", SubsuperscriptBox["D", StyleBox["ub", FontSlant->"Italic"], \((2)\)]}], "/", RowBox[{"\[PartialD]", SuperscriptBox[ StyleBox["s", FontWeight->"Bold"], \((2)\)]}]}]}], TraditionalForm]]], ", whereas differentiation with respect to the parameters in the lower \ network gives rise to a sum of two contributions" }], "Text"], Cell[BoxData[ FormBox[ RowBox[{ FractionBox[ RowBox[{"\[PartialD]", SubsuperscriptBox["D", StyleBox["ub", FontSlant->"Italic"], \((1)\)]}], RowBox[{"\[PartialD]", SuperscriptBox[ StyleBox["s", FontWeight->"Bold"], \((1)\)]}]], "+", RowBox[{ UnderscriptBox["\[Sum]", SuperscriptBox[ StyleBox["x", FontWeight->"Bold"], \((2)\)]], RowBox[{ FractionBox[ RowBox[{"\[PartialD]", RowBox[{"Pr", "(", SuperscriptBox[ StyleBox["x", FontWeight->"Bold"], \((2)\)], ")"}]}], RowBox[{"\[PartialD]", SuperscriptBox[ StyleBox["s", FontWeight->"Bold"], \((1)\)]}]], RowBox[{ SubsuperscriptBox["D", StyleBox["ub", FontSlant->"Italic"], \((2)\)], "(", SuperscriptBox[ StyleBox["x", FontWeight->"Bold"], \((2)\)], ")"}]}]}]}], TraditionalForm]], "Text"], Cell[TextData[{ "the first term of which is analogous to that produced by the upper \ network, and the second term of which describes a back-propagating derivative \ created by the upper network to inform the lower network about the effect \ that varying ", Cell[BoxData[ FormBox[ SuperscriptBox[ StyleBox["s", FontWeight->"Bold"], \((1)\)], TraditionalForm]]], " has on its input probability distribution ", Cell[BoxData[ FormBox[ RowBox[{"Pr", "(", SuperscriptBox[ StyleBox["x", FontWeight->"Bold"], \((2)\)], ")"}], TraditionalForm]]], " [", ButtonBox["10", ButtonData:>"Ref:Luttrell1992a", ButtonStyle->"Hyperlink"], ", ", ButtonBox["11", ButtonData:>"Ref:Luttrell1992b", ButtonStyle->"Hyperlink"], "]; this is self-supervision." }], "Text"], Cell[TextData[{ "In [", ButtonBox["10", ButtonData:>"Ref:Luttrell1992a", ButtonStyle->"Hyperlink"], "] it was shown how this self-supervision mechanism leads to the automatic \ generation of neighbourhood functions of the type specified in ", ButtonBox["equation", ButtonData:>"Eq:Leakage", ButtonStyle->"Hyperlink"], " (", CounterBox["NumberedEquation", "Eq:Leakage"], "). In this ", "model, topographic ordering of the neuron properties occurs as a result of \ the implicit neighbourhood function generated by the interaction between \ different layers of neurons, rather than as a result of an explicit \ neighbourhood function." }], "Text"] }, Closed]], Cell[CellGroupData[{ Cell[TextData[{ CounterBox["Section"], ". Adaptive cluster expansion (ACE)" }], "Section", CellTags->"Sect:AdaptiveClusterExpansion"], Cell[TextData[Cell[TextData[{ " ", ButtonBox["OPEN", ButtonData:>{ URL[ "http://www.luttrell.org.uk/papers/itbrain/fig3.gif"], None}, Active->True, ButtonStyle->"Hyperlink"], " " }]]], "NumberedFigure", TextAlignment->Center, CellTags->"Fig:ACE"], Cell[TextData[{ "Adaptive cluster expansion (ACE) network formed from two-layer networks \ of. the type shown in ", ButtonBox["figure", ButtonData:>"Fig:2LayerCoupled", ButtonStyle->"Hyperlink"], " ", CounterBox["NumberedFigure", "Fig:2LayerCoupled"], "(a). The input to each two-layer network is derived from the firing events \ generated by a pair of two-layer networks, as represented by the two white \ rectangles embedded inside each shaded rectangle." }], "Caption"], Cell[TextData[{ "The three-layer network shown in ", ButtonBox["figure", ButtonData:>"Fig:2LayerCoupled", ButtonStyle->"Hyperlink"], " ", CounterBox["NumberedFigure", "Fig:2LayerCoupled"], "(b) can have its layer-to-layer connections pruned in various ways. A \ particularly useful scheme is shown in ", ButtonBox["figure", ButtonData:>"Fig:ACE", ButtonStyle->"Hyperlink"], " ", CounterBox["NumberedFigure", "Fig:ACE"], ", where a number of two-layer networks of the type shown in ", ButtonBox["figure", ButtonData:>"Fig:2LayerCoupled", ButtonStyle->"Hyperlink"], " ", CounterBox["NumberedFigure", "Fig:2LayerCoupled"], "(a) are combined together hierarchically. The coupling between these \ two-layer networks leads to self- supervision effects. This network is called \ the adaptive cluster expansion (ACE) because it is an adaptive network that \ seeks to encode high-dimensional input patterns in a stepwise fashion, \ starting with low-dimensional pieces (called clusters) of the input, and \ progressively (layer-by-layer) combining these to eventually yield a final \ output that encodes the whole high-dimensional input pattern." }], "Text"], Cell[TextData[{ "In [", ButtonBox["6", ButtonData:>"Ref:Luttrell1989a", ButtonStyle->"Hyperlink"], "] an application of ACE to image compression is described; it uses only a \ single firing event in the output of each embedded two-layer network. There \ are many variations on the connectivity shown in ", ButtonBox["figure", ButtonData:>"Fig:ACE", ButtonStyle->"Hyperlink"], " ", CounterBox["NumberedFigure", "Fig:ACE"], ". For instance, an ACE network can be used for data fusion applications, \ where the clusters in the input layer correspond to the separate sensors \ receiving information, and the output layer (several layers further up the \ ACE hierarchy) holds a unified representation of the information supplied by \ the sensors. The output may need to have certain properties that do not \ emerge spontaneously by self-organisation as the ACE network is trained, in \ which case an external teacher must add some suitable external-supervision \ terms to ", Cell[BoxData[ \(TraditionalForm\`D\)]], ", which then modify the self-supervision terms that already appear in ", ButtonBox["equation", ButtonData:>"Eq:ObjectiveVert2UpperNDeriv", ButtonStyle->"Hyperlink"], " (", CounterBox["NumberedEquation", "Eq:ObjectiveVert2UpperNDeriv"], ") and ", ButtonBox["equation", ButtonData:>"Eq:ObjectiveDeriv3Layer", ButtonStyle->"Hyperlink"], " (", CounterBox["NumberedEquation", "Eq:ObjectiveDeriv3Layer"], ")." }], "Text", CellTags->"Ed:Change2"], Cell[TextData[{ "The ACE network architecture can be applied to the problem of producing an \ estimate of the joint PDF of patterns in a high-dimensional input space [", ButtonBox["9", ButtonData:>"Ref:Luttrell1991", ButtonStyle->"Hyperlink"], ", ", ButtonBox["13", ButtonData:>"Ref:Luttrell1994b", ButtonStyle->"Hyperlink"], "]. If instead of minimising the reconstruction error, the feedforward and \ feedback processing are optimised specifically so that the best possible PDF \ estimate (in the relative entropy sense) is produced, then the overall \ network optimisation criterion is a sum of mutual information within each \ layer of the network, which is a generalisation of the mutual information \ maximisation principle used by Becker and Hinton [", ButtonBox["1", ButtonData:>"Ref:BeckerHinton1992", ButtonStyle->"Hyperlink"], "]." }], "Text"] }, Closed]], Cell[CellGroupData[{ Cell[TextData[{ CounterBox["Section"], ". ", "Conclusions" }], "Section"], Cell["\<\ In a two-layer network the feedforward and feedback processing between the \ input and output layers are mutually coupled via Bayes' theorem. This \ processing can be optimised so as to minimise the average Euclidean \ reconstruction error that occurs when a forward pass followed by a backward \ pass is made through the network. If several such two-layer networks are \ connected together such that the input of each is derived from the output of \ one or more other two-layer networks, then a multi-layer network of connected \ two-layer networks can be built, and supervision of the lower networks by the \ higher networks occurs (i.e. self-supervision).\ \>", "Text"], Cell["\<\ A particular example of this way of combining two-layer networks is the \ adaptive cluster expansion (ACE) network, in which each embedded two-layer \ network produces one or more firing events in its output layer, and two or \ more such output layers are concatenated to produce the input to another \ two-layer network; the overall structure is a hierarchically connected set of \ two-layer networks.\ \>", "Text"], Cell["\<\ Because its layer-to-layer connectivity is strictly hierarchical, the version \ of ACE described in this paper is not as flexible as it potentially could be, \ but it has a large number of simple theoretical properties. It should \ therefore be used as a theoretical laboratory in which ideas can be tested \ out, before trying out more realistic connectivities.\ \>", "Text"] }, Closed]], Cell[CellGroupData[{ Cell["References", "Section"], Cell[TextData[{ "[1] Becker S and Hinton G E 1992 Self-organising neural network that \ discovers surfaces in random-dot stereograms ", StyleBox["Nature", FontSlant->"Italic"], " ", StyleBox["355", FontWeight->"Bold"], " 161-3" }], "Reference", CellTags->"Ref:BeckerHinton1992"], Cell[TextData[{ "[2] Dayan P, Hinton G E, Neal R M and Zemel R S 1995 Helmholtz machines ", StyleBox["Neural Comput.", FontSlant->"Italic"], " ", StyleBox["7", FontWeight->"Bold"], " 889-904" }], "Reference", CellTags->"Ref:DayanHintonNealZemel1995"], Cell[TextData[{ "[3] Hinton G E, Dayan P, Frey B J and Neal R M 1995 The wake-sleep \ algorithm for unsupervised neural networks ", StyleBox["Science", FontSlant->"Italic"], " ", StyleBox["268", FontWeight->"Bold"], " 1158-61" }], "Reference", CellTags->"Ref:HintonDayanFreyNeal1995"], Cell[TextData[{ "[4] Kohonen T 1984 ", StyleBox["Self-organisation and Associative Memory", FontSlant->"Italic"], " (Berlin: Springer)" }], "Reference", CellTags->"Ref:Kohonen1984"], Cell[TextData[{ "[5] Linde Y, Buzo A and Gray R M 1980 An algorithm for vector quantiser \ design ", StyleBox["IEEE Trans. Comput.", FontSlant->"Italic"], " ", StyleBox["28", FontWeight->"Bold"], " 84-95" }], "Reference", CellTags->"Ref:LindeBuzoGray1980"], Cell[TextData[{ Cell[TextData[{ " ", ButtonBox["OPEN", ButtonData:>{ URL[ "http://www.luttrell.org.uk/papers/compress/compress.nb"], None}, Active->True, ButtonStyle->"Hyperlink"], " " }]], "[6] Luttrell S P 1989 Image compression using a multilayer neural network \ ", StyleBox["Pattern Recognition Lett.", FontSlant->"Italic"], " ", StyleBox["10", FontWeight->"Bold"], " 1-7" }], "Reference", CellTags->"Ref:Luttrell1989a"], Cell[TextData[{ Cell[TextData[{ " ", ButtonBox["OPEN", ButtonData:>{ URL[ "http://www.luttrell.org.uk/papers/hiervq/hiervq.nb"], None}, Active->True, ButtonStyle->"Hyperlink"], " " }]], "[7] Luttrell S P 1989 Hierarchical vector quantisation ", StyleBox["IEE Proc.", FontSlant->"Italic"], " I ", StyleBox["136", FontWeight->"Bold"], " 405-13" }], "Reference", CellTags->"Ref:Luttrell1989b"], Cell[TextData[{ Cell[TextData[{ " ", ButtonBox["OPEN", ButtonData:>{ URL[ "http://www.luttrell.org.uk/papers/tvq/tvq.nb"], None}, Active->True, ButtonStyle->"Hyperlink"], " " }]], "[8] Luttrell S P 1990 Derivation of a class of training algorithms ", StyleBox["IEEE Trans. Neural Networks", FontSlant->"Italic"], " ", StyleBox["1", FontWeight->"Bold"], " 229-32" }], "Reference", CellTags->"Ref:Luttrell1990"], Cell[TextData[{ Cell[TextData[{ " ", ButtonBox["OPEN", ButtonData:>{ URL[ "http://www.luttrell.org.uk/papers/spie91/spie91.nb"], None}, Active->True, ButtonStyle->"Hyperlink"], " " }]], "[9] Luttrell S P 1991 A hierarchical network for clutter and texture \ modelling ", StyleBox["Proc. SPIE Conf on Adaptive Signal Processing (San Diego)", FontSlant->"Italic"], " pp 518-28" }], "Reference", CellTags->"Ref:Luttrell1991"], Cell[TextData[{ Cell[TextData[{ " ", ButtonBox["OPEN", ButtonData:>{ URL[ "http://www.luttrell.org.uk/papers/selfsup/selfsup.nb"], None}, Active->True, ButtonStyle->"Hyperlink"], " " }]], "[10] Luttrell S P 1992 Self-supervised adaptive networks ", StyleBox["IEE Proc.", FontSlant->"Italic"], " F ", StyleBox["139", FontWeight->"Bold"], " 371-7" }], "Reference", CellTags->"Ref:Luttrell1992a"], Cell[TextData[{ Cell[TextData[{ " ", ButtonBox["OPEN", ButtonData:>{ URL[ "http://www.luttrell.org.uk/papers/spie92/spie92.nb"], None}, Active->True, ButtonStyle->"Hyperlink"], " " }]], "[11] Luttrell S P 1992 Adaptive Bayesian networks ", StyleBox["Proc. SPIE Conf on Adaptive Signal Processing (Orlando)", FontSlant->"Italic"], " pp 140-51" }], "Reference", CellTags->"Ref:Luttrell1992b"], Cell[TextData[{ Cell[TextData[{ " ", ButtonBox["OPEN", ButtonData:>{ URL[ "http://www.luttrell.org.uk/papers/bayessom/bayessom.nb"], None}, Active->True, ButtonStyle->"Hyperlink"], " " }]], "[", "12] Luttrell S P 1994 A Bayesian analysis of self-organising maps ", StyleBox["Neural Comput.", FontSlant->"Italic"], " ", StyleBox["6", FontWeight->"Bold"], " 767-94" }], "Reference", CellTags->"Ref:Luttrell1994a"], Cell[TextData[{ Cell[TextData[{ " ", ButtonBox["OPEN", ButtonData:>{ URL[ "http://www.luttrell.org.uk/papers/maxent94_1/maxent94_1.nb"], None}, Active->True, ButtonStyle->"Hyperlink"], " " }]], "[13] Luttrell S P 1994 The cluster expansion: a hierarchical density model \ ", StyleBox["Proc. 14th MaxEnt Workshop (Cambridge)", FontSlant->"Italic"] }], "Reference", CellTags->"Ref:Luttrell1994b"], Cell[TextData[{ Cell[TextData[{ " ", ButtonBox["OPEN", ButtonData:>{ URL[ "http://www.luttrell.org.uk/papers/honc/honc.nb"], None}, Active->True, ButtonStyle->"Hyperlink"], " " }]], "[14] Luttrell S P 1996 Designing analysable networks ", StyleBox["Handbook of Neural Computation", FontSlant->"Italic"], " (Bristol: Institute of Physics Publishing) to be published" }], "Reference", CellTags->"Ref:Luttrell1996a"], Cell[TextData[{ Cell[TextData[{ " ", ButtonBox["OPEN", ButtonData:>{ URL[ "http://www.luttrell.org.uk/papers/manna95/manna95.nb"], None}, Active->True, ButtonStyle->"Hyperlink"], " " }]], "[15] Luttrell S P 1996 A theory of self-organising neural networks ", StyleBox["Proc. 1st Conf. on Mathematics of Artificial Neural Networks and \ Applications (Oxford), Ann. Math. Artificial Intelligence", FontSlant->"Italic"], " to be published" }], "Reference", CellTags->"Ref:Luttrell1996b"] }, Closed]] }, Open ]] }, FrontEndVersion->"5.0 for Microsoft Windows", ScreenRectangle->{{0, 1280}, {0, 941}}, WindowSize->{665.375, 641}, WindowMargins->{{307.25, Automatic}, {Automatic, 50}}, Magnification->1, StyleDefinitions -> "Report.nb" ] (******************************************************************* Cached data follows. If you edit this Notebook file directly, not using Mathematica, you must remove the line containing CacheID at the top of the file. The cache data will then be recreated when you save this file from within Mathematica. *******************************************************************) (*CellTagsOutline CellTagsIndex->{ "Fig:2Layer"->{ Cell[4834, 169, 277, 10, 70, "NumberedFigure", CellTags->"Fig:2Layer"]}, "Sect:Optimisation2LayerNetwork"->{ Cell[7615, 249, 146, 5, 70, "Section", CellTags->"Sect:Optimisation2LayerNetwork"]}, "Eq:Objective"->{ Cell[12240, 395, 1893, 51, 70, "NumberedEquation", CellTags->"Eq:Objective"]}, "Eq:ObjectiveSimple"->{ Cell[17216, 545, 651, 20, 70, "NumberedEquation", CellTags->"Eq:ObjectiveSimple"]}, "Eq:Leakage"->{ Cell[29884, 949, 1346, 40, 70, "NumberedEquation", CellTags->"Eq:Leakage"]}, "Eq:ObjectiveN"->{ Cell[34765, 1108, 3151, 80, 70, "NumberedEquation", CellTags->"Eq:ObjectiveN"]}, "Eq:ObjectiveUpperBound"->{ Cell[41274, 1303, 3215, 79, 70, "NumberedEquation", CellTags->"Eq:ObjectiveUpperBound"]}, "Ed:Change3"->{ Cell[44492, 1384, 2022, 65, 70, "Text", CellTags->"Ed:Change3"]}, "Sect:OptimisationCoupled2LayerNetworks"->{ Cell[46551, 1454, 161, 5, 70, "Section", CellTags->"Sect:OptimisationCoupled2LayerNetworks"]}, "Fig:2LayerCoupled"->{ Cell[46715, 1461, 284, 10, 70, "NumberedFigure", CellTags->"Fig:2LayerCoupled"]}, "Ed:Change1"->{ Cell[47002, 1473, 391, 7, 70, "Caption", CellTags->"Ed:Change1"]}, "Eq:ObjectiveVert2UpperN"->{ Cell[49182, 1537, 488, 12, 70, "NumberedEquation", CellTags->"Eq:ObjectiveVert2UpperN"]}, "Eq:ConnectLayers"->{ Cell[51613, 1613, 731, 20, 70, "NumberedEquation", CellTags->"Eq:ConnectLayers"]}, "Eq:ObjectiveVert2UpperNDeriv"->{ Cell[52722, 1649, 1847, 48, 70, "NumberedEquation", CellTags->"Eq:ObjectiveVert2UpperNDeriv"]}, "Eq:ObjectiveDeriv3Layer"->{ Cell[55306, 1730, 2847, 78, 70, "NumberedEquation", CellTags->"Eq:ObjectiveDeriv3Layer"]}, "Sect:AdaptiveClusterExpansion"->{ Cell[61659, 1917, 142, 5, 70, "Section", CellTags->"Sect:AdaptiveClusterExpansion"]}, "Fig:ACE"->{ Cell[61804, 1924, 274, 10, 70, "NumberedFigure", CellTags->"Fig:ACE"]}, "Ed:Change2"->{ Cell[63788, 1982, 1533, 40, 70, "Text", CellTags->"Ed:Change2"]}, "Ref:BeckerHinton1992"->{ Cell[67907, 2090, 298, 10, 70, "Reference", CellTags->"Ref:BeckerHinton1992"]}, "Ref:DayanHintonNealZemel1995"->{ Cell[68208, 2102, 270, 9, 70, "Reference", CellTags->"Ref:DayanHintonNealZemel1995"]}, "Ref:HintonDayanFreyNeal1995"->{ Cell[68481, 2113, 304, 10, 70, "Reference", CellTags->"Ref:HintonDayanFreyNeal1995"]}, "Ref:Kohonen1984"->{ Cell[68788, 2125, 193, 6, 70, "Reference", CellTags->"Ref:Kohonen1984"]}, "Ref:LindeBuzoGray1980"->{ Cell[68984, 2133, 276, 10, 70, "Reference", CellTags->"Ref:LindeBuzoGray1980"]}, "Ref:Luttrell1989a"->{ Cell[69263, 2145, 494, 20, 70, "Reference", CellTags->"Ref:Luttrell1989a"]}, "Ref:Luttrell1989b"->{ Cell[69760, 2167, 452, 18, 70, "Reference", CellTags->"Ref:Luttrell1989b"]}, "Ref:Luttrell1990"->{ Cell[70215, 2187, 471, 18, 70, "Reference", CellTags->"Ref:Luttrell1990"]}, "Ref:Luttrell1991"->{ Cell[70689, 2207, 476, 16, 70, "Reference", CellTags->"Ref:Luttrell1991"]}, "Ref:Luttrell1992a"->{ Cell[71168, 2225, 455, 18, 70, "Reference", CellTags->"Ref:Luttrell1992a"]}, "Ref:Luttrell1992b"->{ Cell[71626, 2245, 444, 15, 70, "Reference", CellTags->"Ref:Luttrell1992b"]}, "Ref:Luttrell1994a"->{ Cell[72073, 2262, 482, 20, 70, "Reference", CellTags->"Ref:Luttrell1994a"]}, "Ref:Luttrell1994b"->{ Cell[72558, 2284, 455, 16, 70, "Reference", CellTags->"Ref:Luttrell1994b"]}, "Ref:Luttrell1996a"->{ Cell[73016, 2302, 467, 15, 70, "Reference", CellTags->"Ref:Luttrell1996a"]}, "Ref:Luttrell1996b"->{ Cell[73486, 2319, 538, 16, 70, "Reference", CellTags->"Ref:Luttrell1996b"]} } *) (*CellTagsIndex CellTagsIndex->{ {"Fig:2Layer", 74707, 2356}, {"Sect:Optimisation2LayerNetwork", 74828, 2359}, {"Eq:Objective", 74943, 2362}, {"Eq:ObjectiveSimple", 75058, 2365}, {"Eq:Leakage", 75170, 2368}, {"Eq:ObjectiveN", 75278, 2371}, {"Eq:ObjectiveUpperBound", 75399, 2374}, {"Ed:Change3", 75517, 2377}, {"Sect:OptimisationCoupled2LayerNetworks", 75639, 2380}, {"Fig:2LayerCoupled", 75769, 2383}, {"Ed:Change1", 75879, 2386}, {"Eq:ObjectiveVert2UpperN", 75987, 2389}, {"Eq:ConnectLayers", 76111, 2392}, {"Eq:ObjectiveVert2UpperNDeriv", 76240, 2395}, {"Eq:ObjectiveDeriv3Layer", 76377, 2398}, {"Sect:AdaptiveClusterExpansion", 76515, 2401}, {"Fig:ACE", 76626, 2404}, {"Ed:Change2", 76726, 2407}, {"Ref:BeckerHinton1992", 76830, 2410}, {"Ref:DayanHintonNealZemel1995", 76956, 2413}, {"Ref:HintonDayanFreyNeal1995", 77088, 2416}, {"Ref:Kohonen1984", 77208, 2419}, {"Ref:LindeBuzoGray1980", 77321, 2422}, {"Ref:Luttrell1989a", 77437, 2425}, {"Ref:Luttrell1989b", 77549, 2428}, {"Ref:Luttrell1990", 77660, 2431}, {"Ref:Luttrell1991", 77770, 2434}, {"Ref:Luttrell1992a", 77881, 2437}, {"Ref:Luttrell1992b", 77993, 2440}, {"Ref:Luttrell1994a", 78105, 2443}, {"Ref:Luttrell1994b", 78217, 2446}, {"Ref:Luttrell1996a", 78329, 2449}, {"Ref:Luttrell1996b", 78441, 2452} } *) (*NotebookFileOutline Notebook[{ Cell[CellGroupData[{ Cell[1776, 53, 26, 0, 70, "Section 1"], Cell[CellGroupData[{ Cell[1827, 57, 39, 0, 70, "Subsection"], Cell[1869, 59, 62, 0, 70, "Text"], Cell[1934, 61, 332, 14, 70, "Text"], Cell[2269, 77, 285, 9, 70, "Text"], Cell[2557, 88, 345, 10, 70, "Text"] }, Closed]] }, Closed]], Cell[CellGroupData[{ Cell[2951, 104, 99, 2, 70, "Title"], Cell[3053, 108, 144, 4, 70, "Author"], Cell[3200, 114, 140, 4, 70, "Text"], Cell[3343, 120, 166, 3, 70, "Text"], Cell[3512, 125, 61, 0, 70, "Text"], Cell[3576, 127, 476, 9, 70, "Abstract"], Cell[CellGroupData[{ Cell[4077, 140, 79, 4, 70, "Section 1"], Cell[4159, 146, 672, 21, 70, "Text"], Cell[4834, 169, 277, 10, 70, "NumberedFigure", CellTags->"Fig:2Layer"], Cell[5114, 181, 272, 5, 70, "Caption"], Cell[5389, 188, 1363, 30, 70, "Text"], Cell[6755, 220, 823, 24, 70, "Text"] }, Closed]], Cell[CellGroupData[{ Cell[7615, 249, 146, 5, 70, "Section", CellTags->"Sect:Optimisation2LayerNetwork"], Cell[7764, 256, 2301, 68, 70, "Text"], Cell[10068, 326, 1692, 47, 70, "Text"], Cell[11763, 375, 474, 18, 70, "Text"], Cell[12240, 395, 1893, 51, 70, "NumberedEquation", CellTags->"Eq:Objective"], Cell[14136, 448, 3077, 95, 70, "Text"], Cell[17216, 545, 651, 20, 70, "NumberedEquation", CellTags->"Eq:ObjectiveSimple"], Cell[17870, 567, 5181, 163, 70, "Text"], Cell[23054, 732, 2265, 65, 70, "Text"], Cell[25322, 799, 4467, 144, 70, "Text"], Cell[29792, 945, 89, 2, 70, "Text"], Cell[29884, 949, 1346, 40, 70, "NumberedEquation", CellTags->"Eq:Leakage"], Cell[31233, 991, 2632, 80, 70, "Text"], Cell[33868, 1073, 894, 33, 70, "Text"], Cell[34765, 1108, 3151, 80, 70, "NumberedEquation", CellTags->"Eq:ObjectiveN"], Cell[37919, 1190, 3352, 111, 70, "Text"], Cell[41274, 1303, 3215, 79, 70, "NumberedEquation", CellTags->"Eq:ObjectiveUpperBound"], Cell[44492, 1384, 2022, 65, 70, "Text", CellTags->"Ed:Change3"] }, Closed]], Cell[CellGroupData[{ Cell[46551, 1454, 161, 5, 70, "Section", CellTags->"Sect:OptimisationCoupled2LayerNetworks"], Cell[46715, 1461, 284, 10, 70, "NumberedFigure", CellTags->"Fig:2LayerCoupled"], Cell[47002, 1473, 391, 7, 70, "Caption", CellTags->"Ed:Change1"], Cell[47396, 1482, 1783, 53, 70, "Text"], Cell[49182, 1537, 488, 12, 70, "NumberedEquation", CellTags->"Eq:ObjectiveVert2UpperN"], Cell[49673, 1551, 1937, 60, 70, "Text"], Cell[51613, 1613, 731, 20, 70, "NumberedEquation", CellTags->"Eq:ConnectLayers"], Cell[52347, 1635, 372, 12, 70, "Text"], Cell[52722, 1649, 1847, 48, 70, "NumberedEquation", CellTags->"Eq:ObjectiveVert2UpperNDeriv"], Cell[54572, 1699, 497, 18, 70, "Text"], Cell[55072, 1719, 231, 9, 70, "Text"], Cell[55306, 1730, 2847, 78, 70, "NumberedEquation", CellTags->"Eq:ObjectiveDeriv3Layer"], Cell[58156, 1810, 688, 17, 70, "Text"], Cell[58847, 1829, 1224, 35, 70, "Text"], Cell[60074, 1866, 866, 26, 70, "Text"], Cell[60943, 1894, 679, 18, 70, "Text"] }, Closed]], Cell[CellGroupData[{ Cell[61659, 1917, 142, 5, 70, "Section", CellTags->"Sect:AdaptiveClusterExpansion"], Cell[61804, 1924, 274, 10, 70, "NumberedFigure", CellTags->"Fig:ACE"], Cell[62081, 1936, 493, 12, 70, "Caption"], Cell[62577, 1950, 1208, 30, 70, "Text"], Cell[63788, 1982, 1533, 40, 70, "Text", CellTags->"Ed:Change2"], Cell[65324, 2024, 892, 20, 70, "Text"] }, Closed]], Cell[CellGroupData[{ Cell[66253, 2049, 82, 5, 70, "Section"], Cell[66338, 2056, 682, 10, 70, "Text"], Cell[67023, 2068, 426, 7, 70, "Text"], Cell[67452, 2077, 386, 6, 70, "Text"] }, Closed]], Cell[CellGroupData[{ Cell[67875, 2088, 29, 0, 70, "Section"], Cell[67907, 2090, 298, 10, 70, "Reference", CellTags->"Ref:BeckerHinton1992"], Cell[68208, 2102, 270, 9, 70, "Reference", CellTags->"Ref:DayanHintonNealZemel1995"], Cell[68481, 2113, 304, 10, 70, "Reference", CellTags->"Ref:HintonDayanFreyNeal1995"], Cell[68788, 2125, 193, 6, 70, "Reference", CellTags->"Ref:Kohonen1984"], Cell[68984, 2133, 276, 10, 70, "Reference", CellTags->"Ref:LindeBuzoGray1980"], Cell[69263, 2145, 494, 20, 70, "Reference", CellTags->"Ref:Luttrell1989a"], Cell[69760, 2167, 452, 18, 70, "Reference", CellTags->"Ref:Luttrell1989b"], Cell[70215, 2187, 471, 18, 70, "Reference", CellTags->"Ref:Luttrell1990"], Cell[70689, 2207, 476, 16, 70, "Reference", CellTags->"Ref:Luttrell1991"], Cell[71168, 2225, 455, 18, 70, "Reference", CellTags->"Ref:Luttrell1992a"], Cell[71626, 2245, 444, 15, 70, "Reference", CellTags->"Ref:Luttrell1992b"], Cell[72073, 2262, 482, 20, 70, "Reference", CellTags->"Ref:Luttrell1994a"], Cell[72558, 2284, 455, 16, 70, "Reference", CellTags->"Ref:Luttrell1994b"], Cell[73016, 2302, 467, 15, 70, "Reference", CellTags->"Ref:Luttrell1996a"], Cell[73486, 2319, 538, 16, 70, "Reference", CellTags->"Ref:Luttrell1996b"] }, Closed]] }, Open ]] } ] *) (******************************************************************* End of Mathematica Notebook file. *******************************************************************)