Much of the work done in the Levy research lab has focused on neural nets with neurons whose connections from one into another are randomly distributed. The research presented here is a first step in introducing a topology where the probability of connection from one neuron into another is a function of distance between those neurons. One motivation for studying neural nets with this relationship is to explore how a model that incorporates the interaction between the hippocampus and neocortex could be created. As a first approximation, the hippocampus is being treated as having random connections, and the neocortex as having connections that are a function of distance between neurons.
In order to investigate the effects of topology on the Levy minimal model of neural networks, neurons were arranged on a hexagonal grid with their probability of connection being a function of the Euclidean 2-d distance between neurons on this grid. The experiments performed so far have been with a competitive version of this model, such that only neurons with the top 10% internal activity are selected to fire in any given time-step. Additionally, before each testing or training trial the neurons were initialized to indicate that no firing had occurred in the previous time-step.
All experiments were performed using a hexagonal grid of radius 50 neurons with periodic boundaries. An example of a hexagonal grid of radius 2 neurons with periodic boundaries is shown below.
In this picture, all hexagons actually refer to the same hexagon translated in the plane. From this example we can see that the center hexagon's upper-left neuron neighbors two neurons in the right part of the upper-left hexagon and the top hexagon's lower left neuron, in addition to the three neurons in the center hexagon that would naturally be neighbored without a periodic boundary. The corresponding neurons in the center hexagon are circled to demonstrate their actual placement. Furthermore, in any hexagon the upper-left neuron will neighbor the far-right and lower-left neurons with respect to the periodic boundaries. The reason for choosing to implement periodic boundaries was to make any edge effects as transparent as possible. Other options considered were to reduce the connectivity of the edge neurons by not creating connections where the connections would have wrapped around or to choose connections for edge neurons from those neurons that are available without wrapping around. The former option would result in creating biases against edge neurons that could be communicated to their efferent neurons, and the latter option would result in increased symmetry of connections in the edge neurons as well as creating problems when there were not enough neurons actually available to choose from.
The hexagonal grid of radius 50 neurons necessarily contained 7,651 neurons total. Each of these neurons had an afferent connection to 5% or 383 of the other neurons, with no neuron having a connection to itself. The afferent connections were distributed such that they approximately represented a Gaussian probability with respect to distance. It should be noted that the number of available connections increases with distance, so that the actual number of connections at a given distance did not follow a Gaussian distribution. Each experiment had the probability of connection set to zero at a distance beyond 14, 16, or 18 neurons, such that all afferent connections were constrained to distances of 13, 15, or 17 neurons respectively. This maximal distance of connectivity will be referred to as the connectivity radius. It is also worth noting that at a distance of 11 neurons there are only 374 neurons to create connections, so that the earliest distance at which the probability of connection could be set to zero is 13. The figures below show the probability of a connection as a function of distance for the 3 different values of connectivity radius, with the dotted black line representing the local connectivity within this radius.
| Connectivity Radius | P(x->y) | P(x->y|y->x) | P(not(x->y)) | P(not(x->y)|not(y->x)) |
|---|---|---|---|---|
| 13 neurons | 0.050 | 0.695 | 0.950 | 0.984 |
| 15 neurons | 0.050 | 0.534 | 0.950 | 0.975 |
| 17 neurons | 0.050 | 0.446 | 0.950 | 0.971 |
The symmetry of connection is expected to contribute significantly to the type of problems the network can and cannot solve, as it greatly increases the probability of creating strong basins of attraction. For each connectivity radius five different topologies were created by changing the random seed used to create them.
The patterns that were presented to the networks during training would have either 2%, 5%, 10%, 20%, 40%, 60%, 80%, or 100% of the active neurons for each time step, with this percentage labeled as percent external. For a given experiment, a single pattern would be presented for 20 trials of 50 time-steps each. Each pattern was composed of two sub-patterns of equal size. The distance between the centers of the patterns was varied to determine the maximal separation at which one sub-pattern could be recalled when the network is presented with the other. At each percentage level four to eight patterns were created and presented to each of the five topologies, generating between twenty (20) and forty (40) experiments each. The following figures give an example at each percentage at a separation of 28 neurons between the centers of the sub-patterns.
In addition to examining the individual combinations of parameters, two main techniquess were used to investigate the effects of the independent variables. The first technique was to examine how the recall percentage depended on the percent external and connectivity radius. The second technique involved defining a sub-pattern separation point at which failure begins for a specified connectivity radius and percent external. This point was defined as the distance where exactly 50% of the experiments failed. If no such distance existed, then the point would be interpolated between the distance where success was greater than 50% and the distance where success was less than 50%. If multiple distances existed, then the point would be defined as the average value of those distances. The largest factor in determining the percentage of experiments with successful recall is the distance between the sub-patterns. For small enough separations, recall was 100%, regardless of connectivity radius or percent external. For large enough separations, recall was always 0%, although larger values of connectivity radius should be tested. Possibly as the connectivity radius approaches the size of the topology radius, arbitrarily large separations can be learned.
For the connectivity radii tested, increasing the size of the connectivity radius improved the ability to recall the missing sub-pattern with larger separations. This improvement was approximately equal to one half the change in connectivity radius. Changing the percent external could have a dramatic effect for a fixed separation distance and fixed connectivity radius, but in general this effect was largest when the percent external was large enough to prevent the creation of bridge neurons, those neurons that formed between the two sub-patterns in order to create a mechanism for recall. Additionally, increasing percent external generally decreased the distance at which 50% successful recall occurred. For very small values of percent external, e.g., less than 5%, this distance was increased by increasing percent external.
The following figure is an example of how success is affected by percent external and connectivity radius when the patterns are separated by 30 neurons.
The next figure shows the distance between sub-patterns that results in 50% successful recall. When necessary, this distance wa interpolated.
This figure demonstrates the same relationship as the previous figure, except that the distance is divided by the connectivity radius. By this measure the smallest conenctivity radius performs best which could be a result of its very large symmetry of connection.
Click here to see a detailed example for a connectivity radius of 17 with 5% percent external and 33 neurons between the centers of the sub-patterns.