We recently reduced the decoder network to its 2-dimensional essence: a network with 12 weights organized into two subnetworks, each with 6 weights (a fully-connected 2-input, 2-output network with biases). The network transforms its two inputs to two outputs using the subnetwork gated by a single input bit and the standard sigmoid function. By collecting all (x,y) states that the network can generate in the limit (in an evoked response to an infinite string of random gating bits), we get an image of its possible state space based solely on its weights, not its initial condition.
Using the "Blind Watchmaker" paradigm (Dawkins, 1987), where a human acts as the fitness function, we can explore the space of attractors, to find out what is possible, and to pick out "interesting" shapes, such as the "galaxy" on the next page. We have also used hillclimbing to match attractors to particular shapes, such as the circle on the next page, and optimized simple measures such as the "depth" of the attractor, resulting in the spiral on page after that, which takes a long time for reconstruction algorithms. For comparison, we also show a random set of weights and their attractor. Each figure is generated from the set of 12 weights given in the tables below.
These two-dimensional images are at the heart of understanding how to encode tree structures into neural activity patterns. The decoder of the RAAM architecture can also be seen as two nonlinear transforms applied non-deterministically as we decode a tree. Thus, applying them to any initial condition, reprsenting a given tree, will ultimately result in landing on the attractor determined by the weights of the decoder. The set of trees representable by the system is thus the set of equivalence classes over the transient structures from all possible initial conditions.