Neural networks and neural memory

Abstract

A neural pattern matcher is made up of an array of first sum and threshold SAT1 devices 18 each of which receives a number of inputs and a threshold value, and fires a 1 output if the number of inputs exceeds the threshold value. The outputs of the array of the SAT1 devices may be considered as a 2D image or generic template against which new data supplied into the registers 26 making up a data plane 24 are correlated at a correlation plane 20 of EX-NOR gates 22. The outputs of the EX-NOR gates themselves may be summed and thresholded by a seconded sum and threshold device 28 to provide a neural output ‘1’ or ‘0’ indicating match or no match. The matcher may therefore behave as a neural auto-associative memory which continually adapts to the input data to recognize data of a particular specified class.

Description

This invention relates to neural networks incorporating sum and threshold devices and in particular, but not exclusively to such networks capable of functioning as a neural pattern matcher. The invention also extends to sum and threshold devices for receiving weightless synaptic inputs and a weighted threshold value.

The apparatus and methods described herein may usefully incorporate, utilise, be used with or incorporated into any of the apparatus or methods described in our co-pending U.K. Patent Application No. 9726752.0 or our co-pending PCT Patent Applications Nos. PCT/GB98/______, PCT/GB98/______, PCT/GB98/______, (Our references 03-7127, XA1154, XA1156 and XA1000), the entire contents of which are incorporated herein by reference.

Terminology

The term “Hamming value” is used to define the number of bits set in 1-dimensional arrays such as a binary number, tuple, vector or 2 or higher dimensional arrays, that is the number of 1's set. The Hamming value relationship of two binary numbers or arrays indicates which has the greater Hamming value or whether the Hamming values are the same.

The term “weighted binary” is used in the conventional sense to indicate that successive bit positions are weighted, particularly . . . 16, 8, 4, 2, 1 although other weighted representations are possible. “Weightless binary” is a set of binary digits 1 and 0, each representing just “1” and “0” respectively. There is no least significant bit (LSB) or most significant bit (MSB). The set of bits may be ordered or without order. If all the 1's are grouped together e.g. [111000] then the code is referred to as a thermometer code, thermocode or bar graph code, all collectively referred to herein as “thermometer codes”. Equally, the term thermometer code is used broadly to cover 1 or higher dimensional arrays in which the set bits have been aggregated around a pre-set focal bit, which may be anywhere in the array.

A set of weightless bits is referred to herein as a “weightless tuple” or “weightless vector” and these terms are not intended to be restricted to ordered sets.

In traditional neural networks, a real-valued synaptic value is multiplied by a synaptic connection strength or weight value, and summed with other similarly treated synapses before they are all summed and thresholded to form a neural output. The weight value is a real-valued synaptic connection strength and hence the common usage of the term “weighted neural network”. However, it is also possible to have binary RAM-based neural networks that do not employ real-valued connection weights but instead rely on the values of the binary bits being either 0 or 1. Accordingly, there are two contexts of weightlessness: without synaptic connection strength, and without binary code weighting. The arrangements described herein employ weightless binary manipulation mechanisms and may be used to engineer weightless artificial neural networks, otherwise referred to as weightless-weightless artificial neural networks.

In one context, this invention is concerned with the comparison of two weightless vectors in terms of their Hamming values. This process is broadly equivalent to the function of a binary neuron. If the neuron receives a vector, A, of weightless synaptic values (e.g. [10110010]), and a vector, T, of weightless neural threshold values (e.g. [00101000]), the neuron may be required to fire because the Hamming value of A is greater than the Hamming value of T. In this example, the threshold, T, can be thought of as a set of inhibitory synaptic values which must be exceeded if the neuron is to be fired. The neural networks, devices, and techniques disclosed herein may be used in flight control systems, voting systems with redundancy, safety critical systems, telecommunications systems, decision making systems, and artificial intelligence systems, such as neural networks.

According to one aspect, this invention provides a neural network comprising:—

• • an array of bit memory means defining a neural memory for storing binary bits representing a plurality of exemplars,
  • an array of sum and threshold devices each for receiving as inputs respective bits from said bit memory means and for providing a preset output if the sum of said inputs exceeds a preset threshold, thereby to obtain a generic template representing said exemplars, and
  • means for comparing or correlating a set of input data with said generic template and providing an output representative of extent of matching between said set of input data and said generic template.

Preferably, said neural memory comprises means for storing a plurality of 2-dimensional data arrays. The neural network preferably includes means for presenting said input data in parallel to a data plane for correlation with said generic template. The means for correlating preferably comprises an array of logic elements. In a preferred embodiment the network includes further sum and threshold means for comparing the sum of said correlation results with a threshold and for providing an output representative of a match, if said sum exceeds said threshold.

The sum and threshold devices may take many forms but the, or at least one of the, sum and threshold devices preferably comprises a Hamming value comparator made up of a plurality of interconnected bit manipulation cells, each bit manipulation cell being operable to effect at least one of a bit shift and a bit elimination operation.

In another aspect, this invention provides a device for providing an output representative of a sum and threshold function performed on a weightless input and a threshold value, which comprises means for converting said weightless input into thermometer code (as herein defined), and means for monitoring the bit at a bit position corresponding to said threshold.

In a further aspect, this invention provides a neural network comprising:—

• • an array of bit memory means defining a neural network for storing binary bits representing a plurality of exemplars, and
  • an array of sum and threshold devices each for receiving as inputs respective bits from said bit memory means and for providing a preset output if the sum of said inputs exceeds a preset threshold, thereby to obtain a generic template representing said exemplars.

Whilst the invention has been described above, it extends to any inventive combination of the features set out above or in the following description.

The invention may be performed in various ways, and, by way of example only, various embodiments thereof will now be described in detail, reference being made to the accompanying drawings which utilise the conventional symbols for logic gates and in which:—

FIG. 1 is a schematic diagram of a sum and threshold (SAT) device;

FIG. 2 is a diagram of a neural network for processing data in accordance with this invention;

FIG. 3 is a diagram of a sum and threshold element with weightless synaptic inputs and a weighted binary threshold;

FIG. 4 is a diagram of a sum and threshold element of the type in FIG. 3, employing a thermometer code converter; and

FIG. 5 is a diagram of a bit manipulator cell used in the thermometer code converter of FIG. 4.

The embodiments described herein make use of sum and threshold devices, which may take many forms. Examples of novel Hamming value comparators which may serve as sum and threshold devices are described in our copending UK Patent Application No 9726752.0 and our copending International Patent Application No. PCT/GB98/______ (Our reference XA1154). Alternatively, the sum and threshold detectors may take the form of a binary thermometer code converter of the type described in our co-pending International Patent Application No. PCT/GB98/______ (our reference 03-7127) with a suitable output selector, as to be described below. The Hamming Comparators and Binary Code Converters described in these documents have the advantage that they can be implemented asynchronously, and thus be robust, fault tolerant and highly immune to RFI/EMI effects. However, of course, conventional sum and threshold devices may also be used in carrying out this invention.

Referring now to FIG. 1, neural data and a neural threshold are supplied to a Hamming value comparator 10 of one of the types discussed above, whether of one, two or of greater dimension. The input and threshold tuples are weightless. The output of the comparator 10 indicates whether the neural data has exceeded the neural threshold. The output is then viewed as the single output of the comparator or neuron which is taken to have “fired” if the bit has set. In this sense, the Hamming value comparator 10 acts as a binary neuron.

FIG. 2 shows an embodiment of a neural pattern matcher 12 which uses an array of sum and threshold devices, which may be those of the type just described, or any other suitable SAT element. For ease of visualisation, this is illustrated as a 3-dimensional array, comprising a neural memory block 14 of dimensions w×d×m, where ‘m’ is the number of bits in the digital input word, ‘w’ is the number of bits in the width of the input pattern and ‘d’ is the number of exemplars in the neural memory. The arrangement of neural memory is referred to elsewhere herein as neuroram.

The 3-dimensional array further includes a first sum and threshold region 16 (otherwise referred to as SAT columns) made up of (w×m) SAT devices 18 each marked SAT1 having d inputs (only one of these shown for clarity). Beneath the sum and threshold region 116 there is a correlation plane 20 made up of (w×m) 2 input EX-NOR gates 22 each of which receives an input from an associated SAT1 device 14 and an input from an associated data plane 24 which, optionally, may be held in separate bit memories 26. The outputs of the EX-NOR gates 22 in the correlation plane 20 are passed horizontally to a second sum and threshold region (referred to as a SAT plane), which consist of a single SAT2 device 28 with w×m inputs.

Respective thresholds T₁ of up to d bits are supplied to the SAT1 devices 18, and a threshold T₂ of up to w×m bits is supplied to the SAT2 device 28.

In use, incoming digital data is encoded using a unit Hamming distance code such as thermometer code, Gray code, or is in the form of a half tone bit map etc. Data for training (or for recognition after learning) is presented in parallel to the input of the data plane 24, or sequentially using a planar shift register (not shown). Optionally, the coded data may also be scrambled with a binary key string using EX-OR gates. The neural memory is programmed with ‘d’ exemplars each of w×m bits which can be prestored, learnt in a separate training routine, or adaptively altered during use.

The thresholds T₁ and T₂ determine the learning rate and quality factor of the neural pattern matcher/neural filter and are set according to fixed values, e.g. 66%. Alternatively the threshold may be adaptively set to determine the irritability (firing rate) of the system. Thresholds T₁ and T₂ may be thought of as a control over the degree of confidence attached to a match.

In considering operation of the device it is helpful to consider the action of one of the SAT1 devices. If the threshold T₁ for that particular device is set at 66% of the number of exemplars (e.g. T₁ is 8 if the neural memory is 12 bits deep), then the SAT1 device will provide a set bit or “fire” if there are more than 8 bits set in the column of neural memory above it. If T₁ is raised, then the SAT1 device will not fire until a greater proportion of the bits in the memory above it are set; in other words the degree of similarity of the bits in that column must be higher for the SAT1 to fire.

The outputs of all the (w×m) SAT1 devices may be considered as a generic template against which new data in the data plane is compared.

The threshold T₂ is an indicator of the degree of confidence insofar as the greater T₂, the greater the number of correlations there have to be between the input data plane and the generic template plane for the pattern to be accepted.

The update of the neural memory can be “supervised” during a training phase in which “d” exemplars or patterns of the required class are presented to the system and the neural memory then frozen. Alternatively, the update of neural memory can be unsupervised, so that the system continuously learns or adapts to incoming patterns after some initial patterns have been fed in. Initially it may be wise to set the neural memory to random values when used in an unsupervised mode of training. To prevent drifting, a portion of the memory may be made “read only”.

In this arrangement, the array may be regarded as a pattern matcher, or a category or class correlator. It is analogous in some respects to a cortical column in physiological neural systems, and a group of arrays or neurorams is analogous to a cortical map. A group of neurorams can be ordered linearly or in planes. Planar arrangements can be formed from sub-patterns of neurorams, such as triangles, squares, hexagons, etc. The array provides a hardware embodiment of a neural auto-associative memory.

As a modification of the sum and threshold technique described above, a Sum and Threshold element has been designed that accepts a weightless binary input and a weighted binary threshold. This Sum and Threshold element utilises a thermometer code converter, for example based on the thermometer code converter array described in our copending UK Patent Application 9726752.0 or International Patent Application PCT/GB98/______ (03-7127) and the appropriate output or outputs of the array are monitored or processed in accordance with the value of the threshold.

FIG. 3 shows a general arrangement of a sum and threshold element with weightless synaptic inputs and a weighted binary threshold. N weightless bits are supplied to a thermometer code converter 30 to obtain thermometer code N_t, which is passed to one or more selectors 32, each of which also receives a respective weighted threshold value T₁, T₂, etc. The selector 32 decodes the weighted threshold value and looks at the appropriate bit position in the thermometer code N_t, and if set (indicating that the thermometer code has a Hamming value greater than the specified threshold), the selector sets its output bit to indicate that N>T.

FIG. 4 is a circuit diagram of an example of a sum and threshold device having eight synaptic inputs and a choice of four thresholds. The device comprises a thermometer code converter section 34, made up of 2-bit manipulator cells 36 of the type shown in FIG. 5. The 2-bit manipulator cells each comprise an OR gate 38 and an AND gate 40, interconnected such that inputs a,b map to outputs Ya,Yb as follows:—

• • Ya=A OR B
  • Yb=A AND B

It should be noted that in the device of FIG. 4, there are eight inputs, thus requiring odd layers nominally of four 2-bit manipulator cells 36 wide and even layers nominally of three 2-bit manipulator cells 38 wide, making up eight layers in all, although the fifth to eighth layers have been truncated in this case.

The thresholds in this example are I>6, I>5, I>4 and I>3, meaning that only the fourth to seventh outputs (from the bottom of the array as viewed) are required. Because of the truncation, the fifth to seventh layers include AND is gates 42 at the lower truncation boundary, and the seventh layer includes an OR 44 gate.

The output of the thermometer code conversion section 34 passes to a weighted binary selector 46 which acts as a threshold decoder. Such devices are already known for use as binary multiplexers or logical selectors. In this example, which allows selection of one of four threshold values 3, 4, 5, 6, the selector 40 comprises two weighted inputs 48, 50 which are each connected to the inputs of 3-input AND gates 52, the other input of each AND gate being a respective output from the thermometer code conversion section 34. Selected terminals of the lower three AND gates are inverted, and the outputs of the AND gates pass to an OR gate 54. Different permutations of 0's and 1's applied to the weighted inputs select different bit positions at the output of the thermometer code conversion section 34.

The selector 46 has the following mapping:—

Inputs	Thermometer Code
T1	T0	Bit Position
0	0	4
0	1	5
1	0	6
1	1	7

Thus if the weighted input is (1,0) the device will fire only if the Hamming value of the weightless input is greater than 5.

If two or more thresholds are to be determined, then further weighted binary selectors could be connected to the output of the thermometer code converter, as shown in FIG. 3.

It will be appreciated also that the circuit could be simplified to respond to a given specific threshold; in this instance a binary selector as such would not be required and instead the output of the thermometer code converter corresponding to T_fixed+1 where T_fixed is the fixed threshold, would be the output.

The devices described above provide robust hardware implementation which is fault tolerant due to its weightless techniques.

In general, the implementation of the above arrangement is technology independent; electronic, ionic, magnetic or electromagnetic (e.g. optical) implementation are all suitable.

Claims

1-7. (Cancelled)

8. A device for providing an output representative of a sum and threshold function performed on a weightless input and a threshold value, which comprises means for converting said weightless input into thermometer code as herein defined, and means for monitoring the bit at a bit position corresponding to said threshold.

9. A device for providing an output representative of a sum and threshold function performed on a weightless input and a threshold value, which comprises a converter for converting said weightless input into thermometer code as herein defined, and a monitor for monitoring the bit at a bit position corresponding to said threshold.