Sparse distributed memory

From Wikipedia, the free encyclopedia

https://en.wikipedia.org/wiki/Sparse_distributed_memory 

Sparse distributed memory (SDM) is a mathematical model of human long-term memory introduced by Pentti Kanerva in 1988 while he was at NASA Ames Research Center. It is a generalized random-access memory (RAM) for long (e.g., 1,000 bit) binary words. These words serve as both addresses to and data for the memory. The main attribute of the memory is sensitivity to similarity, meaning that a word can be read back not only by giving the original write address but also by giving one close to it, as measured by the number of mismatched bits (i.e., the Hamming distance between memory addresses).

SDM implements transformation from logical space to physical space using distributed data representation and storage, similarly to encoding processes in human memory. A value corresponding to a logical address is stored into many physical addresses. This way of storing is robust and not deterministic. A memory cell is not addressed directly. If input data (logical addresses) are partially damaged at all, we can still get correct output data.

The theory of the memory is mathematically complete and has been verified by computer simulation. It arose from the observation that the distances between points of a high-dimensional space resemble the proximity relations between concepts in human memory. The theory is also practical in that memories based on it can be implemented with conventional random-access memory elements.

Definition

Human memory has a tendency to congregate memories based on similarities between them (although they may not be related), such as "firetrucks are red and apples are red". Sparse distributed memory is a mathematical representation of human memory, and uses high-dimensional space to help model the large amounts of memory that mimics that of the human neural network. An important property of such high dimensional spaces is that two randomly chosen vectors are relatively far away from each other, meaning that they are uncorrelated. SDM can be considered a realization of Locality-sensitive hashing.

The underlying idea behind a SDM is the mapping of a huge binary memory onto a smaller set of physical locations, so-called hard locations. As a general guideline, those hard locations should be uniformly distributed in the virtual space, to mimic the existence of the larger virtual space as accurately as possible. Every datum is stored distributed by a set of hard locations, and retrieved by averaging those locations. Therefore, recall may not be perfect, accuracy depending on the saturation of the memory.

Kanerva's proposal is based on four basic ideas:

• 1. The boolean space ${\displaystyle \{0,1\}^{n}}$, or ${\displaystyle 2^{n}}$ points in ${\displaystyle 10^{0}<n<10^{5}}$ dimensions, exhibits properties which are similar to humans' intuitive notions of relationships between the concepts. This means that it makes sense to store data as points of the mentioned space where each memory item is stored as an n-bit vector.
• 2. Neurons with n inputs can be used as address decoders of a random-access memory
• 3. Unifying principle: data stored into the memory can be used as addresses to the same memory. Distance between two points is a measure of similarity between two memory items. The closer the points, the more similar the stored vectors.
• 4. Time can be traced in the memory as a function of where the data are stored, if the data are organized as sequences of events

The binary space N

The SDM works with n-dimensional vectors with binary components. Depending on the context, the vectors are called points, patterns, addresses, words, memory items, data, or events. This section is mostly about the properties of the vector space N = ${\displaystyle \{0,1\}^{n}}$. Let n be number of dimensions of the space. The number of points, or possible memory items, is then ${\displaystyle 2^{n}}$. We will denote this number by N and will use N and ${\displaystyle 2^{n}}$ to stand also for the space itself.

Concepts Related to the space N: ${\displaystyle \{0,1\}^{n}}$

• Origin, 0: The point with all coordinates 0 is called the origin, 0 = 000...00.
• Complement, 'x: The complement, or opposite, of point x is the n-tuple that has ones where x has zeros and vice versa.
• Norm, |x|: The norm of point x is the number of ones in its binary representation.
• Difference, x − y: The difference of two points x and y is the n-tuple that has ones where x and y differ and zeros elsewhere. It is the bitwise 'exclusive or': x − y = x ⊕ y. The difference commutes: x − y = y − x.
• Distance, d(x, y) The distance between two points x and y is the number of dimensions at which x and y differ. It is called the Hamming distance (its square root is the Euclidean distance) and is expressed in bits. Distance is the norm of the difference: d(x, y) = |x − y|
• Betweenness, x:y:z: Point y is between points x and z if and only if the distance from x to z is the sum of the distances from x to y and from y to z; that is, x:y:z ⇔ d(x, z) = d(x, y) + d(y, z). It is easily seen that every bit of a point in between is a copy of the corresponding bit of an endpoint.
• Orthogonality, x ⊥ y: Point x is orthogonal to point y, or the two are perpendicular or indifferent, if and only if the distance between the two is half the number of dimensions: x ⊥ y ⇔ d(x, y) = n/2. The distance n/2 is called indifference distance of space N. If x is orthogonal to y, it is also orthogonal to its complement 'y (x is halfway between y and 'y).
• Circle, O(r,x) A circle with radius r and center x is the set of points that are at most r bits from x: O(r,x) = {y | d(x, y) ≤ r}.

Properties of the space N: ${\displaystyle \{0,1\}^{n}}$

The space N can be represented by the vertices of the unit cube in n-dimensional Euclidean space. The vertices lie on the surface of an n-dimensional sphere with (Euclidean-metric) radius ${\displaystyle {\sqrt {n}}/2}$. This gives rise to the sphere analogy. We will call a space spherical if

• 1. any point x has a unique opposite 'x,
• 2. the entire space is between any point x and its opposite 'x, and
• 3. all points are "equal" (meaning that for any two points x and y there is a distance preserving automorphism of the space that maps x to y, so that from any of its points the space "looks" the same).

The surface of a sphere (in Euclidean 3d-space) clearly is spherical. According to definition, N is also spherical, since y ⊕ x ⊕ (…) is an automorphism that maps x to y. Because N is spherical, it is helpful to think of it as the surface of a sphere with circumference 2n. All points of N are equally qualified as points of origin, and a point and its complement are like two poles at distance n from each other, with the entire space in between. The points halfway between the poles and perpendicular to them are like the equator.

Distribution of the space N

The number of points that are exactly d bits from an arbitrary point x (say, from the point 0) is the number of ways to choose d coordinates from a total of n coordinates, and is therefore given by the binomial coefficient: ${\displaystyle {\binom {n}{d}}={\frac {n!}{d!(n-d)!}}}$

The distribution of N thus is the binomial distribution with parameters n and p, where p = 1/2. The mean of the binomial distribution is n/2, and the variance is n/4. This distribution function will be denoted by N(d). The normal distribution F with mean n/2 and standard deviation ${\displaystyle {\sqrt {n}}/2}$ is a good approximation to it: N(d) = Pr{d(x, y) ≤ d} ≅ F{(d − n / 2)/ ${\displaystyle {\sqrt {n/4}}}$}

Tendency to orthogonality

An outstanding property of N is that most of it lies at approximately the mean (indifference) distance n/2 from a point (and its complement). In other words, most of the space is nearly orthogonal to any given point, and the larger n is, the more pronounced is this effect.

As neural network

The SDM may be regarded either as a content-addressable extension of a classical random-access memory (RAM) or as a special type of three layer feedforward neural network. The main SDM alterations to the RAM are:

• The SDM calculates Hamming distances between the reference address and each location address. For each distance which is less or equal to the given radius the corresponding location is selected.
• The memory is represented by counters (where n is number of locations and m is the input data length) instead of single-bit storage elements.
• Writing to the memory, instead of overwriting, is as follows:
  • if the i-bit of the input data is 1, the corresponding counters (counters in the selected locations (rows) and in the i-th columns) are incremented,
  • if the i-bit of the input data is 0, the corresponding counters are decremented.
• Reading (or recall) from the memory is similar:
  • The contents of the selected locations are summed columnwise.
  • Each sum is thresholded. If the sum is greater than or equal to the threshold value the corresponding output bit is set to 1, in the opposite case it is cleared. Note that the thresholds may be zero, if the training input vectors are closed to orthogonal ones.

Neuron model

An idealized description of neuron is as follows: a neuron has a cell body with two kinds of branches: dendrites and an axon. It receives input signals from other neurons via dendrites, integrates (sums) them and generates its own (electric) output signal which is sent to outside neurons via axon. The points of electric contact between neurons are called synapses.

When a neuron generates signal it is firing and after firing it must recover before it fires again. The relative importance of a synapse to the firing of neuron is called synaptic weight (or input coefficient). There are two kinds of synapses: excitatory that trigger neuron to fire and inhibitory that hinder firing. The neuron is either excitatory or inhibitory according to the kinds of synapses its axon makes.

A neuron fires when the sum of inputs exceed a specific threshold. The higher the threshold the more important it is that excitatory synapses have input while inhibitory ones do not. Whether a recovered neuron actually fires depends on whether it received sufficient excitatory input (beyond the threshold) and not too much of inhibitory input within a certain period.

The formal model of neuron makes further simplifying assumptions. An n-input neuron is modeled by a linear threshold function ${\displaystyle F:\{0,1\}^{n}->\{0,1\}}$ as follows :

For ${\displaystyle i=0,...,n-1}$ where n is the number of inputs, let ${\displaystyle F_{t}}$ be the output at time t: ${\displaystyle F_{t}\in \{0,1\}}$, and let ${\displaystyle x_{i,t}}$ be the i-th input at time t: ${\displaystyle x_{i,t}\in \{0,1\}}$. Let ${\displaystyle w_{i}}$ be the weight of the i-th input and let ${\displaystyle c}$ be the threshold.

The weighted sum of the inputs at time t is defined by ${\displaystyle S_{t}=\sum _{i=0}^{n-1}w_{i}x_{i,t}}$

The neuron output at time t is then defined as a boolean function: ${\displaystyle \mathbf {F} _{t}={\begin{cases}1&{\text{if }}S_{t}>=c,\\0&{\text{otherwise }}.\end{cases}}}$

Where F_t=1 means that the neuron fires at time t and F_t=0 that it doesn't, i.e. in order for neuron to fire the weighted sum must reach or exceed the threshold . Excitatory inputs increase the sum and inhibitory inputs decrease it.

Neuron as address-decoder

Kanerva's key thesis is that certain neurons could have their input coefficients and thresholds fixed over the entire life of an organism and used as address decoders where n-tuple of input coefficients (the pattern to which neurons respond most readily) determines the n-bit memory address, and the threshold controls the size of the region of similar address patterns to which the neuron responds.

This mechanism is complementary to adjustable synapses or adjustable weights in a neural network (perceptron convergence learning), as this fixed accessing mechanism would be a permanent frame of reference which allows to select the synapses in which the information is stored and from which it is retrieved under given set of circumstances. Furthermore, an encoding of the present circumstance would serve as an address.

The address a of a neuron with input coefficients w where ${\displaystyle w_{0},..,w_{n_{1}}}$ is defined as an n-bit input pattern that maximizes the weighted sum. The maximum occurs when the inhibitory inputs are zeros and the excitatory inputs are ones. The i-th bit of address is:

${\displaystyle \mathbf {a} _{i}={\begin{cases}1&{\text{if }}w_{i}>0,\\0&{\text{if }}w_{i}<0.\end{cases}}}$ (assuming weights are non-zero)

The maximum weighted sum ${\displaystyle S(w)}$ is then the sum of all positive coefficients: ${\displaystyle S(w)=\sum _{w_{i}>0}w_{i}}$

And the minimum weighted sum ${\displaystyle s(w)}$ would correspond to a point opposite the neuron address a`: ${\displaystyle s(w)=\sum _{w_{i}<0}w_{i}}$

When the threshold c is in range ${\displaystyle s(w)<c<S(w)}$ the output of the neuron is 0 for some addresses (input patterns) and 1 for others. If the threshold is above S the output is always 0, if it's below s the output is always 1. So by a proper choice of the threshold a neuron responds only to just one address. When the threshold is S (the maximum for the weighted sum) the neuron responds only to its own address and acts like an address decoder of a conventional random-access memory.

Memory location

SDM is designed to cope with address patterns that span an enormous address space (order of ${\displaystyle 2^{1000}}$). SDM assumes that the address patterns actually describing physical situations of interest are sparsely scattered throughout the input space. It is impossible to reserve a separate physical location corresponding to each possible input; SDM implements only a limited number of physical or hard locations. The physical location is called a memory (or hard) location.

Every hard location has associated with it two items:

• a fixed hard address, which is the N-bit address of the location
• a contents portion that is M-bits wide and that can accumulate multiple M-bit data patterns written into the location. The contents' portion is not fixed; it is modified by data patterns written into the memory.

In SDM a word could be stored in memory by writing it in a free storage location and at the same time providing the location with the appropriate address decoder. A neuron as an address decoder would select a location based on similarity of the location's address to the retrieval cue. Unlike conventional Turing machines SDM is taking advantage of parallel computing by the address decoders. The mere accessing the memory is regarded as computing, the amount of which increases with memory size.

Address pattern

An N-bit vector used in writing to and reading from the memory. The address pattern is a coded description of an environmental state. (e.g. N = 256.)

Data pattern

An M-bit vector that is the object of the writing and reading operations. Like the address pattern, it is a coded description of an environmental state. (e.g. M = 256.)

Writing

Writing is the operation of storing a data pattern into the memory using a particular address pattern. During a write, the input to the memory consists of an address pattern and a data pattern. The address pattern is used to select hard memory locations whose hard addresses are within a certain cutoff distance from the address pattern. The data pattern is stored into each of the selected locations.

Reading

Reading is the operation of retrieving a data pattern from the memory using a particular address pattern. During a read, an address pattern is used to select a certain number of hard memory locations (just like during a write). The contents of the selected locations are bitwise summed and thresholded to derive an M-bit data pattern. This serves as the output read from the memory.

Pointer chains

All of the items are linked in a single list (or array) of pointers to memory locations, and are stored in RAM. Each address in an array points to an individual line in the memory. That line is then returned if it is similar to other lines. Neurons are utilized as address decoders and encoders, similar to the way neurons work in the brain, and return items from the array that match or are similar.

Critical distance

Kanerva's model of memory has a concept of a critical point: prior to this point, a previously stored item can be easily retrieved; but beyond this point an item cannot be retrieved. Kanerva has methodically calculated this point for a particular set of (fixed) parameters. The corresponding critical distance of a Sparse Distributed Memory can be approximately evaluated minimizing the following equation with the restriction ${\displaystyle d\in N}$ and ${\displaystyle d\leqslant n}$. The proof can be found in,

${\displaystyle {\tilde {f}}(d)=\left\{{\frac {1}{2}}\cdot \left[1-N\left(z<{\frac {w\cdot shared(d)}{\sqrt {\theta }}}\right)+N\left(z<{\frac {-w\cdot shared(d)}{\sqrt {\theta }}}\right)\right]-{\frac {d}{n}}\right\}^{2}}$

Where:

• ${\displaystyle d}$: is the distance to the target;
• ${\displaystyle n}$: is the number of dimensions;
• ${\displaystyle N}$: is the normalized normal distribution with mean zero and variance one;
• ${\displaystyle w}$: is the number of times the target bitstring was written in memory;
• ${\displaystyle \theta }$: is the total of random bitstrings in all ${\displaystyle h}$ hard-locations activated by a read operation; i.e., the size of a cell assembly;
• ${\displaystyle shared(d)}$: is the mean number of shared hard-locations activated by two bitstrings ${\displaystyle d}$ bits away from each other. One can find some values for a 1000-dimensional SDM in Kanerva's book, Table 7.1, p. 63, or the equations to calculate to any SDM in Appendix B, p. 125 of the same book.

Probabilistic interpretation

An associative memory system using sparse, distributed representations can be reinterpreted as an importance sampler, a Monte Carlo method of approximating Bayesian inference. The SDM can be considered a Monte Carlo approximation to a multidimensional conditional probability integral. The SDM will produce acceptable responses from a training set when this approximation is valid, that is, when the training set contains sufficient data to provide good estimates of the underlying joint probabilities and there are enough Monte Carlo samples to obtain an accurate estimate of the integral.

Biological plausibility

Sparse coding may be a general strategy of neural systems to augment memory capacity. To adapt to their environments, animals must learn which stimuli are associated with rewards or punishments and distinguish these reinforced stimuli from similar but irrelevant ones. Such task requires implementing stimulus-specific associative memories in which only a few neurons out of a population respond to any given stimulus and each neuron responds to only a few stimuli out of all possible stimuli.

Theoretical work on SDM by Kanerva has suggested that sparse coding increases the capacity of associative memory by reducing overlap between representations. Experimentally, sparse representations of sensory information have been observed in many systems, including vision, audition, touch, and olfaction. However, despite the accumulating evidence for widespread sparse coding and theoretical arguments for its importance, a demonstration that sparse coding improves the stimulus-specificity of associative memory has been lacking until recently.

Some progress has been made in 2014 by Gero Miesenböck's lab at the University of Oxford analyzing Drosophila Olfactory system. In Drosophila, sparse odor coding by the Kenyon cells of the mushroom body is thought to generate a large number of precisely addressable locations for the storage of odor-specific memories. Lin et al. demonstrated that sparseness is controlled by a negative feedback circuit between Kenyon cells and the GABAergic anterior paired lateral (APL) neuron. Systematic activation and blockade of each leg of this feedback circuit show that Kenyon cells activate APL and APL inhibits Kenyon cells. Disrupting the Kenyon cell-APL feedback loop decreases the sparseness of Kenyon cell odor responses, increases inter-odor correlations, and prevents flies from learning to discriminate similar, but not dissimilar, odors. These results suggest that feedback inhibition suppresses Kenyon cell activity to maintain sparse, decorrelated odor coding and thus the odor-specificity of memories. A 2017 publication in Science showed that fly olfactory circuit implements an improved version of binary locality sensitive hashing via sparse, random projections.

Applications

In applications of the memory, the words are patterns of features. Some features are produced by a sensory system, others control a motor system. There is a current pattern (of e.g. 1000 bits), which is the current contents of the system's focus. The sensors feed into the focus, the motors are driven from the focus, and the memory is accessed through the focus.

What goes on in the world – the system's "subjective" experience – is represented internally by a sequence of patterns in the focus. The memory stores this sequence and can recreate it later in the focus if addressed with a pattern similar to one encountered in the past. Thus, the memory learns to predict what is about to happen. Wide applications of the memory would be in systems that deal with real-world information in real time.

The applications include vision – detecting and identifying objects in a scene and anticipating subsequent scenes – robotics, signal detection and verification, and adaptive learning and control. On the theoretical side, the working of the memory may help us understand memory and learning in humans and animals.

The Best Match Search

SDM can be applied to the problem of finding the best match to a test word in a dataset of stored words. Or, in other words, the Nearest neighbor search problem.

Consider a memory with N locations where ${\displaystyle N=2^{n}}$. Let each location have the capacity for one n-bit word (e.g. N= 2¹⁰⁰ 100-bit words), and let the address decoding be done by N address decoder neurons. Set the threshold of each neuron x to its maximum weighted sum ${\displaystyle |x|}$ and use a common parameter d to adjust all thresholds when accessing the memory. The effective threshold of neuron x will be then ${\displaystyle x-|d|}$ which means that the location x is accessible every time the address x is within d bits of the address presented to memory (i.e. the address held by the address register). With ${\displaystyle d=0}$ we have a conventional random-access memory. Assume further that each location has a special location-occupied bit that can be accessed in the same way as the regular datum bits. Writing a word to a location sets this location-occupied bit. Assume that only occupied location can be read.

To file the data in memory, start by setting ${\displaystyle d=n}$ and issue a command to clear the location-occupied bit. This single operation marks all memory as unoccupied regardless of the values of the address register. Then set ${\displaystyle d=0}$ and write each word y of the data set with y itself as the address. Notice that each write operation affects only one location: the location y. Filing time is thus proportional to the number of words in the dataset.

Finding the best match for a test word z, involves placing z in the address register and finding the least distance d for which there is an occupied location. We can start the search by setting ${\displaystyle d=0}$ and incrementing d successively until an occupied location is found. This method gives average search times that are proportional to the number of address bits or slightly less than ${\displaystyle n/2}$ because the nearest occupied location can be expected to be just under ${\displaystyle n/2}$ bits from z (with binary search on d this would be O(log(n)).

With 100-bit words 2¹⁰⁰ locations would be needed, i.e. an enormously large memory. However if we construct the memory as we store the words of the dataset we need only one location (and one address decoder) for each word of the data set. None of the unoccupied locations need to be present. This represents the aspect of sparseness in SDM.

Speech recognition

SDM can be applied in transcribing speech, with the training consisting of "listening" to a large corpus of spoken language. Two hard problems with natural speech are how to detect word boundaries and how to adjust to different speakers. The memory should be able to handle both. First, it stores sequences of patterns as pointer chains. In training – in listening to speech – it will build a probabilistic structure with the highest incidence of branching at word boundaries. In transcribing speech, these branching points are detected and tend to break the stream into segments that correspond to words. Second, the memory's sensitivity to similarity is its mechanism for adjusting to different speakers – and to the variations in the voice of the same speaker.

"Realizing forgetting"

Decay Functions

The exponential decay function

The negated-translated sigmoid function

At the University of Memphis, Uma Ramamurthy, Sidney K. D'Mello, and Stan Franklin created a modified version of the sparse distributed memory system that represents "realizing forgetting." It uses a decay equation to better show interference in data. The sparse distributed memory system distributes each pattern into approximately one hundredth of the locations, so interference can have detrimental results.

Two possible examples of decay from this modified sparse distributed memory are presented

Exponential decay mechanism: ${\displaystyle \!f(x)=1+e^{-ax}}$

Negated-translated sigmoid decay mechanism: ${\displaystyle f(x)=1-[{\frac {1}{1+e^{-a(x-c)}}}]}$

In the exponential decay function, it approaches zero more quickly as x increases, and a is a constant (usually between 3-9) and c is a counter. For the negated-translated sigmoid function, the decay is similar to the exponential decay function when a is greater than 4.

As the graph approaches 0, it represents how the memory is being forgotten using decay mechanisms.

Genetic sparse distributed memory

Ashraf Anwar, Stan Franklin, and Dipankar Dasgupta at The University of Memphis; proposed a model for SDM initialization using Genetic Algorithms and Genetic Programming (1999).

Genetic memory uses genetic algorithm and sparse distributed memory as a pseudo artificial neural network. It has been considered for use in creating artificial life.

Statistical prediction

SDM has been applied to statistical prediction, the task of associating extremely large perceptual state vectors with future events. In conditions of near- or over- capacity, where the associative memory behavior of the model breaks down, the processing performed by the model can be interpreted as that of a statistical predictor and each data counter in an SDM can be viewed as an independent estimate of the conditional probability of a binary function f being equal to the activation set defined by the counter's memory location.

Artificial general intelligence

• LIDA uses sparse distributed memory to help model cognition in biological systems. The sparse distributed memory places space is recalling or recognizing the object that it has in relation to other objects. It was developed by Stan Franklin, the creator of the "realizing forgetting" modified sparse distributed memory system. Transient episodic and declarative memories have distributed representations in LIDA (based on modified version of SDM), there is evidence that this is also the case in the nervous system.
• CMatie is a 'conscious' software agent developed to manage seminar announcements in the Mathematical Sciences Department at the University of Memphis. It's based on SDM augmented with the use of genetic algorithms as an associative memory.
• Hierarchical temporal memory utilizes SDM for storing sparse distributed representations of the data.

(Also see Cognitive architecture & Artificial General Intelligence for a list of SDM related projects)

Reinforcement learning

SDMs provide a linear, local function approximation scheme, designed to work when a very large/high-dimensional input (address) space has to be mapped into a much smaller physical memory. In general, local architectures, SDMs included, can be subject to the curse of dimensionality, as some target functions may require, in the worst case, an exponential number of local units to be approximated accurately across the entire input space. However, it is widely believed that most decision-making systems need high accuracy only around low-dimensional manifolds of the state space, or important state "highways". The work in Ratitch et al. combined the SDM memory model with the ideas from memory-based learning, which provides an approximator that can dynamically adapt its structure and resolution in order to locate regions of the state space that are "more interesting" and allocate proportionally more memory resources to model them accurately.

Object indexing in computer vision

Dana H. Ballard's lab demonstrated a general-purpose object indexing technique for computer vision that combines the virtues of principal component analysis with the favorable matching properties of high-dimensional spaces to achieve high precision recognition. The indexing algorithm uses an active vision system in conjunction with a modified form of SDM and provides a platform for learning the association between an object's appearance and its identity.

Extensions

Many extensions and improvements to SDM have been proposed, e.g.:

• Ternary memory space: This enables the memory to be used as a Transient Episodic Memory (TEM) in cognitive software agents. TEM is a memory with high specificity and low retention, used for events having features of a particular time and place.
• Integer SDM that uses modular arithmetic integer vectors rather than binary vectors. This extension improves the representation capabilities of the memory and is more robust over normalization. It can also be extended to support forgetting and reliable sequence storage.
• Using word vectors of larger size than address vectors: This extension preserves many of the desirable properties of the original SDM: auto-associability, content addressability, distributed storage and robustness over noisy inputs. In addition, it adds new functionality, enabling an efficient auto-associative storage of sequences of vectors, as well as of other data structures such as trees.
• Constructing SDM from Spiking Neurons: Despite the biological likeness of SDM most of the work undertaken to demonstrate its capabilities to date has used highly artificial neuron models which abstract away the actual behaviour of neurons in the brain. Recent work by Steve Furber's lab at the University of Manchester proposed adaptations to SDM, e.g. by incorporating N-of-M rank codes into how populations of neurons may encode information—which may make it possible to build an SDM variant from biologically plausible components. This work has been incorporated into SpiNNaker (Spiking Neural Network Architecture) which is being used as the Neuromorphic Computing Platform for the Human Brain Project.
• Non-random distribution of locations: Although the storage locations are initially distributed randomly in the binary N address space, the final distribution of locations depends upon the input patterns presented, and may be non-random thus allowing better flexibility and generalization. The data pattern is first stored at locations which lie closest to the input address. The signal (i.e. data pattern) then spreads throughout the memory, and a small percentage of the signal strength (e.g. 5%) is lost at each subsequent location encountered. Distributing the signal in this way removes the need for a select read/write radius, one of the problematic features of the original SDM. All locations selected in a write operation do not now receive a copy of the original binary pattern with equal strength. Instead they receive a copy of the pattern weighted with a real value from 1.0->0.05 to store in real valued counters (rather than binary counters in Kanerva's SDM). This rewards the nearest locations with a greater signal strength, and uses the natural architecture of the SDM to attenuate the signal strength. Similarly in reading from the memory, output from the nearest locations is given a greater weight than from more distant locations. The new signal method allows the total signal strength received by a location to be used as a measure of the fitness of a location and is flexible to varying input (as the loss factor does not have to be changed for input patterns of different lengths).
• SDMSCue (Sparse Distributed Memory for Small Cues): Ashraf Anwar & Stan Franklin at The University of Memphis, introduced a variant of SDM capable of Handling Small Cues; namely SDMSCue in 2002. The key idea is to use multiple Reads/Writes, and space projections to reach a successively longer cue.

Related patents

• Method and apparatus for a sparse distributed memory system US 5113507 A, Universities Space Research Association, 1992
• Method and device for storing and recalling information implementing a kanerva memory system US 5829009 A, Texas Instruments, 1998
• Digital memory, Furber, Stephen. US 7512572 B2, 2009

Implementation

• C Binary Vector Symbols (CBVS): includes SDM implementation in C as a part of vector symbolic architecture developed by EISLAB at Luleå University of Technology: http://pendicular.net/cbvs.php
• CommonSense ToolKit (CSTK) for realtime sensor data processing developed at the Lancaster University includes implementation of SDM in C++: http://cstk.sourceforge.net/
• Julia implementation by Brian Hayes: https://github.com/bit-player/sdm-julia 
• Learning Intelligent Distribution Agent (LIDA) developed by Stan Franklin's lab at the University of Memphis includes implementation of SDM in Java: http://ccrg.cs.memphis.edu/framework.html
• Python implementation: https://github.com/msbrogli/sdm
• Python and OpenCL implementation: https://github.com/msbrogli/sdm-framework
• APL implementation
• LISP implementation for the Connection Machine
• FPGA implementation
• The original hardware implementation developed by NASA
• An Implementation in C done at the Research Institute for Advanced Computer Science at NASA Ames

Oceanography

From Wikipedia, the free encyclopedia

https://en.wikipedia.org/wiki/Oceanography 

Thermohaline circulation

Oceanography (from Ancient Greek ὠκεανός (ōkeanós) 'ocean', and γραφή (graphḗ) 'writing'), also known as oceanology and ocean science, is the scientific study of the oceans. It is an important Earth science, which covers a wide range of topics, including ecosystem dynamics; ocean currents, waves, and geophysical fluid dynamics; plate tectonics and the geology of the sea floor; and fluxes of various chemical substances and physical properties within the ocean and across its boundaries. These diverse topics reflect multiple disciplines that oceanographers utilize to glean further knowledge of the world ocean, including astronomy, biology, chemistry, climatology, geography, geology, hydrology, meteorology and physics. Paleoceanography studies the history of the oceans in the geologic past. An oceanographer is a person who studies many matters concerned with oceans, including marine geology, physics, chemistry and biology.

History

Map of the Gulf Stream by Benjamin Franklin, 1769–1770. Courtesy of the NOAA Photo Library.

Early history

Humans first acquired knowledge of the waves and currents of the seas and oceans in pre-historic times. Observations on tides were recorded by Aristotle and Strabo in 384-322 BC. Early exploration of the oceans was primarily for cartography and mainly limited to its surfaces and of the animals that fishermen brought up in nets, though depth soundings by lead line were taken.

The Portuguese campaign of Atlantic navigation is the earliest example of a systematic scientific large project, sustained over many decades, studying the currents and winds of the Atlantic.

The work of Pedro Nunes (1502-1578) is remembered in the navigation context for the determination of the loxodromic curve: the shortest course between two points on the surface of a sphere represented onto a two-dimensional map. When he published his "Treatise of the Sphere" (1537), mostly a commentated translation of earlier work by others, he included a treatise on geometrical and astronomic methods of navigation. There he states clearly that Portuguese navigations were not an adventurous endeavour:

"nam se fezeram indo a acertar: mas partiam os nossos mareantes muy ensinados e prouidos de estromentos e regras de astrologia e geometria que sam as cousas que os cosmographos ham dadar apercebidas (...) e leuaua cartas muy particularmente rumadas e na ja as de que os antigos vsauam" (were not done by chance: but our seafarers departed well taught and provided with instruments and rules of astrology (astronomy) and geometry which were matters the cosmographers would provide (...) and they took charts with exact routes and no longer those used by the ancient).

His credibility rests on being personally involved in the instruction of pilots and senior seafarers from 1527 onwards by Royal appointment, along with his recognized competence as mathematician and astronomer. The main problem in navigating back from the south of the Canary Islands (or south of Boujdour) by sail alone, is due to the change in the regime of winds and currents: the North Atlantic gyre and the Equatorial counter current  will push south along the northwest bulge of Africa, while the uncertain winds where the Northeast trades meet the Southeast trades (the doldrums)  leave a sailing ship to the mercy of the currents. Together, prevalent current and wind make northwards progress very difficult or impossible. It was to overcome this problem and clear the passage to India around Africa as a viable maritime trade route, that a systematic plan of exploration was devised by the Portuguese. The return route from regions south of the Canaries became the 'volta do largo' or 'volta do mar'. The 'rediscovery' of the Azores islands in 1427 is merely a reflection of the heightened strategic importance of the islands, now sitting on the return route from the western coast of Africa (sequentially called 'volta de Guiné' and 'volta da Mina'); and the references to the Sargasso Sea (also called at the time 'Mar da Baga'), to the west of the Azores, in 1436, reveals the western extent of the return route. This is necessary, under sail, to make use of the southeasterly and northeasterly winds away from the western coast of Africa, up to the northern latitudes where the westerly winds will bring the seafarers towards the western coasts of Europe.

The secrecy involving the Portuguese navigations, with the death penalty for the leaking of maps and routes, concentrated all sensitive records in the Royal Archives, completely destroyed by the Lisbon earthquake of 1775. However, the systematic nature of the Portuguese campaign, mapping the currents and winds of the Atlantic, is demonstrated by the understanding of the seasonal variations, with expeditions setting sail at different times of the year taking different routes to take account of seasonal predominate winds. This happens from as early as late 15th century and early 16th: Bartolomeu Dias followed the African coast on his way south in August 1487, while Vasco da Gama would take an open sea route from the latitude of Sierra Leone, spending 3 months in the open sea of the South Atlantic to profit from the southwards deflection of the southwesterly on the Brazilian side (and the Brazilian current going southward) - Gama departed in July 1497); and Pedro Alvares Cabral, departing March 1500) took an even larger arch to the west, from the latitude of Cape Verde, thus avoiding the summer monsoon (which would have blocked the route taken by Gama at the time he set sail). Furthermore, there were systematic expeditions pushing into the western Northern Atlantic (Teive, 1454; Vogado, 1462; Teles, 1474; Ulmo, 1486). The documents relating to the supplying of ships, and the ordering of sun declination tables for the southern Atlantic for as early as 1493–1496, all suggest a well-planned and systematic activity happening during the decade long period between Bartolomeu Dias finding the southern tip of Africa, and Gama's departure; additionally, there are indications of further travels by Bartolomeu Dias in the area. The most significant consequence of this systematic knowledge was the negotiation of the Treaty of Tordesillas in 1494, moving the line of demarcation 270 leagues to the west (from 100 to 370 leagues west of the Azores), bringing what is now Brazil into the Portuguese area of domination. The knowledge gathered from open sea exploration allowed for the well-documented extended periods of sail without sight of land, not by accident but as pre-determined planned route; for example, 30 days for Bartolomeu Dias culminating on Mossel Bay, the 3 months Gama spent in the South Atlantic to use the Brazil current (southward), or the 29 days Cabral took from Cape Verde up to landing in Monte Pascoal, Brazil.

The Danish expedition to Arabia 1761-67 can be said to be the world's first oceanographic expedition, as the ship Grønland had on board a group of scientists, including naturalist Peter Forsskål, who was assigned an explicit task by the king, Frederik V, to study and describe the marine life in the open sea, including finding the cause of mareel, or milky seas. For this purpose, the expedition was equipped with nets and scrapers, specifically designed to collect samples from the open waters and the bottom at great depth.

Although Juan Ponce de León in 1513 first identified the Gulf Stream, and the current was well known to mariners, Benjamin Franklin made the first scientific study of it and gave it its name. Franklin measured water temperatures during several Atlantic crossings and correctly explained the Gulf Stream's cause. Franklin and Timothy Folger printed the first map of the Gulf Stream in 1769–1770.

1799 map of the currents in the Atlantic and Indian Oceans, by James Rennell

Information on the currents of the Pacific Ocean was gathered by explorers of the late 18th century, including James Cook and Louis Antoine de Bougainville. James Rennell wrote the first scientific textbooks on oceanography, detailing the current flows of the Atlantic and Indian oceans. During a voyage around the Cape of Good Hope in 1777, he mapped "the banks and currents at the Lagullas". He was also the first to understand the nature of the intermittent current near the Isles of Scilly, (now known as Rennell's Current).

Sir James Clark Ross took the first modern sounding in deep sea in 1840, and Charles Darwin published a paper on reefs and the formation of atolls as a result of the second voyage of HMS Beagle in 1831–1836. Robert FitzRoy published a four-volume report of Beagle's three voyages. In 1841–1842 Edward Forbes undertook dredging in the Aegean Sea that founded marine ecology.

The first superintendent of the United States Naval Observatory (1842–1861), Matthew Fontaine Maury devoted his time to the study of marine meteorology, navigation, and charting prevailing winds and currents. His 1855 textbook Physical Geography of the Sea was one of the first comprehensive oceanography studies. Many nations sent oceanographic observations to Maury at the Naval Observatory, where he and his colleagues evaluated the information and distributed the results worldwide.

Modern oceanography

Knowledge of the oceans remained confined to the topmost few fathoms of the water and a small amount of the bottom, mainly in shallow areas. Almost nothing was known of the ocean depths. The British Royal Navy's efforts to chart all of the world's coastlines in the mid-19th century reinforced the vague idea that most of the ocean was very deep, although little more was known. As exploration ignited both popular and scientific interest in the polar regions and Africa, so too did the mysteries of the unexplored oceans.

HMS Challenger undertook the first global marine research expedition in 1872.

The seminal event in the founding of the modern science of oceanography was the 1872–1876 Challenger expedition. As the first true oceanographic cruise, this expedition laid the groundwork for an entire academic and research discipline. In response to a recommendation from the Royal Society, the British Government announced in 1871 an expedition to explore world's oceans and conduct appropriate scientific investigation. Charles Wyville Thompson and Sir John Murray launched the Challenger expedition. Challenger, leased from the Royal Navy, was modified for scientific work and equipped with separate laboratories for natural history and chemistry. Under the scientific supervision of Thomson, Challenger travelled nearly 70,000 nautical miles (130,000 km) surveying and exploring. On her journey circumnavigating the globe, 492 deep sea soundings, 133 bottom dredges, 151 open water trawls and 263 serial water temperature observations were taken. Around 4,700 new species of marine life were discovered. The result was the Report Of The Scientific Results of the Exploring Voyage of H.M.S. Challenger during the years 1873–76. Murray, who supervised the publication, described the report as "the greatest advance in the knowledge of our planet since the celebrated discoveries of the fifteenth and sixteenth centuries". He went on to found the academic discipline of oceanography at the University of Edinburgh, which remained the centre for oceanographic research well into the 20th century. Murray was the first to study marine trenches and in particular the Mid-Atlantic Ridge, and map the sedimentary deposits in the oceans. He tried to map out the world's ocean currents based on salinity and temperature observations, and was the first to correctly understand the nature of coral reef development.

In the late 19th century, other Western nations also sent out scientific expeditions (as did private individuals and institutions). The first purpose built oceanographic ship, Albatros, was built in 1882. In 1893, Fridtjof Nansen allowed his ship, Fram, to be frozen in the Arctic ice. This enabled him to obtain oceanographic, meteorological and astronomical data at a stationary spot over an extended period.

Ocean currents (1911)

 

Writer and geographer John Francon Williams FRGS commemorative plaque, Clackmannan Cemetery 2019

In 1881 the geographer John Francon Williams published a seminal book, Geography of the Oceans. Between 1907 and 1911 Otto Krümmel published the Handbuch der Ozeanographie, which became influential in awakening public interest in oceanography. The four-month 1910 North Atlantic expedition headed by John Murray and Johan Hjort was the most ambitious research oceanographic and marine zoological project ever mounted until then, and led to the classic 1912 book The Depths of the Ocean.

The first acoustic measurement of sea depth was made in 1914. Between 1925 and 1927 the "Meteor" expedition gathered 70,000 ocean depth measurements using an echo sounder, surveying the Mid-Atlantic Ridge.

In 1934, Easter Ellen Cupp, the first woman to have earned a PhD (at Scripps) in the United States, completed a major work on diatoms that remained the standard taxonomy in the field until well after her death in 1999. In 1940, Cupp was let go from her position at Scripps. Sverdrup specifically commended Cupp as a conscientious and industrious worker and commented that his decision was no reflection on her ability as a scientist. Sverdrup used the instructor billet vacated by Cupp to employ Marston Sargent,a biologist studying marine algae, which was not a new research program at Scripps. Financial pressures did not prevent Sverdrup from retaining the services of two other young post-doctoral students, Walter Munk and Roger Revelle. Cupp's partner, Dorothy Rosenbury, found her a position teaching high school, where she remained for the rest of her career. (Russell, 2000)

Sverdrup, Johnson and Fleming published The Oceans in 1942, which was a major landmark. The Sea (in three volumes, covering physical oceanography, seawater and geology) edited by M.N. Hill was published in 1962, while Rhodes Fairbridge's Encyclopedia of Oceanography was published in 1966.

The Great Global Rift, running along the Mid Atlantic Ridge, was discovered by Maurice Ewing and Bruce Heezen in 1953 and mapped by Heezen and Marie Tharp using bathymetric data; in 1954 a mountain range under the Arctic Ocean was found by the Arctic Institute of the USSR. The theory of seafloor spreading was developed in 1960 by Harry Hammond Hess. The Ocean Drilling Program started in 1966. Deep-sea vents were discovered in 1977 by Jack Corliss and Robert Ballard in the submersible DSV Alvin.

In the 1950s, Auguste Piccard invented the bathyscaphe and used the bathyscaphe Trieste to investigate the ocean's depths. The United States nuclear submarine Nautilus made the first journey under the ice to the North Pole in 1958. In 1962 the FLIP (Floating Instrument Platform), a 355-foot (108 m) spar buoy, was first deployed.

In 1968, Tanya Atwater led the first all-woman oceanographic expedition. Until that time, gender policies restricted women oceanographers from participating in voyages to a significant extent.

From the 1970s, there has been much emphasis on the application of large scale computers to oceanography to allow numerical predictions of ocean conditions and as a part of overall environmental change prediction. Early techniques included analog computers (such as the Ishiguro Storm Surge Computer) generally now replaced by numerical methods (eg SLOSH.) An oceanographic buoy array was established in the Pacific to allow prediction of El Niño events.

1990 saw the start of the World Ocean Circulation Experiment (WOCE) which continued until 2002. Geosat seafloor mapping data became available in 1995.

Study of the oceans is critical to understanding shifts in Earth's energy balance along with related global and regional changes in climate, the biosphere and biogeochemistry. The atmosphere and ocean are linked because of evaporation and precipitation as well as thermal flux (and solar insolation). Recent studies have advanced knowledge on ocean acidification, ocean heat content, ocean currents, sea level rise, the oceanic carbon cycle, the water cycle, Arctic sea ice decline, coral bleaching, marine heatwaves, extreme weather, coastal erosion and many other phenomena in regards to ongoing climate change and climate feedbacks.

In general, understanding the world ocean through further scientific study enables better stewardship and sustainable utilization of Earth's resources.

Branches

Oceanographic frontal systems on the Southern Hemisphere

 

The Applied Marine Physics Building at the University of Miami's Rosenstiel School of Marine and Atmospheric Science on Virginia Key, September 2007

The study of oceanography is divided into these five branches:

Biological oceanography

Main article: Marine life

See also: Marine biology

Biological oceanography investigates the ecology and biology of marine organisms in the context of the physical, chemical and geological characteristics of their ocean environment.

Chemical oceanography

Main article: Chemical oceanography

Chemical oceanography is the study of the chemistry of the ocean. Whereas chemical oceanography is primarily occupied with the study and understanding of seawater properties and its changes, ocean chemistry focuses primarily on the geochemical cycles. The following is a central topic investigated by chemical oceanography.

Ocean acidification

Main article: Ocean acidification

Ocean acidification describes the decrease in ocean pH that is caused by anthropogenic carbon dioxide (CO₂) emissions into the atmosphere. Seawater is slightly alkaline and had a preindustrial pH of about 8.2. More recently, anthropogenic activities have steadily increased the carbon dioxide content of the atmosphere; about 30–40% of the added CO₂ is absorbed by the oceans, forming carbonic acid and lowering the pH (now below 8.1) through ocean acidification. The pH is expected to reach 7.7 by the year 2100.

An important element for the skeletons of marine animals is calcium, but calcium carbonate becomes more soluble with pressure, so carbonate shells and skeletons dissolve below the carbonate compensation depth. Calcium carbonate becomes more soluble at lower pH, so ocean acidification is likely to affect marine organisms with calcareous shells, such as oysters, clams, sea urchins and corals, and the carbonate compensation depth will rise closer to the sea surface. Affected planktonic organisms will include pteropods, coccolithophorids and foraminifera, all important in the food chain. In tropical regions, corals are likely to be severely affected as they become less able to build their calcium carbonate skeletons, in turn adversely impacting other reef dwellers.

The current rate of ocean chemistry change seems to be unprecedented in Earth's geological history, making it unclear how well marine ecosystems will adapt to the shifting conditions of the near future. Of particular concern is the manner in which the combination of acidification with the expected additional stressors of higher temperatures and lower oxygen levels will impact the seas.

Geological oceanography

Main article: Marine geology

Geological oceanography is the study of the geology of the ocean floor including plate tectonics and paleoceanography.

Physical oceanography

Main article: Physical oceanography

Physical oceanography studies the ocean's physical attributes including temperature-salinity structure, mixing, surface waves, internal waves, surface tides, internal tides, and currents. The following are central topics investigated by physical oceanography.

Seismic Oceanography

Main article: Seismic Oceanography

Ocean currents

Further information: Ocean current

Since the early ocean expeditions in oceanography, a major interest was the study of ocean currents and temperature measurements. The tides, the Coriolis effect, changes in direction and strength of wind, salinity, and temperature are the main factors determining ocean currents. The thermohaline circulation (THC) (thermo- referring to temperature and -haline referring to salt content) connects the ocean basins and is primarily dependent on the density of sea water. It is becoming more common to refer to this system as the 'meridional overturning circulation' because it more accurately accounts for other driving factors beyond temperature and salinity.

• Examples of sustained currents are the Gulf Stream and the Kuroshio Current which are wind-driven western boundary currents.

Ocean heat content

Further information: Oceanic heat content

Oceanic heat content (OHC) refers to the extra heat stored in the ocean from changes in Earth's energy balance. The increase in the ocean heat play an important role in sea level rise, because of thermal expansion. Ocean warming accounts for 90% of the energy accumulation associated with global warming since 1971.

Paleoceanography

Main article: Paleoceanography

Paleoceanography is the study of the history of the oceans in the geologic past with regard to circulation, chemistry, biology, geology and patterns of sedimentation and biological productivity. Paleoceanographic studies using environment models and different proxies enable the scientific community to assess the role of the oceanic processes in the global climate by the reconstruction of past climate at various intervals. Paleoceanographic research is also intimately tied to palaeoclimatology.

Oceanographic institutions

Oceanographic Museum

 

See also: List of oceanographic institutions and programs

The first international organization of oceanography was created in 1902 as the International Council for the Exploration of the Sea. In 1903 the Scripps Institution of Oceanography was founded, followed by Woods Hole Oceanographic Institution in 1930, Virginia Institute of Marine Science in 1938, and later the Lamont–Doherty Earth Observatory at Columbia University, and the School of Oceanography at University of Washington. In Britain, the National Oceanography Centre (an institute of the Natural Environment Research Council) is the successor to the UK's Institute of Oceanographic Sciences. In Australia, CSIRO Marine and Atmospheric Research (CMAR), is a leading centre. In 1921 the International Hydrographic Bureau (IHB) was formed in Monaco.

Related disciplines

• Biogeochemistry – Study of chemical cycles of the earth that are either driven by or influence biological activity
• Biogeography – Study of the distribution of species and ecosystems in geographic space and through geological time
• Climatology – Scientific study of climate, defined as weather conditions averaged over a period of time
• Coastal geography – Study of the region between the ocean and the land
• Environmental science – The integrated, quantitative, and interdisciplinary approach to the study of environmental systems.
• Geophysics – Physics of the Earth and its vicinity
• Glaciology – Scientific study of ice and natural phenomena involving ice
• Hydrography – Applied science of measurement and description of physical features of bodies of water
• Hydrology – Science of the movement, distribution, and quality of water on Earth and other planets
• Limnology – Science of inland aquatic ecosystems
• Meteorology – Interdisciplinary scientific study of the atmosphere focusing on weather forecasting
• MetOce