Brain–computer interface
From Wikipedia, the free encyclopedia
A brain–computer interface (BCI), sometimes called a mind-machine interface (MMI), or sometimes called a direct neural interface (DNI), synthetic telepathy interface (STI) or a brain–machine interface (BMI), is a direct communication pathway between the brain and an external device. BCIs are often directed at assisting, augmenting, or repairing human cognitive or sensory-motor functions.
Research on BCIs began in the 1970s at the University of California Los Angeles (UCLA) under a grant from the National Science Foundation, followed by a contract from DARPA.[1][2] The papers published after this research also mark the first appearance of the expression brain–computer interface in scientific literature.
The field of BCI research and development has since focused primarily on neuroprosthetics applications that aim at restoring damaged hearing, sight and movement. Thanks to the remarkable cortical plasticity of the brain, signals from implanted prostheses can, after adaptation, be handled by the brain like natural sensor or effector channels.[3] Following years of animal experimentation, the first neuroprosthetic devices implanted in humans appeared in the mid-1990s.
Berger's first recording device was very rudimentary. He inserted silver wires under the scalps of his patients. These were later replaced by silver foils attached to the patients' head by rubber bandages. Berger connected these sensors to a Lippmann capillary electrometer, with disappointing results. More sophisticated measuring devices, such as the Siemens double-coil recording galvanometer, which displayed electric voltages as small as one ten thousandth of a volt, led to success.
Berger analyzed the interrelation of alternations in his EEG wave diagrams with brain diseases. EEGs permitted completely new possibilities for the research of human brain activities.
The difference between BCIs and neuroprosthetics is mostly in how the terms are used: neuroprosthetics typically connect the nervous system to a device, whereas BCIs usually connect the brain (or nervous system) with a computer system. Practical neuroprosthetics can be linked to any part of the nervous system—for example, peripheral nerves—while the term "BCI" usually designates a narrower class of systems which interface with the central nervous system.
The terms are sometimes, however, used interchangeably. Neuroprosthetics and BCIs seek to achieve the same aims, such as restoring sight, hearing, movement, ability to communicate, and even cognitive function. Both use similar experimental methods and surgical techniques.
Monkey operating a robotic arm with brain–computer interfacing (Schwartz lab, University of Pittsburgh)
In 1969 the operant conditioning studies of Fetz and colleagues, at the Regional Primate Research Center and Department of Physiology and Biophysics, University of Washington School of Medicine in Seattle, showed for the first time that monkeys could learn to control the deflection of a biofeedback meter arm with neural activity.[7] Similar work in the 1970s established that monkeys could quickly learn to voluntarily control the firing rates of individual and multiple neurons in the primary motor cortex if they were rewarded for generating appropriate patterns of neural activity.[8]
Studies that developed algorithms to reconstruct movements from motor cortex neurons, which control movement, date back to the 1970s. In the 1980s, Apostolos Georgopoulos at Johns Hopkins University found a mathematical relationship between the electrical responses of single motor cortex neurons in rhesus macaque monkeys and the direction in which they moved their arms (based on a cosine function). He also found that dispersed groups of neurons, in different areas of the monkey's brains, collectively controlled motor commands, but was able to record the firings of neurons in only one area at a time, because of the technical limitations imposed by his equipment.[9]
There has been rapid development in BCIs since the mid-1990s.[10] Several groups have been able to capture complex brain motor cortex signals by recording from neural ensembles (groups of neurons) and using these to control external devices.
Yang Dan and colleagues' recordings of cat vision using a BCI implanted in the lateral geniculate nucleus (top row: original image; bottom row: recording)
In 1999, researchers led by Yang Dan at the University of California, Berkeley decoded neuronal firings to reproduce images seen by cats. The team used an array of electrodes embedded in the thalamus (which integrates all of the brain’s sensory input) of sharp-eyed cats. Researchers targeted 177 brain cells in the thalamus lateral geniculate nucleus area, which decodes signals from the retina. The cats were shown eight short movies, and their neuron firings were recorded. Using mathematical filters, the researchers decoded the signals to generate movies of what the cats saw and were able to reconstruct recognizable scenes and moving objects.[11] Similar results in humans have since been achieved by researchers in Japan (see below).
After conducting initial studies in rats during the 1990s, Nicolelis and his colleagues developed BCIs that decoded brain activity in owl monkeys and used the devices to reproduce monkey movements in robotic arms. Monkeys have advanced reaching and grasping abilities and good hand manipulation skills, making them ideal test subjects for this kind of work.
By 2000 the group succeeded in building a BCI that reproduced owl monkey movements while the monkey operated a joystick or reached for food.[12] The BCI operated in real time and could also control a separate robot remotely over Internet protocol. But the monkeys could not see the arm moving and did not receive any feedback, a so-called open-loop BCI.
Diagram of the BCI developed by Miguel Nicolelis and colleagues for use on Rhesus monkeys
Later experiments by Nicolelis using rhesus monkeys succeeded in closing the feedback loop and reproduced monkey reaching and grasping movements in a robot arm. With their deeply cleft and furrowed brains, rhesus monkeys are considered to be better models for human neurophysiology than owl monkeys. The monkeys were trained to reach and grasp objects on a computer screen by manipulating a joystick while corresponding movements by a robot arm were hidden.[13][14] The monkeys were later shown the robot directly and learned to control it by viewing its movements. The BCI used velocity predictions to control reaching movements and simultaneously predicted handgripping force.
Donoghue's group reported training rhesus monkeys to use a BCI to track visual targets on a computer screen(closed-loop BCI) with or without assistance of a joystick.[15] Schwartz's group created a BCI for three-dimensional tracking in virtual reality and also reproduced BCI control in a robotic arm.[16] The same group also created headlines when they demonstrated that a monkey could feed itself pieces of fruit and marshmallows using a robotic arm controlled by the animal's own brain signals.[17][18][19]
Andersen's group used recordings of premovement activity from the posterior parietal cortex in their BCI, including signals created when experimental animals anticipated receiving a reward.[20]
Miguel Nicolelis and colleagues demonstrated that the activity of large neural ensembles can predict arm position. This work made possible creation of BCIs that read arm movement intentions and translate them into movements of artificial actuators. Carmena and colleagues[13] programmed the neural coding in a BCI that allowed a monkey to control reaching and grasping movements by a robotic arm. Lebedev and colleagues[14] argued that brain networks reorganize to create a new representation of the robotic appendage in addition to the representation of the animal's own limbs.
The biggest impediment to BCI technology at present is the lack of a sensor modality that provides safe, accurate and robust access to brain signals. It is conceivable or even likely, however, that such a sensor will be developed within the next twenty years. The use of such a sensor should greatly expand the range of communication functions that can be provided using a BCI.
Development and implementation of a BCI system is complex and time consuming. In response to this problem, Gerwin Schalk has been developing a general-purpose system for BCI research, called BCI2000. BCI2000 has been in development since 2000 in a project led by the Brain–Computer Interface R&D Program at the Wadsworth Center of the New York State Department of Health in Albany, New York, USA.
A new 'wireless' approach uses light-gated ion channels such as Channelrhodopsin to control the activity of genetically defined subsets of neurons in vivo. In the context of a simple learning task, illumination of transfected cells in the somatosensory cortex influenced the decision making process of freely moving mice.[22]
The use of BMIs has also led to a deeper understanding of neural networks and the central nervous system. Research has shown that despite the inclination of neuroscientists to believe that neurons have the most effect when working together, single neurons can be conditioned through the use of BMIs to fire at a pattern that allows primates to control motor outputs. The use of BMIs has led to development of the single neuron insufficiency principle which states that even with a well tuned firing rate single neurons can only carry a narrow amount of information and therefore the highest level of accuracy is achieved by recording firings of the collective ensemble. Other principles discovered with the use of BMIs include the neuronal multitasking principle, the neuronal mass principle, the neural degeneracy principle, and the plasticity principle.[23]
Jens Naumann, a man with acquired blindness, being interviewed about his vision BCI on CBS's The Early Show
Invasive BCI research has targeted repairing damaged sight and providing new functionality for people with paralysis. Invasive BCIs are implanted directly into the grey matter of the brain during neurosurgery. Because they lie in the grey matter, invasive devices produce the highest quality signals of BCI devices but are prone to scar-tissue build-up, causing the signal to become weaker, or even non-existent, as the body reacts to a foreign object in the brain.
In vision science, direct brain implants have been used to treat non-congenital (acquired) blindness. One of the first scientists to produce a working brain interface to restore sight was private researcher William Dobelle.
Dobelle's first prototype was implanted into "Jerry", a man blinded in adulthood, in 1978. A single-array BCI containing 68 electrodes was implanted onto Jerry’s visual cortex and succeeded in producing phosphenes, the sensation of seeing light. The system included cameras mounted on glasses to send signals to the implant. Initially, the implant allowed Jerry to see shades of grey in a limited field of vision at a low frame-rate. This also required him to be hooked up to a mainframe computer, but shrinking electronics and faster computers made his artificial eye more portable and now enable him to perform simple tasks unassisted.[24]
Dummy unit illustrating the design of a BrainGate interface
In 2002, Jens Naumann, also blinded in adulthood, became the first in a series of 16 paying patients to receive Dobelle’s second generation implant, marking one of the earliest commercial uses of BCIs.
The second generation device used a more sophisticated implant enabling better mapping of phosphenes into coherent vision. Phosphenes are spread out across the visual field in what researchers call "the starry-night effect". Immediately after his implant, Jens was able to use his imperfectly restored vision to drive an automobile slowly around the parking area of the research institute.[25] Unfortunately, Dobelle died in 2004[26] before his processes and developments were documented. Subsequently, when Mr. Naumann and the other patients in the program began having problems with their vision, there was no relief and they eventually lost their "sight" again. Naumann wrote about his experience with Dobelle's work in Search for Paradise: A Patient's Account of the Artificial Vision Experiment[25] and has returned to his farm in Southeast Ontario, Canada, to resume his normal activities.[27]
Researchers at Emory University in Atlanta, led by Philip Kennedy and Roy Bakay, were first to install a brain implant in a human that produced signals of high enough quality to simulate movement. Their patient, Johnny Ray (1944–2002), suffered from ‘locked-in syndrome’ after suffering a brain-stem stroke in 1997. Ray’s implant was installed in 1998 and he lived long enough to start working with the implant, eventually learning to control a computer cursor; he died in 2002 of a brain aneurysm.[28]
Tetraplegic Matt Nagle became the first person to control an artificial hand using a BCI in 2005 as part of the first nine-month human trial of Cyberkinetics’s BrainGate chip-implant. Implanted in Nagle’s right precentral gyrus (area of the motor cortex for arm movement), the 96-electrode BrainGate implant allowed Nagle to control a robotic arm by thinking about moving his hand as well as a computer cursor, lights and TV.[29] One year later, professor Jonathan Wolpaw received the prize of the Altran Foundation for Innovation to develop a Brain Computer Interface with electrodes located on the surface of the skull, instead of directly in the brain.
More recently, research teams led by the Braingate group at Brown University[30] and a group led by University of Pittsburgh Medical Center,[31] both in collaborations with the United States Department of Veterans Affairs, have demonstrated further success in direct control of robotic prosthetic limbs with many degrees of freedom using direct connections to arrays of neurons in the motor cortex of patients with tetraplegia.
Electrocorticography (ECoG) measures the electrical activity of the brain taken from beneath the skull in a similar way to non-invasive electroencephalography (see below), but the electrodes are embedded in a thin plastic pad that is placed above the cortex, beneath the dura mater.[32] ECoG technologies were first trialed in humans in 2004 by Eric Leuthardt and Daniel Moran from Washington University in St Louis. In a later trial, the researchers enabled a teenage boy to play Space Invaders using his ECoG implant.[33] This research indicates that control is rapid, requires minimal training, and may be an ideal tradeoff with regards to signal fidelity and level of invasiveness.
(Note: these electrodes had not been implanted in the patient with the intention of developing a BCI. The patient had been suffering from severe epilepsy and the electrodes were temporarily implanted to help his physicians localize seizure foci; the BCI researchers simply took advantage of this.)[citation needed]
Signals can be either subdural or epidural, but are not taken from within the brain parenchyma itself. It has not been studied extensively until recently due to the limited access of subjects. Currently, the only manner to acquire the signal for study is through the use of patients requiring invasive monitoring for localization and resection of an epileptogenic focus.
ECoG is a very promising intermediate BCI modality because it has higher spatial resolution, better signal-to-noise ratio, wider frequency range, and less training requirements than scalp-recorded EEG, and at the same time has lower technical difficulty, lower clinical risk, and probably superior long-term stability than intracortical single-neuron recording. This feature profile and recent evidence of the high level of control with minimal training requirements shows potential for real world application for people with motor disabilities.[34][35]
Light Reactive Imaging BCI devices are still in the realm of theory. These would involve implanting a laser inside the skull. The laser would be trained on a single neuron and the neuron's reflectance measured by a separate sensor. When the neuron fires, the laser light pattern and wavelengths it reflects would change slightly. This would allow researchers to monitor single neurons but require less contact with tissue and reduce the risk of scar-tissue build-up.[citation needed]
Recordings of brainwaves produced by an electroencephalogram
Electroencephalography (EEG) is the most studied potential non-invasive interface, mainly due to its fine temporal resolution, ease of use, portability and low set-up cost. The technology is highly susceptible to noise however. Another substantial barrier to using EEG as a brain–computer interface is the extensive training required before users can work the technology. For example, in experiments beginning in the mid-1990s, Niels Birbaumer at the University of Tübingen in Germany trained severely paralysed people to self-regulate the slow cortical potentials in their EEG to such an extent that these signals could be used as a binary signal to control a computer cursor.[36] (Birbaumer had earlier trained epileptics to prevent impending fits by controlling this low voltage wave.) The experiment saw ten patients trained to move a computer cursor by controlling their brainwaves. The process was slow, requiring more than an hour for patients to write 100 characters with the cursor, while training often took many months.
Another research parameter is the type of oscillatory activity that is measured. Birbaumer's later research with Jonathan Wolpaw at New York State University has focused on developing technology that would allow users to choose the brain signals they found easiest to operate a BCI, including mu and beta rhythms.
A further parameter is the method of feedback used and this is shown in studies of P300 signals. Patterns of P300 waves are generated involuntarily (stimulus-feedback) when people see something they recognize and may allow BCIs to decode categories of thoughts without training patients first. By contrast, the biofeedback methods described above require learning to control brainwaves so the resulting brain activity can be detected.
Lawrence Farwell and Emanuel Donchin developed an EEG-based brain–computer interface in the 1980s.[37] Their "mental prosthesis" used the P300 brainwave response to allow subjects, including one paralyzed Locked-In syndrome patient, to communicate words, letters and simple commands to a computer and thereby to speak through a speech synthesizer driven by the computer. A number of similar devices have been developed since then. In 2000, for example, research by Jessica Bayliss at the University of Rochester showed that volunteers wearing virtual reality helmets could control elements in a virtual world using their P300 EEG readings, including turning lights on and off and bringing a mock-up car to a stop.[38]
While an EEG based brain-computer interface has been pursued extensively by a number of research labs, recent advancements made by Bin He and his team at the University of Minnesota suggest the potential of an EEG based brain-computer interface to accomplish tasks close to invasive brain-computer interface. Using advanced functional neuroimaging including BOLD functional MRI and EEG source imaging, Bin He and co-workers identified the co-variation and co-localization of electrophysiological and hemodynamic signals induced by motor imagination.[39] Refined by a neuroimaging approach and by a training protocol, Bin He and co-workers demonstrated the ability of a non-invasive EEG based brain-computer interface to control the flight of a virtual helicopter in 3-dimensional space, based upon motor imagination.[40] In June 2013 it was announced that Bin He had developed the technique to enable a remote-control helicopter to be guided through an obstacle course.[41]
In addition to a brain-computer interface based on brain waves, as recorded from scalp EEG electrodes, Bin He and co-workers explored a virtual EEG signal-based brain-computer interface by first solving the EEG inverse problem and then used the resulting virtual EEG for brain-computer interface tasks. Well-controlled studies suggested the merits of such a source analysis based brain-computer interface.[42]
The electrode was tested on an electrical test bench and on human subjects in four modalities of EEG activity, namely: (1) spontaneous EEG, (2) sensory event-related potentials, (3) brain stem potentials, and (4) cognitive event-related potentials. The performance of the dry electrode compared favorably with that of the standard wet electrodes in terms of skin preparation, no gel requirements (dry), and higher signal-to-noise ratio.[44]
In 1999 researchers at Case Western Reserve University, in Cleveland, Ohio, led by Hunter Peckham, used 64-electrode EEG skullcap to return limited hand movements to quadriplegic Jim Jatich. As Jatich concentrated on simple but opposite concepts like up and down, his beta-rhythm EEG output was analysed using software to identify patterns in the noise. A basic pattern was identified and used to control a switch: Above average activity was set to on, below average off. As well as enabling Jatich to control a computer cursor the signals were also used to drive the nerve controllers embedded in his hands, restoring some movement.[45]
Experiments by Eduardo Miranda, at the University of Plymouth in the UK, has aimed to use EEG recordings of mental activity associated with music to allow the disabled to express themselves musically through an encephalophone.[50] Ramaswamy Palaniappan has pioneered the development of BCI for use in biometrics to identify/authenticate a person.[51] The method has also been suggested for use as PIN generation device (for example in ATM and internet banking transactions.[52] The group which is now at University of Wolverhampton has previously developed analogue cursor control using thoughts.[53]
Researchers at the University of Twente in the Netherlands have been conducting research on using BCIs for non-disabled individuals, proposing that BCIs could improve error handling, task performance, and user experience and that they could broaden the user spectrum.[54] They particularly focused on BCI games,[55] suggesting that BCI games could provide challenge, fantasy and sociality to game players and could, thus, improve player experience.[56]
The Emotiv company has been selling a commercial video game controller, known as The Epoc, since December 2009. The Epoc uses electromagnetic sensors.[57][58]
The first BCI session with 100% accuracy (based on 80 right-hand and 80 left-hand movement imaginations) was recorded in 1998 by Christoph Guger. The BCI system used 27 electrodes overlaying the sensorimotor cortex, weighted the electrodes with Common Spatial Patterns, calculated the running variance and used a linear discriminant analysis.[59]
Research is ongoing into military use of BCIs and since the 1970s DARPA has been funding research on this topic.[1][2] The current focus of research is user-to-user communication through analysis of neural signals.[60] The project "Silent Talk" aims to detect and analyze the word-specific neural signals, using EEG, which occur before speech is vocalized, and to see if the patterns are generalizable.[61]
ATR Labs' reconstruction of human vision using fMRI (top row: original image; bottom row: reconstruction from mean of combined readings)
Magnetoencephalography (MEG) and functional magnetic resonance imaging (fMRI) have both been used successfully as non-invasive BCIs.[62] In a widely reported experiment, fMRI allowed two users being scanned to play Pong in real-time by altering their haemodynamic response or brain blood flow through biofeedback techniques.[63]
fMRI measurements of haemodynamic responses in real time have also been used to control robot arms with a seven second delay between thought and movement.[64]
In 2008 research developed in the Advanced Telecommunications Research (ATR) Computational Neuroscience Laboratories in Kyoto, Japan, allowed the scientists to reconstruct images directly from the brain and display them on a computer in black and white at a resolution of 10x10 pixels. The article announcing these achievements was the cover story of the journal Neuron of 10 December 2008.[65]
In 2011 researchers from UC Berkeley published[66] a study reporting second-by-second reconstruction of videos watched by the study's subjects, from fMRI data. This was achieved by creating a statistical model relating visual patterns in videos shown to the subjects, to the brain activity caused by watching the videos. This model was then used to look up the 100 one-second video segments, in a database of 18 million seconds of random YouTube videos, whose visual patterns most closely matched the brain activity recorded when subjects watched a new video. These 100 one-second video extracts were then combined into a mashed-up image that resembled the video being watched.[67][68][69]
However, since Neurogaming is still in its first stages, not much is written about the new industry. The first NeuroGaming Conference was held in San Francisco on May 1–2, 2013.[74]
Steady-state visually evoked potentials (SSVEPs) use potentials generated by exciting the retina, using visual stimuli modulated at certain frequencies. SSVEP’s stimuli are often formed from alternating checkerboard patterns and at times simply use flashing images . The frequency of the phase reversal of the stimulus used can be clearly distinguished in the spectrum of an EEG; this makes detection of SSVEP stimuli relatively easy . SSVEP has proved to be successful within many BCI systems . This is due to several factors, the signal elicited is measurable in as large a population as the transient VEP and blink movement and electro cardiographic artefacts do not affect the frequencies monitored. In addition, the SSVEP signal is exceptionally robust; the topographic organization of the primary visual cortex is such that a broader area obtains afferents from the central or fovial region of the visual field .SSVEP does have several problems however. As SSVEPs use flashing stimuli to infer a user’s intent, the user must gaze at one of the flashing or iterating symbols in order to interact with the system. It is, therefore, likely that the symbols could become irritating and uncomfortable to use during longer play sessions, which can often last more than an hour which may not be an ideal gameplay.
Another type of VEP used with applications is the P300 potential. The P300 event-related potential is a positive peak in the EEG that occurs at roughly 300 ms after the appearance of a target stimulus (a stimulus for which the user is waiting or seeking) or oddball stimuli . The P300 amplitude decreases as the target stimuli and the ignored stimuli grow more similar.The P300 is thought to be related to a higher level attention process or an orienting response Using P300 as a control scheme has the advantage of the participant only having to attend limited training sessions. The first application to use the P300 model was the P300 matrix . Within this system, a subject would choose a letter from a grid of 6 by 6 letters and numbers. The rows and columns of the grid flashed sequentially and every time the selected “choice letter” was illuminated the user’s P300 was (potentially) elicited. However, the communication process, at approximately 17 characters per minute, was quite slow. The P300 is a BCI that offers a discrete selection rather than a continuous control mechanism. The advantage of P300 use within games is that the player does not have to teach himself/herself how to use a completely new control system and so only has to undertake short training instances, to learn the gameplay mechanics and basic use of the BCI paradigm.[75]
Research into synthetic telepathy using subvocalization is taking place at the University of California, Irvine under lead scientist Mike D'Zmura. The first such communication took place in the 1960s using EEG to create Morse code using brain alpha waves. Using EEG to communicate imagined speech is less accurate than the invasive method of placing an electrode between the skull and the brain.[77] On February 27, 2013 Duke University researchers successfully connected the brains of two rats with electronic interfaces that allowed them to directly share information, in the first-ever direct brain-to-brain interface.[78][79]
Philip Kennedy founded Neural Signals in 1987 to develop BCIs that would allow paralysed patients to communicate with the outside world and control external devices. As well as an invasive BCI, the company also sells an implant to restore speech. Neural Signals' "Brain Communicator" BCI device uses glass cones containing microelectrodes coated with proteins to encourage the electrodes to bind to neurons.
Although 16 paying patients were treated using William Dobelle's vision BCI, new implants ceased within a year of Dobelle's death in 2004. A company controlled by Dobelle, Avery Biomedical Devices, and Stony Brook University are continuing development of the implant, which has not yet received Food and Drug Administration approval for human implantation in the United States.[80]
Ambient, at a TI developers conference in early 2008, demonstrated a product they have in development called The Audeo. The Audeo aims to create a human–computer interface for communication without the need of physical motor control or speech production. Using signal processing, unpronounced speech can be translated from intercepted neurological signals.[81]
Mindball is a product, developed and commercialized by the Swedish company Interactive Productline, in which players compete to control a ball's movement across a table by becoming more relaxed and focused.[82] Interactive Productline's objective is to develop and sell easily understandable EEG products that train the ability to relax and focus.[83]
An Austrian company called Guger Technologies or[84] g.tec, has been offering Brain Computer Interface systems since 1999. The company provides base BCI models as development platforms for the research community to build upon, including the P300 Speller, Motor Imagery, and Steady-State Visual Evoked Potential. g.tec recently developed the g.SAHARA dry electrode system, which can provide signals comparable to gel-based systems.[85]
Spanish company Starlab, entered this market in 2009 with a wireless 4-channel system called Enobio. In 2011 Enobio 8 and 20 channel (CE Medical) was released and is now commercialised by Starlab spin-off Neuroelectrics Designed for medical and research purposes the system provides an all in one solution and a platform for application development.[86]
There are three main consumer-devices commercial-competitors in this area (launch date mentioned in brackets) which have launched such devices primarily for gaming- and PC-users:
Mind Solutions Inc. holds a patent for an EEG device currently in prototype phase, and has also developed multiple software applications for future use with their headset. These software applications have been used primarily on EEG devices currently on the market. Their software is geared towards aiding the handicapped and a platform for future generation gaming. It has 3 software titles complete and on the market that currently require an Emotiv headset to run.[87]
Development of the first working neurochip was claimed by a Caltech team led by Jerome Pine and Michael Maher in 1997.[89] The Caltech chip had room for 16 neurons.
In 2003 a team led by Theodore Berger, at the University of Southern California, started work on a neurochip designed to function as an artificial or prosthetic hippocampus. The neurochip was designed to function in rat brains and was intended as a prototype for the eventual development of higher-brain prosthesis. The hippocampus was chosen because it is thought to be the most ordered and structured part of the brain and is the most studied area. Its function is to encode experiences for storage as long-term memories elsewhere in the brain.[90]
Thomas DeMarse at the University of Florida used a culture of 25,000 neurons taken from a rat's brain to fly a F-22 fighter jet aircraft simulator.[91] After collection, the cortical neurons were cultured in a petri dish and rapidly began to reconnect themselves to form a living neural network. The cells were arranged over a grid of 60 electrodes and used to control the pitch and yaw functions of the simulator. The study's focus was on understanding how the human brain performs and learns computational tasks at a cellular level.
Researchers are well aware that sound ethical guidelines, appropriately moderated enthusiasm in media coverage and education about BCI systems will be of utmost importance for the societal acceptance of this technology. Thus, recently more effort is made inside the BCI community to create consensus on ethical guidelines for BCI research, development and dissemination.[95]
Research on BCIs began in the 1970s at the University of California Los Angeles (UCLA) under a grant from the National Science Foundation, followed by a contract from DARPA.[1][2] The papers published after this research also mark the first appearance of the expression brain–computer interface in scientific literature.
The field of BCI research and development has since focused primarily on neuroprosthetics applications that aim at restoring damaged hearing, sight and movement. Thanks to the remarkable cortical plasticity of the brain, signals from implanted prostheses can, after adaptation, be handled by the brain like natural sensor or effector channels.[3] Following years of animal experimentation, the first neuroprosthetic devices implanted in humans appeared in the mid-1990s.
History
The history of brain–computer interfaces (BCIs) starts with Hans Berger's discovery of the electrical activity of the human brain and the development of electroencephalography (EEG). In 1924 Berger was the first to record human brain activity by means of EEG. Berger was able to identify oscillatory activity in the brain by analyzing EEG traces. One wave he identified was the alpha wave (8–13 Hz), also known as Berger's wave.Berger's first recording device was very rudimentary. He inserted silver wires under the scalps of his patients. These were later replaced by silver foils attached to the patients' head by rubber bandages. Berger connected these sensors to a Lippmann capillary electrometer, with disappointing results. More sophisticated measuring devices, such as the Siemens double-coil recording galvanometer, which displayed electric voltages as small as one ten thousandth of a volt, led to success.
Berger analyzed the interrelation of alternations in his EEG wave diagrams with brain diseases. EEGs permitted completely new possibilities for the research of human brain activities.
BCI versus neuroprosthetics
Neuroprosthetics is an area of neuroscience concerned with neural prostheses. That is, using artificial devices to replace the function of impaired nervous systems and brain related problems, or of sensory organs. The most widely used neuroprosthetic device is the cochlear implant which, as of December 2010, had been implanted in approximately 220,000 people worldwide.[4] There are also several neuroprosthetic devices that aim to restore vision, including retinal implants.The difference between BCIs and neuroprosthetics is mostly in how the terms are used: neuroprosthetics typically connect the nervous system to a device, whereas BCIs usually connect the brain (or nervous system) with a computer system. Practical neuroprosthetics can be linked to any part of the nervous system—for example, peripheral nerves—while the term "BCI" usually designates a narrower class of systems which interface with the central nervous system.
The terms are sometimes, however, used interchangeably. Neuroprosthetics and BCIs seek to achieve the same aims, such as restoring sight, hearing, movement, ability to communicate, and even cognitive function. Both use similar experimental methods and surgical techniques.
Animal BCI research
Several laboratories have managed to record signals from monkey and rat cerebral cortices to operate BCIs to produce movement. Monkeys have navigated computer cursors on screen and commanded robotic arms to perform simple tasks simply by thinking about the task and seeing the visual feedback, but without any motor output.[5] In May 2008 photographs that showed a monkey at the University of Pittsburgh Medical Center operating a robotic arm by thinking were published in a number of well known science journals and magazines.[6] Other research on cats has decoded their neural visual signals.Early work
In 1969 the operant conditioning studies of Fetz and colleagues, at the Regional Primate Research Center and Department of Physiology and Biophysics, University of Washington School of Medicine in Seattle, showed for the first time that monkeys could learn to control the deflection of a biofeedback meter arm with neural activity.[7] Similar work in the 1970s established that monkeys could quickly learn to voluntarily control the firing rates of individual and multiple neurons in the primary motor cortex if they were rewarded for generating appropriate patterns of neural activity.[8]
Studies that developed algorithms to reconstruct movements from motor cortex neurons, which control movement, date back to the 1970s. In the 1980s, Apostolos Georgopoulos at Johns Hopkins University found a mathematical relationship between the electrical responses of single motor cortex neurons in rhesus macaque monkeys and the direction in which they moved their arms (based on a cosine function). He also found that dispersed groups of neurons, in different areas of the monkey's brains, collectively controlled motor commands, but was able to record the firings of neurons in only one area at a time, because of the technical limitations imposed by his equipment.[9]
There has been rapid development in BCIs since the mid-1990s.[10] Several groups have been able to capture complex brain motor cortex signals by recording from neural ensembles (groups of neurons) and using these to control external devices.
Prominent research successes
Kennedy and Yang Dan
Phillip Kennedy (who later founded Neural Signals in 1987) and colleagues built the first intracortical brain–computer interface by implanting neurotrophic-cone electrodes into monkeys.[citation needed]
In 1999, researchers led by Yang Dan at the University of California, Berkeley decoded neuronal firings to reproduce images seen by cats. The team used an array of electrodes embedded in the thalamus (which integrates all of the brain’s sensory input) of sharp-eyed cats. Researchers targeted 177 brain cells in the thalamus lateral geniculate nucleus area, which decodes signals from the retina. The cats were shown eight short movies, and their neuron firings were recorded. Using mathematical filters, the researchers decoded the signals to generate movies of what the cats saw and were able to reconstruct recognizable scenes and moving objects.[11] Similar results in humans have since been achieved by researchers in Japan (see below).
Nicolelis
Miguel Nicolelis, a professor at Duke University, in Durham, North Carolina, has been a prominent proponent of using multiple electrodes spread over a greater area of the brain to obtain neuronal signals to drive a BCI. Such neural ensembles are said to reduce the variability in output produced by single electrodes, which could make it difficult to operate a BCI.After conducting initial studies in rats during the 1990s, Nicolelis and his colleagues developed BCIs that decoded brain activity in owl monkeys and used the devices to reproduce monkey movements in robotic arms. Monkeys have advanced reaching and grasping abilities and good hand manipulation skills, making them ideal test subjects for this kind of work.
By 2000 the group succeeded in building a BCI that reproduced owl monkey movements while the monkey operated a joystick or reached for food.[12] The BCI operated in real time and could also control a separate robot remotely over Internet protocol. But the monkeys could not see the arm moving and did not receive any feedback, a so-called open-loop BCI.
.jpg)
Later experiments by Nicolelis using rhesus monkeys succeeded in closing the feedback loop and reproduced monkey reaching and grasping movements in a robot arm. With their deeply cleft and furrowed brains, rhesus monkeys are considered to be better models for human neurophysiology than owl monkeys. The monkeys were trained to reach and grasp objects on a computer screen by manipulating a joystick while corresponding movements by a robot arm were hidden.[13][14] The monkeys were later shown the robot directly and learned to control it by viewing its movements. The BCI used velocity predictions to control reaching movements and simultaneously predicted handgripping force.
Donoghue, Schwartz and Andersen
Other laboratories which have developed BCIs and algorithms that decode neuron signals include those run by John Donoghue at Brown University, Andrew Schwartz at the University of Pittsburgh and Richard Andersen at Caltech. These researchers have been able to produce working BCIs, even using recorded signals from far fewer neurons than did Nicolelis (15–30 neurons versus 50–200 neurons).Donoghue's group reported training rhesus monkeys to use a BCI to track visual targets on a computer screen(closed-loop BCI) with or without assistance of a joystick.[15] Schwartz's group created a BCI for three-dimensional tracking in virtual reality and also reproduced BCI control in a robotic arm.[16] The same group also created headlines when they demonstrated that a monkey could feed itself pieces of fruit and marshmallows using a robotic arm controlled by the animal's own brain signals.[17][18][19]
Andersen's group used recordings of premovement activity from the posterior parietal cortex in their BCI, including signals created when experimental animals anticipated receiving a reward.[20]
Other research
In addition to predicting kinematic and kinetic parameters of limb movements, BCIs that predict electromyographic or electrical activity of the muscles of primates are being developed.[21] Such BCIs could be used to restore mobility in paralyzed limbs by electrically stimulating muscles.Miguel Nicolelis and colleagues demonstrated that the activity of large neural ensembles can predict arm position. This work made possible creation of BCIs that read arm movement intentions and translate them into movements of artificial actuators. Carmena and colleagues[13] programmed the neural coding in a BCI that allowed a monkey to control reaching and grasping movements by a robotic arm. Lebedev and colleagues[14] argued that brain networks reorganize to create a new representation of the robotic appendage in addition to the representation of the animal's own limbs.
The biggest impediment to BCI technology at present is the lack of a sensor modality that provides safe, accurate and robust access to brain signals. It is conceivable or even likely, however, that such a sensor will be developed within the next twenty years. The use of such a sensor should greatly expand the range of communication functions that can be provided using a BCI.
Development and implementation of a BCI system is complex and time consuming. In response to this problem, Gerwin Schalk has been developing a general-purpose system for BCI research, called BCI2000. BCI2000 has been in development since 2000 in a project led by the Brain–Computer Interface R&D Program at the Wadsworth Center of the New York State Department of Health in Albany, New York, USA.
A new 'wireless' approach uses light-gated ion channels such as Channelrhodopsin to control the activity of genetically defined subsets of neurons in vivo. In the context of a simple learning task, illumination of transfected cells in the somatosensory cortex influenced the decision making process of freely moving mice.[22]
The use of BMIs has also led to a deeper understanding of neural networks and the central nervous system. Research has shown that despite the inclination of neuroscientists to believe that neurons have the most effect when working together, single neurons can be conditioned through the use of BMIs to fire at a pattern that allows primates to control motor outputs. The use of BMIs has led to development of the single neuron insufficiency principle which states that even with a well tuned firing rate single neurons can only carry a narrow amount of information and therefore the highest level of accuracy is achieved by recording firings of the collective ensemble. Other principles discovered with the use of BMIs include the neuronal multitasking principle, the neuronal mass principle, the neural degeneracy principle, and the plasticity principle.[23]
The BCI Award
The Annual BCI Research Award, endowed with 3,000 USD, is awarded in recognition of outstanding and innovative research in the field of Brain-Computer Interfaces. Each year, a renowned research laboratory is asked to judge the submitted projects and to award the prize. The jury consists of world-leading BCI experts recruited by the awarding laboratory. Following list consists the winners of the BCI Award:- Motor imagery-based Brain-Computer Interface robotic rehabilitation for stroke.
- 2011: Moritz Grosse-Wentrup and Bernhard Schölkopf, (Max Planck Institute for Intelligent Systems, Germany)
- What are the neuro-physiological causes of performance variations in brain-computer interfacing?
- 2012: Surjo R. Soekadar and Niels Birbaumer, (Applied Neurotechnology Lab, University Hospital Tübingen and Institute of Medical Psychology and Behavioral Neurobiology, Eberhard Karls University, Tübingen, Germany)
- Improving Efficacy of Ipsilesional Brain-Computer Interface Training in Neurorehabilitation of Chronic Stroke.
- 2013: M. C. Dadarlata,b, J. E. O’Dohertya, P. N. Sabesa,b (aDepartment of Physiology, Center for Integrative Neuroscience, San Francisco, CA, US, bUC Berkeley-UCSF Bioengineering Graduate Program, University of California, San Francisco, CA, US),
- A learning-based approach to artificial sensory feedback: intracortical microstimulation replaces and augments vision.
Human BCI research
Invasive BCIs
Vision
Invasive BCI research has targeted repairing damaged sight and providing new functionality for people with paralysis. Invasive BCIs are implanted directly into the grey matter of the brain during neurosurgery. Because they lie in the grey matter, invasive devices produce the highest quality signals of BCI devices but are prone to scar-tissue build-up, causing the signal to become weaker, or even non-existent, as the body reacts to a foreign object in the brain.
In vision science, direct brain implants have been used to treat non-congenital (acquired) blindness. One of the first scientists to produce a working brain interface to restore sight was private researcher William Dobelle.
Dobelle's first prototype was implanted into "Jerry", a man blinded in adulthood, in 1978. A single-array BCI containing 68 electrodes was implanted onto Jerry’s visual cortex and succeeded in producing phosphenes, the sensation of seeing light. The system included cameras mounted on glasses to send signals to the implant. Initially, the implant allowed Jerry to see shades of grey in a limited field of vision at a low frame-rate. This also required him to be hooked up to a mainframe computer, but shrinking electronics and faster computers made his artificial eye more portable and now enable him to perform simple tasks unassisted.[24]
In 2002, Jens Naumann, also blinded in adulthood, became the first in a series of 16 paying patients to receive Dobelle’s second generation implant, marking one of the earliest commercial uses of BCIs.
The second generation device used a more sophisticated implant enabling better mapping of phosphenes into coherent vision. Phosphenes are spread out across the visual field in what researchers call "the starry-night effect". Immediately after his implant, Jens was able to use his imperfectly restored vision to drive an automobile slowly around the parking area of the research institute.[25] Unfortunately, Dobelle died in 2004[26] before his processes and developments were documented. Subsequently, when Mr. Naumann and the other patients in the program began having problems with their vision, there was no relief and they eventually lost their "sight" again. Naumann wrote about his experience with Dobelle's work in Search for Paradise: A Patient's Account of the Artificial Vision Experiment[25] and has returned to his farm in Southeast Ontario, Canada, to resume his normal activities.[27]
Movement
BCIs focusing on motor neuroprosthetics aim to either restore movement in individuals with paralysis or provide devices to assist them, such as interfaces with computers or robot arms.Researchers at Emory University in Atlanta, led by Philip Kennedy and Roy Bakay, were first to install a brain implant in a human that produced signals of high enough quality to simulate movement. Their patient, Johnny Ray (1944–2002), suffered from ‘locked-in syndrome’ after suffering a brain-stem stroke in 1997. Ray’s implant was installed in 1998 and he lived long enough to start working with the implant, eventually learning to control a computer cursor; he died in 2002 of a brain aneurysm.[28]
Tetraplegic Matt Nagle became the first person to control an artificial hand using a BCI in 2005 as part of the first nine-month human trial of Cyberkinetics’s BrainGate chip-implant. Implanted in Nagle’s right precentral gyrus (area of the motor cortex for arm movement), the 96-electrode BrainGate implant allowed Nagle to control a robotic arm by thinking about moving his hand as well as a computer cursor, lights and TV.[29] One year later, professor Jonathan Wolpaw received the prize of the Altran Foundation for Innovation to develop a Brain Computer Interface with electrodes located on the surface of the skull, instead of directly in the brain.
More recently, research teams led by the Braingate group at Brown University[30] and a group led by University of Pittsburgh Medical Center,[31] both in collaborations with the United States Department of Veterans Affairs, have demonstrated further success in direct control of robotic prosthetic limbs with many degrees of freedom using direct connections to arrays of neurons in the motor cortex of patients with tetraplegia.
Partially invasive BCIs
Partially invasive BCI devices are implanted inside the skull but rest outside the brain rather than within the grey matter. They produce better resolution signals than non-invasive BCIs where the bone tissue of the cranium deflects and deforms signals and have a lower risk of forming scar-tissue in the brain than fully invasive BCIs.Electrocorticography (ECoG) measures the electrical activity of the brain taken from beneath the skull in a similar way to non-invasive electroencephalography (see below), but the electrodes are embedded in a thin plastic pad that is placed above the cortex, beneath the dura mater.[32] ECoG technologies were first trialed in humans in 2004 by Eric Leuthardt and Daniel Moran from Washington University in St Louis. In a later trial, the researchers enabled a teenage boy to play Space Invaders using his ECoG implant.[33] This research indicates that control is rapid, requires minimal training, and may be an ideal tradeoff with regards to signal fidelity and level of invasiveness.
(Note: these electrodes had not been implanted in the patient with the intention of developing a BCI. The patient had been suffering from severe epilepsy and the electrodes were temporarily implanted to help his physicians localize seizure foci; the BCI researchers simply took advantage of this.)[citation needed]
Signals can be either subdural or epidural, but are not taken from within the brain parenchyma itself. It has not been studied extensively until recently due to the limited access of subjects. Currently, the only manner to acquire the signal for study is through the use of patients requiring invasive monitoring for localization and resection of an epileptogenic focus.
ECoG is a very promising intermediate BCI modality because it has higher spatial resolution, better signal-to-noise ratio, wider frequency range, and less training requirements than scalp-recorded EEG, and at the same time has lower technical difficulty, lower clinical risk, and probably superior long-term stability than intracortical single-neuron recording. This feature profile and recent evidence of the high level of control with minimal training requirements shows potential for real world application for people with motor disabilities.[34][35]
Light Reactive Imaging BCI devices are still in the realm of theory. These would involve implanting a laser inside the skull. The laser would be trained on a single neuron and the neuron's reflectance measured by a separate sensor. When the neuron fires, the laser light pattern and wavelengths it reflects would change slightly. This would allow researchers to monitor single neurons but require less contact with tissue and reduce the risk of scar-tissue build-up.[citation needed]
Non-invasive BCIs
As well as invasive experiments, there have also been experiments in humans using non-invasive neuroimaging technologies as interfaces. Signals recorded in this way have been used to power muscle implants and restore partial movement in an experimental volunteer. Although they are easy to wear, non-invasive implants produce poor signal resolution because the skull dampens signals, dispersing and blurring the electromagnetic waves created by the neurons. Although the waves can still be detected it is more difficult to determine the area of the brain that created them or the actions of individual neurons.EEG
Overview
Electroencephalography (EEG) is the most studied potential non-invasive interface, mainly due to its fine temporal resolution, ease of use, portability and low set-up cost. The technology is highly susceptible to noise however. Another substantial barrier to using EEG as a brain–computer interface is the extensive training required before users can work the technology. For example, in experiments beginning in the mid-1990s, Niels Birbaumer at the University of Tübingen in Germany trained severely paralysed people to self-regulate the slow cortical potentials in their EEG to such an extent that these signals could be used as a binary signal to control a computer cursor.[36] (Birbaumer had earlier trained epileptics to prevent impending fits by controlling this low voltage wave.) The experiment saw ten patients trained to move a computer cursor by controlling their brainwaves. The process was slow, requiring more than an hour for patients to write 100 characters with the cursor, while training often took many months.
Another research parameter is the type of oscillatory activity that is measured. Birbaumer's later research with Jonathan Wolpaw at New York State University has focused on developing technology that would allow users to choose the brain signals they found easiest to operate a BCI, including mu and beta rhythms.
A further parameter is the method of feedback used and this is shown in studies of P300 signals. Patterns of P300 waves are generated involuntarily (stimulus-feedback) when people see something they recognize and may allow BCIs to decode categories of thoughts without training patients first. By contrast, the biofeedback methods described above require learning to control brainwaves so the resulting brain activity can be detected.
Lawrence Farwell and Emanuel Donchin developed an EEG-based brain–computer interface in the 1980s.[37] Their "mental prosthesis" used the P300 brainwave response to allow subjects, including one paralyzed Locked-In syndrome patient, to communicate words, letters and simple commands to a computer and thereby to speak through a speech synthesizer driven by the computer. A number of similar devices have been developed since then. In 2000, for example, research by Jessica Bayliss at the University of Rochester showed that volunteers wearing virtual reality helmets could control elements in a virtual world using their P300 EEG readings, including turning lights on and off and bringing a mock-up car to a stop.[38]
While an EEG based brain-computer interface has been pursued extensively by a number of research labs, recent advancements made by Bin He and his team at the University of Minnesota suggest the potential of an EEG based brain-computer interface to accomplish tasks close to invasive brain-computer interface. Using advanced functional neuroimaging including BOLD functional MRI and EEG source imaging, Bin He and co-workers identified the co-variation and co-localization of electrophysiological and hemodynamic signals induced by motor imagination.[39] Refined by a neuroimaging approach and by a training protocol, Bin He and co-workers demonstrated the ability of a non-invasive EEG based brain-computer interface to control the flight of a virtual helicopter in 3-dimensional space, based upon motor imagination.[40] In June 2013 it was announced that Bin He had developed the technique to enable a remote-control helicopter to be guided through an obstacle course.[41]
In addition to a brain-computer interface based on brain waves, as recorded from scalp EEG electrodes, Bin He and co-workers explored a virtual EEG signal-based brain-computer interface by first solving the EEG inverse problem and then used the resulting virtual EEG for brain-computer interface tasks. Well-controlled studies suggested the merits of such a source analysis based brain-computer interface.[42]
Dry active electrode arrays
In the early 1990s Babak Taheri, at University of California, Davis demonstrated the first single and also multichannel dry active electrode arrays using micro-machining. The single channel dry EEG electrode construction and results were published in 1994.[43] The arrayed electrode was also demonstrated to perform well compared to Silver/Silver Chloride electrodes. The device consisted of four sites of sensors with integrated electronics to reduce noise by impedance matching. The advantages of such electrodes are: (1) no electrolyte used, (2) no skin preparation, (3) significantly reduced sensor size, and (4) compatibility with EEG monitoring systems. The active electrode array is an integrated system made of an array of capacitive sensors with local integrated circuitry housed in a package with batteries to power the circuitry. This level of integration was required to achieve the functional performance obtained by the electrode.The electrode was tested on an electrical test bench and on human subjects in four modalities of EEG activity, namely: (1) spontaneous EEG, (2) sensory event-related potentials, (3) brain stem potentials, and (4) cognitive event-related potentials. The performance of the dry electrode compared favorably with that of the standard wet electrodes in terms of skin preparation, no gel requirements (dry), and higher signal-to-noise ratio.[44]
In 1999 researchers at Case Western Reserve University, in Cleveland, Ohio, led by Hunter Peckham, used 64-electrode EEG skullcap to return limited hand movements to quadriplegic Jim Jatich. As Jatich concentrated on simple but opposite concepts like up and down, his beta-rhythm EEG output was analysed using software to identify patterns in the noise. A basic pattern was identified and used to control a switch: Above average activity was set to on, below average off. As well as enabling Jatich to control a computer cursor the signals were also used to drive the nerve controllers embedded in his hands, restoring some movement.[45]
Prosthesis control
Non-invasive BCIs have also been applied to enable brain-control of prosthetic upper and lower extremity devices in people with paralysis. For example, Gert Pfurtscheller of Graz University of Technology and colleagues demonstrated a BCI-controlled functional electrical stimulation system to restore upper extremity movements in a person with tetraplegia due to spinal cord injury.[46] Between 2012 and 2013, researchers at the University of California, Irvine demonstrated for the first time that it is possible to use BCI technology to restore brain-controlled walking after spinal cord injury. In their study, a person with paraplegia due to spinal cord injury was able to operate a BCI-robotic gait orthosis to regain basic brain-controlled ambulation.[47][48]Other research
Electronic neural networks have been deployed which shift the learning phase from the user to the computer. Experiments by scientists at the Fraunhofer Society in 2004 using neural networks led to noticeable improvements within 30 minutes of training.[49]Experiments by Eduardo Miranda, at the University of Plymouth in the UK, has aimed to use EEG recordings of mental activity associated with music to allow the disabled to express themselves musically through an encephalophone.[50] Ramaswamy Palaniappan has pioneered the development of BCI for use in biometrics to identify/authenticate a person.[51] The method has also been suggested for use as PIN generation device (for example in ATM and internet banking transactions.[52] The group which is now at University of Wolverhampton has previously developed analogue cursor control using thoughts.[53]
Researchers at the University of Twente in the Netherlands have been conducting research on using BCIs for non-disabled individuals, proposing that BCIs could improve error handling, task performance, and user experience and that they could broaden the user spectrum.[54] They particularly focused on BCI games,[55] suggesting that BCI games could provide challenge, fantasy and sociality to game players and could, thus, improve player experience.[56]
The Emotiv company has been selling a commercial video game controller, known as The Epoc, since December 2009. The Epoc uses electromagnetic sensors.[57][58]
The first BCI session with 100% accuracy (based on 80 right-hand and 80 left-hand movement imaginations) was recorded in 1998 by Christoph Guger. The BCI system used 27 electrodes overlaying the sensorimotor cortex, weighted the electrodes with Common Spatial Patterns, calculated the running variance and used a linear discriminant analysis.[59]
Research is ongoing into military use of BCIs and since the 1970s DARPA has been funding research on this topic.[1][2] The current focus of research is user-to-user communication through analysis of neural signals.[60] The project "Silent Talk" aims to detect and analyze the word-specific neural signals, using EEG, which occur before speech is vocalized, and to see if the patterns are generalizable.[61]
MEG and MRI
Magnetoencephalography (MEG) and functional magnetic resonance imaging (fMRI) have both been used successfully as non-invasive BCIs.[62] In a widely reported experiment, fMRI allowed two users being scanned to play Pong in real-time by altering their haemodynamic response or brain blood flow through biofeedback techniques.[63]
fMRI measurements of haemodynamic responses in real time have also been used to control robot arms with a seven second delay between thought and movement.[64]
In 2008 research developed in the Advanced Telecommunications Research (ATR) Computational Neuroscience Laboratories in Kyoto, Japan, allowed the scientists to reconstruct images directly from the brain and display them on a computer in black and white at a resolution of 10x10 pixels. The article announcing these achievements was the cover story of the journal Neuron of 10 December 2008.[65]
In 2011 researchers from UC Berkeley published[66] a study reporting second-by-second reconstruction of videos watched by the study's subjects, from fMRI data. This was achieved by creating a statistical model relating visual patterns in videos shown to the subjects, to the brain activity caused by watching the videos. This model was then used to look up the 100 one-second video segments, in a database of 18 million seconds of random YouTube videos, whose visual patterns most closely matched the brain activity recorded when subjects watched a new video. These 100 one-second video extracts were then combined into a mashed-up image that resembled the video being watched.[67][68][69]
Neurogaming
Currently, there is a new field of gaming called Neurogaming, which uses non-invasive BCI in order to improve gameplay so that users can interact with a console without the use of a traditional controller.[70] Some Neurogaming software use a player's brain waves, heart rate, expressions, pupil dilation, and even emotions to complete tasks or affect the mood of the game.[71] For example, game developers at Emotiv have created non-invasive BCI that will determine the mood of a player and adjust music or scenery accordingly. This new form of interaction between player and software will enable a player to have a more realistic gaming experience.[72] Because there will be less disconnect between a player and console, Neurogaming will allow individuals to utilize their "psychological state"[73] and have their reactions transfer to games in real-time.[72]However, since Neurogaming is still in its first stages, not much is written about the new industry. The first NeuroGaming Conference was held in San Francisco on May 1–2, 2013.[74]
BCI Control Strategies in Neurogaming
Motor Imagery
Motor imagery involves the imagination of the movement of various body parts resulting in sensorimotor cortex activation,which modulates sensorimotor oscillations in the EEG. This can be detected by the BCI to infer a user’s intent. Motor imagery typically requires a number of sessions of training before acceptable control of the BCI is acquired. These training sessions may take a number of hours over several days before users can consistently employ the technique with acceptable levels of precision.Regardless of the duration of the training session,users are unable to master the control scheme.This results in very slow pace of the gameplay .[75]Bio/Neurofeedback for Passive BCI Designs
Biofeedback is used to monitor a subject’s mental relaxation. In some cases, biofeedback does not monitor electroencephalography (EEG), but instead bodily parameters such as electromyography(EMG), galvanic skin resistance (GSR), and heart rate variability (HRV).Many biofeedback systems are used to treat certain disorders such as attention deficit hyperactivity disorder (ADHD), sleep problems in children,teeth grinding, and chronic pain. EEG biofeedback systems typically monitor four different bands (theta: 4–7 Hz, alpha:8–12 Hz, SMR: 12–15 Hz, beta: 15–18 Hz) and challenge the subject to control them. Passive BCI involves using BCI to enrich human–machine interaction with implicit information on the actual user’s state, for example, simulations to detect when users intend to push brakes during an emergency car stopping procedure. Game developers using passive BCIs need to acknowledge that through repetition of game levels the user’s cognitive state will change or adapt. Within the first play of a level, the user will react to things differently from during the second play: for example, the user will be less surprised at an event in the game if he/she is expecting it.[75]Visual Evoked Potential (VEP)
A VEP is an electrical potential recorded after a subject is presented with a type of visual stimuli. There are several types of VEPs.Steady-state visually evoked potentials (SSVEPs) use potentials generated by exciting the retina, using visual stimuli modulated at certain frequencies. SSVEP’s stimuli are often formed from alternating checkerboard patterns and at times simply use flashing images . The frequency of the phase reversal of the stimulus used can be clearly distinguished in the spectrum of an EEG; this makes detection of SSVEP stimuli relatively easy . SSVEP has proved to be successful within many BCI systems . This is due to several factors, the signal elicited is measurable in as large a population as the transient VEP and blink movement and electro cardiographic artefacts do not affect the frequencies monitored. In addition, the SSVEP signal is exceptionally robust; the topographic organization of the primary visual cortex is such that a broader area obtains afferents from the central or fovial region of the visual field .SSVEP does have several problems however. As SSVEPs use flashing stimuli to infer a user’s intent, the user must gaze at one of the flashing or iterating symbols in order to interact with the system. It is, therefore, likely that the symbols could become irritating and uncomfortable to use during longer play sessions, which can often last more than an hour which may not be an ideal gameplay.
Another type of VEP used with applications is the P300 potential. The P300 event-related potential is a positive peak in the EEG that occurs at roughly 300 ms after the appearance of a target stimulus (a stimulus for which the user is waiting or seeking) or oddball stimuli . The P300 amplitude decreases as the target stimuli and the ignored stimuli grow more similar.The P300 is thought to be related to a higher level attention process or an orienting response Using P300 as a control scheme has the advantage of the participant only having to attend limited training sessions. The first application to use the P300 model was the P300 matrix . Within this system, a subject would choose a letter from a grid of 6 by 6 letters and numbers. The rows and columns of the grid flashed sequentially and every time the selected “choice letter” was illuminated the user’s P300 was (potentially) elicited. However, the communication process, at approximately 17 characters per minute, was quite slow. The P300 is a BCI that offers a discrete selection rather than a continuous control mechanism. The advantage of P300 use within games is that the player does not have to teach himself/herself how to use a completely new control system and so only has to undertake short training instances, to learn the gameplay mechanics and basic use of the BCI paradigm.[75]
Synthetic telepathy/silent communication
In a $6.3 million Army initiative to invent devices for telepathic communication, Gerwin Schalk, underwritten in a $2.2 million grant, found that it is possible to use ECoG signals to discriminate the vowels and consonants embedded in spoken and in imagined words. The results shed light on the distinct mechanisms associated with production of vowels and consonants, and could provide the basis for brain-based communication using imagined speech.[35][76]Research into synthetic telepathy using subvocalization is taking place at the University of California, Irvine under lead scientist Mike D'Zmura. The first such communication took place in the 1960s using EEG to create Morse code using brain alpha waves. Using EEG to communicate imagined speech is less accurate than the invasive method of placing an electrode between the skull and the brain.[77] On February 27, 2013 Duke University researchers successfully connected the brains of two rats with electronic interfaces that allowed them to directly share information, in the first-ever direct brain-to-brain interface.[78][79]
Commercialization
John Donoghue and fellow researchers founded Cyberkinetics. The company markets its electrode arrays under the BrainGate product name and has set the development of practical BCIs for humans as its major goal. The BrainGate is based on the Utah Array developed by Dick Normann.Philip Kennedy founded Neural Signals in 1987 to develop BCIs that would allow paralysed patients to communicate with the outside world and control external devices. As well as an invasive BCI, the company also sells an implant to restore speech. Neural Signals' "Brain Communicator" BCI device uses glass cones containing microelectrodes coated with proteins to encourage the electrodes to bind to neurons.
Although 16 paying patients were treated using William Dobelle's vision BCI, new implants ceased within a year of Dobelle's death in 2004. A company controlled by Dobelle, Avery Biomedical Devices, and Stony Brook University are continuing development of the implant, which has not yet received Food and Drug Administration approval for human implantation in the United States.[80]
Ambient, at a TI developers conference in early 2008, demonstrated a product they have in development called The Audeo. The Audeo aims to create a human–computer interface for communication without the need of physical motor control or speech production. Using signal processing, unpronounced speech can be translated from intercepted neurological signals.[81]
Mindball is a product, developed and commercialized by the Swedish company Interactive Productline, in which players compete to control a ball's movement across a table by becoming more relaxed and focused.[82] Interactive Productline's objective is to develop and sell easily understandable EEG products that train the ability to relax and focus.[83]
An Austrian company called Guger Technologies or[84] g.tec, has been offering Brain Computer Interface systems since 1999. The company provides base BCI models as development platforms for the research community to build upon, including the P300 Speller, Motor Imagery, and Steady-State Visual Evoked Potential. g.tec recently developed the g.SAHARA dry electrode system, which can provide signals comparable to gel-based systems.[85]
Spanish company Starlab, entered this market in 2009 with a wireless 4-channel system called Enobio. In 2011 Enobio 8 and 20 channel (CE Medical) was released and is now commercialised by Starlab spin-off Neuroelectrics Designed for medical and research purposes the system provides an all in one solution and a platform for application development.[86]
There are three main consumer-devices commercial-competitors in this area (launch date mentioned in brackets) which have launched such devices primarily for gaming- and PC-users:
- Neural Impulse Actuator (April 2008)
- Emotiv Systems (December 2009)
- NeuroSky (MindSet – June 2009; Uncle Milton Force Trainer – Fall 2009, Mattel MindFlex – Summer, 2009)
Mind Solutions Inc. holds a patent for an EEG device currently in prototype phase, and has also developed multiple software applications for future use with their headset. These software applications have been used primarily on EEG devices currently on the market. Their software is geared towards aiding the handicapped and a platform for future generation gaming. It has 3 software titles complete and on the market that currently require an Emotiv headset to run.[87]
Cell-culture BCIs
Researchers have built devices to interface with neural cells and entire neural networks in cultures outside animals. As well as furthering research on animal implantable devices, experiments on cultured neural tissue have focused on building problem-solving networks, constructing basic computers and manipulating robotic devices. Research into techniques for stimulating and recording from individual neurons grown on semiconductor chips is sometimes referred to as neuroelectronics or neurochips.[88]
Development of the first working neurochip was claimed by a Caltech team led by Jerome Pine and Michael Maher in 1997.[89] The Caltech chip had room for 16 neurons.
In 2003 a team led by Theodore Berger, at the University of Southern California, started work on a neurochip designed to function as an artificial or prosthetic hippocampus. The neurochip was designed to function in rat brains and was intended as a prototype for the eventual development of higher-brain prosthesis. The hippocampus was chosen because it is thought to be the most ordered and structured part of the brain and is the most studied area. Its function is to encode experiences for storage as long-term memories elsewhere in the brain.[90]
Thomas DeMarse at the University of Florida used a culture of 25,000 neurons taken from a rat's brain to fly a F-22 fighter jet aircraft simulator.[91] After collection, the cortical neurons were cultured in a petri dish and rapidly began to reconnect themselves to form a living neural network. The cells were arranged over a grid of 60 electrodes and used to control the pitch and yaw functions of the simulator. The study's focus was on understanding how the human brain performs and learns computational tasks at a cellular level.
Ethical considerations
Important ethical, legal and societal issues related to brain-computer interfacing are:[92][93][94][95]- conceptual issues (researchers disagree over what is and what is not a brain-computer interface),[95]
- obtaining informed consent from people who have difficulty communicating,
- risk/benefit analysis,
- shared responsibility of BCI teams (e.g. how to ensure that responsible group decisions can be made),
- the consequences of BCI technology for the quality of life of patients and their families,
- side-effects (e.g. neurofeedback of sensorimotor rhythm training is reported to affect sleep quality),
- personal responsibility and its possible constraints (e.g. who is responsible for erroneous actions with a neuroprosthesis),
- issues concerning personality and personhood and its possible alteration,
- therapeutic applications and their possible exceedance,
- questions of research ethics that arise when progressing from animal experimentation to application in human subjects,
- mind-reading and privacy,
- mind-control,
- use of the technology in advanced interrogation techniques by governmental authorities,
- selective enhancement and social stratification, and
- communication to the media.
Researchers are well aware that sound ethical guidelines, appropriately moderated enthusiasm in media coverage and education about BCI systems will be of utmost importance for the societal acceptance of this technology. Thus, recently more effort is made inside the BCI community to create consensus on ethical guidelines for BCI research, development and dissemination.[95]
Low-cost BCI-based Interfaces
Recently a number of companies have scaled back medical grade EEG technology (and in one case, NeuroSky, rebuilt the technology from the ground up[clarification needed]) to create inexpensive BCIs. This technology has been built into toys and gaming devices; some of these toys have been extremely commercially successful like the NeuroSky and Mattel MindFlex.- In 2006 Sony patented a neural interface system allowing radio waves to affect signals in the neural cortex.[96]
- In 2007 NeuroSky released the first affordable consumer based EEG along with the game NeuroBoy. This was also the first large scale EEG device to use dry sensor technology.[97]
- In 2008 OCZ Technology developed a device for use in video games relying primarily on electromyography.[98]
- In 2008 the Final Fantasy developer Square Enix announced that it was partnering with NeuroSky to create a game, Judecca.[99][100]
- In 2009 Mattel partnered with NeuroSky to release the Mindflex, a game that used an EEG to steer a ball through an obstacle course. By far the best selling consumer based EEG to date.[99][101]
- In 2009 Uncle Milton Industries partnered with NeuroSky to release the Star Wars Force Trainer, a game designed to create the illusion of possessing the force.[99][102]
- In 2009 Emotiv Systems released the EPOC, a 14 channel EEG device that can read 4 mental states, 13 conscious states, facial expressions, and head movements. The EPOC is the first commercial BCI to use dry sensor technology, which can be dampened with a saline solution for a better connection.[57]
- In November 2011 Time Magazine selected "necomimi" produced by Neurowear as one of the best inventions of the year. The company announced that it expected to launch a consumer version of the garment, consisting of cat-like ears controlled by a brain-wave reader produced by NeuroSky, in spring 2012.[103]
- In March 2012 g.tec introduced the intendiX-SPELLER, first commercially available BCI system for home use which can be used to control computer games and apps. It can detect different brain signals with an accuracy of 99%.[104] g.tec has hosted several workshop tours to demonstrate the intendiX system and other hardware and software to the public, such as a g.tec workshop tour of the US West Coast during September 2012.
- In January 2013 Hasaca National University (HNU) announced first Masters program in Virtual Reality Brain Computer interface application design.
- In February 2014 They Shall Walk (a nonprofit organization fixed on constructing exoskeletons, dubbed LIFESUITs, for paraplegics and quadriplegics) began a partnership with James W. Shakarji on the development of a wireless BCI via the production and localization of a Schumann resonance cavity.[105]
