US 7835787 B2
An EEG cap (8) having 64 or 128 electrodes (10) is placed on the head of the subject (11) who is viewing CRT monitor (14). The signals on each channel are amplified by amplifier (17) and sent to an analog-to-digital converter (20). PC (23) captures and records the amplified signals and the signals are processed by signal processing PC (26) performing linear signal processing. The resulting signal is sent back to a feedback/display PC (29) having monitor (14).
1. A method for interpreting brain activity comprising the steps of:
attaching a high-density array of brain activity sensors to a subject;
prompting said subject with an external stimulus;
generating a response signal from each of said sensors at a predetermined point in time with reference to the occurrence of said prompting;
computing a respective spatial weighting coefficient corresponding to each of said sensors by applying a linear classifier to said predetermined number of response signals;
weighting each response signal in accordance with its corresponding weighting coefficient; and
linearly integrating the weighted response signals whereby a single output signal is generated, said output signal having a relatively high correlation to the brain activity of said subject in response to said external stimulus.
2. The method of
3. The method of
4. The method of
5. The method of
6. The method of
7. The method of
identifying an expected scalp locality for discriminating components of the output signal relative to the external stimulus;
computing a sensor projection for each of said sensors;
using said sensor projections to identify the scalp locality for the actual discriminating components for said external stimulus; and
evaluating the validity of said output signal by determining whether the identified locality for the actual discriminating components is substantially the same as the expected locality.
8. A system for interpreting brain activity comprising:
a high-density array of brain activity sensors for application to the head of a subject; and
a computer for receiving signals from said sensors, said computer programmed to:
compute a respective spatial weighting coefficient corresponding to each of said sensors by applying a linear classifier to signals generated by said sensors in response to an external stimulus,
weight each of said signals in accordance with its corresponding weighting coefficient, and
linearly integrate the weighted response signals,
whereby a single output signal is generated, said output signal having a relatively high correlation to the brain activity of said subject in response to said external stimulus.
9. The system of
10. The system of
11. The system of
12. The system of
13. The system of
14. The system of
The present application is a continuation of International Patent Application No. PCT/US03/13943, filed May 5, 2003, published on Nov. 13, 2003 as International Patent Publication No. WO 03/093947, which claims priority to U.S. Provisional Patent Application No. 60/377,833, filed on May 3, 2002, the entire disclosure of which is incorporated herein by reference.
The U.S. Government has certain rights in this invention pursuant to the terms of Defense Advanced Research Project Agency (DARPA) contract N00014-010C-0482 and the Department of Defense Multidisciplinary University Research Initiative (MURI) program administered by the Office of Naval Research under Grant N00014-01-1-0625.
The performance of a brain computer interface (BCI) can be optimized by considering the simultaneous adaptation of both a human and machine learner. Preferably, adaptation of both learners occur on-line and in (near) real-time. The human and machine learners are assumed to process data sequentially, with the human learner gating the response of the machine learner. The gating by the human learner captures the dynamic switching between task-dependent strategies, while the machine learner constructs the mappings between brain signals and control signal for a given strategy (or set of strategies). The human and machine co-learn in that they adapt simultaneously to minimize an error metric, or equivalently, maximize a bit rate.
In a typical BCI system, signal acquisition from a human learner, or subject, is typically through one or more modalities (electroencephalography (EEG), magnetoencephalography (MEG), chronic electrode arrays, etc.). A key element of a BCI system is a machine learning or pattern recognition module to interpret the measured brain activity and map it to a set of control signals or, equivalently, a representation for communication, e.g., a visual display.
In addition to the machine learner, the human learner is integral to a BCI system. Adaptation of the human learner is often implicit, for example humans will switch strategies (e.g. think left/right versus up/down) based on their perceived performance. This dynamic switching by the human learner can make adaptation of the machine learner challenging, particularly since this can be viewed as making the input to the machine learner more non-stationary. Since the overall challenge in BCI is to maximize performance of the combined human-machine system (i.e., minimize error rate or conversely maximize bit rate) an approach is required which jointly optimizes the two learners.
Conventional analysis of brain activity using EEG and MEG sensors often relies on averaging over multiple trials to extract statistically relevant differences between two or more experimental conditions. Trial averaging is often used in brain imaging to mitigate low signal-to-interference (SIR) ratios. For example, it is the basis for analysis of event-related potentials (ERPs) as explained in Coles M. G. H. et al., “Event-related brain potentials: An introduction,” Electrophysiology of Mind. Oxford: Oxford University Press (1995). However, for some encephalographic applications, such as seizure prediction, trial averaging is problematic. One application where the problem of single-trial averaging is immediately apparent is the brain computer interface (BCI), i.e., interpreting brain activity for real-time communication. In the simplest case, where one wishes to communicate a binary decision, averaging corresponds to asking the same question over multiple trials and averaging the subject's binary responses. In order to obtain high-bandwidth communication, it is desirable to do as little averaging over time or across trials as possible.
More generally, single-trial analysis of brain activity is important in order to uncover the origin of response variability, for instance, in analysis of error-related negativity (ERN). The ERN is a negative deflection in the EEG following perceived incorrect responses (Gehring, W. J. et al., “A neural system for error detection and compensation,” Psychological Science, 4(6):385-390 (1993); Falkenstein, M. et al., “ERP components on reaction errors and their functional significance: A tutorial,” Biological Psychology, 51:87-107, (2000) or expected losses (Gehring, W. J. et al., “The medical frontal cortex and the rapid processing of monetary gains and loss,” Science, 295: 2279-2282 (2002)) in a forced-choice task. Single-trial detection of the ERN has been proposed as a means of correcting communication errors in a BCI system (Schalk et al., “EEG-based communication: presence of an error potential,” Clinical Neurophysiology, 111:2138-2144, (2000)). With the ability to analyze the precise timing and amplitude of the ERN, on individual trials, one can begin to study parameters that cannot be controlled across trial, such as reaction time or error perception. Such an approach opens up new possibilities for studying the behavioral relevance and neurological origin of the ERN.
With the large number of sensors on a single subject in high-density EEG and magnetoencephalography (G), e.g., 32 or more sensors, an alternative approach to trial averaging is to integrate information over space rather than across trials. A number of methods along these lines have been proposed. Blind source separation analyzes the multivariate statistics of the sensor data to identify spatial linear combinations that are statistically independent over time (Makeig et al., “Independent component analysis of electroencephalographic data,” Advances in Neural Information Processing Systems, 8: 145-151, MIT Press (1996); Vigario et al., “Independent component approach to the analysis of EEG and MEG recordings,” IEEE Transactions on Biomedical Engineering, 47(5): 589-593 (2000); Tang et al., “Localization of Independent Components of Magnetoencephalography in Cognitive Tasks,” Neural Computation, Neural Comput. 14(8): 1827-1858 (2002)). Separating independent signals and removing noise sources and artifacts increases SIR. However, blind source separation does not exploit the timing information of external events that is often available. In most current experimental paradigms subjects are prompted with external stimuli to which they are asked to respond. The timing of the stimuli, as well as the timing of overt responses, is therefore available, but is generally not exploited by the analysis method.
In the context of a BCI system, many methods have applied linear and nonlinear classification to a set of features extracted from the EEG. For example, adaptive autoregressive models have been used to extract features across a limited number of electrodes, with features combined using either linear or nonlinear classifiers to identify the activity from the time course of individual sensors (Pfurtscheller, G. et al., “Motor imagery and direct brain-computer communication,” Proceedings of the IEEE, 89(7):1123-1134, (2001)). Others have proposed to combine sensors in space by computing maximum and minimum eigenvalues of the sensor covariance matrices. The eigenvalues, which capture the power variations of synchronization and desynchronization, are then combined nonlinearly to obtain binary classification (Ramoser et al., “Optimal spatial filtering of single trial EEG during imagined hand movement,” IEEE Transaction on Rehabilitation Engineering, 8(4):441-446 (2000)). Spatial filtering has also been used to improve the signal-to-noise ratio (SNR) of oscillatory activity. However, there has been no systematic effort to choose optimal spatial filters. In the context of the ERN, Gehring et al. (1993) use linear discrimination to identify characteristic time courses in individual electrodes, but do not exploit spatial information. Although many of these aforementioned methods obtain promising performance in terms of classifying covert (purely mental) processes, their neurological interpretation remains obscured.
It is therefore an object of this invention to provide a system and method which will maximize performance of a BCI.
It is a further object of this invention to provide a system and method which will yield good single trial discrimination in a relatively short period of time.
These and other objects are accomplished by use of conventional linear discrimination to compute the optimal spatial integration of a large array of brain activity sensors. This allows exploitation of timing information by discriminating and averaging within a short time window relative to a given external event. Linear integration permits the computation of spatial distributions of the discriminating component activity, which in turn can be compared to functional neuroanatomy to evaluate the validity of the result. The term “component” instead of “source” is preferred so as to avoid confusion with an implied physiological source.
Further objects, features, and advantages of the present invention will become apparent from the following detailed description taken in conjunction with the accompanying drawings showing illustrative embodiments of the present invention, in which:
An exemplary embodiment of the system of the present invention is shown in
The signal processing performed by signal processing PC 26 may be broken into two types—linear discrimination and localization of discriminating components.
As described below, a logistic regression model is used to learn an optimal linear discriminator using the spatial distribution of EEG activity across a high-density sensor array. Denoting x(t) as the M sensor values sampled at time instance t, spatial weighting coefficients v are computed such that
In order to provide a functional neuroanatomical interpretation of the resultant spatial weighting, a forward linear model is used to determine “sensor projections” of the discriminating component activity. In this model, y(t) is treated as a source which is maximally discriminating given the linear model and task. A simple way of visualizing the origin of a source's activity is to display the coupling coefficients of the source with the sensors. The strength of the coupling roughly indicates the closeness of the source to the sensor as well as its orientation. The coupling a is defined as the coefficients that multiply the putative source y(t) to give its additive contribution xy(t) to the sensor readings, xy(t)=ay(t). However, xy(t) is not observable in isolation; instead we observe, x(t)=xy(t)+xy′(t), where xy′(t) represents the activity that is not due to the discriminating component. If the contributions, xy′(t), of other sources are uncorrelated with y(t) we obtain the coupling coefficients by the least-squares solution (Haykin, Adaptive Filter Theory, Englewood Cliffs, N.J., Prentice-Hall, (1996)). Arranging the samples x(t) for different t as columns in the a matrix X, and y(t) as a column vector y the solution is given by
In general other sources are not guaranteed to be uncorrelated with the discriminating component. Therefore a represents the coupling of all component activities that are correlated to the discriminating component y(t). We refer to a as a “sensor projection,” as it measures the activity in the sensors that correlate with a given component. Our approach relies on the linearity of y(t) and the fact that different sources in EEG and MEG add linearly (Baillet, S. et al., “Electromagnetic brain mapping.” IEEE Signal Processing Magazine, 18(6): 14-30, 2001).
Sensor projection a was derived as follows. Assuming the observation vector is x, a linear classifier, y1=vTx, can be built where y1 is the binary number indicating some cognitive event that we are trying to detect. A number of such cognitive events occurring simultaneously is assumed. These are represented as a vector of binary indicators y, with y1 as its first element, and a matrix A that maps these to the observation vectors; i.e., x=Ay. Without restriction y is normalized to be zero mean. We wish to identify this mapping, namely to find the first column of A, which we call a and which is defined as the observation vector that would be obtained if only y1 occurred. The most likely a can be found as follows. Let X be the zero mean observation matrix for many samples, i.e., the tth column is the observation for the tth sample. Let y1 T be the corresponding binary column vectors across these samples given by y1=vTX. The definition for a implies X=ay1. The maximum likelihood estimate for a, given v and X, is given by the least-squares solution, a=Xy1 T(y1y1 T)−1 We would like to determine the conditions under which the least-squares estimate of a is actually proportional to the first column of A. Let the matrix Y be the binary matrix of the simultaneous cognitive events across trials, i.e., the tth column is the cognitive events vector y for the tth trial. Since X=AY, we find that a=AYy1 T(y1y1 T)−1. Note that Y has dimensions of number of cognitive events (N) by number of samples (T), and that the quantity Yy1 T is the column vector of unnormalized correlations between the event indicators y1 and the set of all cognitive events. If this is proportional to the Kronecker delta, δi,1 (i.e., y1 is uncorrelated with the indicators of the other events), then αi∞ΣjAi,jδi,1=Ai,1, and therefore a is proportional to the first column of A.
Predicting explicit (overt) motor response using MEG: Four subjects performed a visual-motor integration task. A “trump” experiment was defined whereby subjects were simultaneously presented with two visual stimuli on a CRT, one of which is the target and “trumps” (beats-out) the other. Subjects were instructed to push a left hand or right hand button, depending on which side the target (trump stimulus) was present. The subject was to discover the target by trial and error using auditory feedback. Each trial began with visual stimulus onset, followed by button push, followed by auditory feedback, indicating if the subject responded correctly. The interval between the motor-response and the next stimulus presentation was 3.0±0.5 sec. Each subject performed 90 trials, which took approximately 10 minutes. MEG data was recorded using 122 sensor at a sampling rate of 300 Hz and high-pass filtered to remove DC drifts. Dipole fits were done using the “xfit” tools available from Neuromag (www.neuromag.com), which assume a spherical head model to find a single equivalent current dipole.
Classifying imagined (covert) motor activity using EEG: Nine subjects performed a visual stimulus driven finger (L/R) tapping task. Subjects were asked to synchronize an explicit or imagined tap by the left, right, or both index fingers to the presentation of a brief temporally predictable signal. Subjects were trained until their explicit taps occurred consistently within 100 ms of the synchronization signal. Subjects were presented visual stimuli indicating with which index finger to tap and if it should be an explicit or imagined tap. 1.25 seconds after the last instruction symbol a fixation point was replaced for 50 ms by the letter “X.” This letter served as a signal to which the instructed tap (whether overt or imagined) was to be synchronized. Each trial lasted for 6 s. After training, each subject received 10 blocks of trials. Each 72-trial block consisted of nine replications of the eight trial types (Explicit vs. Imagined×Left vs. Right vs. Both vs. No Tap) presented in a random order. Trials with noise due to eye blinks were not considered in the EEG analysis. The electromyogram (EMG) was recorded to detect muscle activity during imagined movements. The 59 EEG channels were sampled at 100 Hz and high-pass filtered to remove DC components.
Detection of decision errors from EEG: Seven subjects performed a visual target detection amongst distractors task. On each trial, subjects were presented with a stimulus for 100 ms. There were four possible stimuli, each consisting of a row of five arrows. Subjects were told to respond by pressing a key on the side indicated by the center arrow. They were to ignore the four flanking arrows. On half of the trials, the flanking arrows pointed in the same direction as the target (e.g. <<<<<), on the other half the flankers pointed in the opposite direction (e.g. <<><<). Subjects were slower and made many more errors in the latter case. Following their response, there was an inter-trial interval of 1.5 seconds, after which a new stimulus was presented. Subjects performed 12 blocks of 68 trials each. The 100 ms interval prior to the response was used as the baseline period (separately for each trial and electrode). The sampling rate was 250 Hz. Following the baseline period, trials were manually edited to remove those with blinks, large eye movements, instrument artifacts and amplifier saturation.
Single trial discrimination results are shown for Examples I-III and include trial averaged discriminating component activity
As seen in Table 1, for all three data sets the number of trials for training is comparable to the number of coefficients to be trained. This can lead to serious problems in overtraining. We mitigate these by including multiple training samples for each trial. These samples are obviously not independent; however, they provide evidence for the natural variation of the data and thus make the estimates much more robust. They were shown, through cross-validation, to improve estimated generalization performance. We would expect that increasing the number of independent training samples (e.g., trials) would similarly increase performance of the results presented below.
The sensor projection of the 59 EEG sensors shows a clear left-right polarization over the motor area. In the context of BCI the metric of interest is the bit rate of at which information can be transmitted with imagined motor activity. The information transmitted per trial is given by,
For comparison, an alternative method, first described by Wolpaw et al. (1991), that is based on differences in the power spectrum in electrodes over the left and right motor cortex was also tested. Andersen et al., (“Multivariate Autoregressive Models for Classification of Spontaneous Electroencephalogram During Mental Tasks,” IEEE Transactions on Biomedical Engineering, 45(3):277-286, (1998)) modifies the approach by using six auto-regressive (AR) coefficients to model the power spectrum of each electrode within the analysis window and classify the imagined conditions using a linear discrimination on these AR coefficients. Following Penny et al. (2000), we used electrodes C3 and C4 (international 10/20 electrode placement system—see Towle et al., “The spatial location of EEG electrodes: locating the best-fitting sphere relative to cortical anatomy,” Electroencephalogr Clin. Neurophysiol., 86(1): 1-6, (1993)) and obtain Az=0.65±0.09, and fraction correct of p=0.62±0.07, which corresponds to I=0.054 bits/trial or a bit rate of 4 bit/minute. This is about a fourth of the results obtained with our proposed method.
The results, across the nine subjects, for predicting explicit finger taps from a window 300 ms to 100 ms prior to the taps is Az=0.87±0.08 and a fraction correct of 0.80±0.08. As shown in
The results of Examples I-III demonstrate the utility of linear analysis methods for discriminating between different events in single-trial, stimulus driven experimental paradigms using EEG and MEG. An important aspect of our approach is that linearity enables the computation of sensor projections for the optimally discriminating weighting. This localization can be compared to the functional neuroanatomy, serving as a validation of the data driven linear methods. In all three examples, the activity distribution correlated with the source that optimizes single-trial discrimination localizes to a region that is consistent with the functional neuroanatomy. This is important, for instance in order to determine whether the discrimination model is capturing information directly related to the underlying task-dependent cortical activity, or is instead exploiting an indirect cortical response or other physiological signals correlated with the task (e.g. correlations with the stimulus, eye movements, etc.). Localization of the discriminating component activity and its correlates also enables one to determine the neuroanatomical correlations between different discrimination tasks, as was demonstrated for explicit and imagined motor responses in EEG.
While this invention has been described with reference to several illustrative examples and embodiments, they should not be interpreted as limiting the scope or spirit of the invention. In actual practice many modifications may be made by those of ordinary skill in the art without deviating from the scope of the invention as expressed in the appended claims. For example, the system and method of the present invention may be applied to other encephalographic modalities with linear superposition of activity, such as functional infrared imaging (Boas et al., “Imaging the body with diffuse optical tomography.” IEEE Signal Processing Magazine, 18(6): 57-75, (2001)).