PubPeer: May 29, 2017
Unregistered Submission:
(May 25th, 2017 2:46 am UTC)
In this review the authors attempted to estimate the information generated by neural signals used in different Brain Machine Interface (BMI) studies to compare performances. It seems that the authors have neglected critical assumptions of the estimation technique they used, a mistake that, if confirmed, completely invalidates the results of the main point of their article, compromising their conclusions.
Figure 1 legend states that the bits per trial from 26 BMI studies were estimated using Wolpaw’s information transfer rate method (ITR), an approximation of Shannon’s full mutual information channel theory, with the following expression:
Bits/trial = log2N + P log2P + (1-P) log2[(1-P)/(N-1)]
where N is the number of possible choices (the number of targets in a center-out task as used by the authors) and P is the probability that the desired choice will be selected (used as percent of correct trials by the authors). The estimated bits per trial and bits per second of the 26 studies are shown in Table 1 and represented as histograms in Figure 1C and 1D respectively.
Wolpaw’s approximation used by the authors is valid only if several strict assumptions are true: i) BMI are memoryless and stable discrete transmission channels, ii) all the output commands are equally likely to be selected, iii) P is the same for all choices, and the error is equally distributed among all remaining choices (Wolpaw et al., 1998, Yuan et al, 2013; Thompson et al., 2014). The violation of the assumptions of Wolpaw’s approximation leads to incorrect ITR estimations (Yuan et al, 2013). Because BMI systems typically do not fulfill several of these assumptions, particularly those of uniform selection probability and uniform classification error distribution, researchers are encouraged to be careful in reporting ITR, especially when they are using ITR for comparisons between different BMI systems (Thompson et al. 2014). Yet, Tehovnik et al. 2013 failed in reporting whether the assumptions for Wolpaw’s approximation were true or not for the 26 studies they used. Such omission invalidates their estimations. Additionally, the inspection of the original studies reveals the authors failed at the fundamental aspect of understanding and interpreting the tasks used in some of them. This failure led to incorrect input values for their estimations in at least 2 studies.
The validity of the estimated bits/trial and bits/second presented in Figure 1 and Table 1 is crucial to the credibility of the main conclusions of the review. If these estimations are incorrect, as they seem to be, it would invalidate the main claim of the review, which is the low performance of BMI systems. It will also raise doubts on the remaining points argued by the authors, making their claims substantially weaker. Another review published by the same group (Tehovnik and Chen 2015), which used the estimations from the current one, would be also compromised in its conclusions. In summary, for this review to be considered, the authors must include the ways in which the analyzed BMI studies violate or not the ITR assumptions.
References
Tehovnik EJ, Woods LC, Slocum WM (2013) Transfer of information by BMI. Neuroscience 255:134–46.
Shannon C E and Weaver W (1964) The Mathematical Theory of Communication (Urbana, IL: University of Illinois Press).
Wolpaw J R, Ramoser H, McFarland DJ, Pfurtscheller G (1998) EEG-based communication: improved accuracy by response verification IEEE Trans. Rehabil. Eng. 6:326–33.
Thompson DE, Quitadamo LR, Mainardi L, Laghari KU, Gao S, Kindermans PJ, Simeral JD, Fazel-Rezai R, Matteucci M, Falk TH, Bianchi L, Chestek CA, Huggins JE (2014) Performance measurement for brain-computer or brain-machine interfaces: a tutorial. J. Neural Eng. 11(3):035001.
Yuan P, Gao X, Allison B, Wang Y, Bin G, Gao S (2013) A study of the existing problems of estimating the information transfer rate in online brain–computer interfaces. J. Neural Eng. 10:026014.
We acknowledge in Tehovnik and Chen (2015, pp. 3343) that the metric used to deduce information transfer using the formula [Bits = Log2 N + P Log2 P + (1 - P) Log2 {(1 - P)/(N - 1)}, where N is the number of targets and P is the percent correctness score] overlooks issues such as task target size, for example, which varied from study to study across the 26 studies—based on the center-out task—as examined by Tehovnik et al. (2013). Nevertheless even with this exclusion, the transfer rates on the center-out task were less than or equal to 1 bit per second.
What is clear from our analysis (Tehovnik et al. 2013) is that many studies done in the field of brain-machine interfaces (BMIs) are contaminated by body movements which can inadvertently enhance BMI performance in behaving monkeys (e.g. O’Doherty et al. 2011, Fig. 2d). Body movements have also been suggested to enhance such performance in paralyzed patients (Lebedev and Nicolelis 2017, pp. 789). It is noteworthy that extremely paralyzed patients devoid of all body movements fail to control prosthetic devices neurally (Birbaumer 2006; Nicolas-Alonso and Gomez-Gil 2012). One of the highest information transfer rates for BMI (i.e. 5.3 bits/sec) is achieved in normal subjects using EEG when their eye movements are allowed to signal the location of letters on a letter board (Chen et al. 2015). Thus far such a technology has not been tested on paralyzed patients.
The information transfer rate of Stephen Hawking, who suffers from Lou Gehrig’s disease (or ALS), is estimated to be 0.39 bits per second (De Lange 2011). This rate overlaps with the performance of current BMIs based on the center-out task (Tehovnik et al. 2013). At the moment BMIs fall short of the information transfer rate of the cochlear implant which transfers 10 bits per second (Baranauskas 2014) and which is one of the most successful prosthetic devices on record. The recent demonstration of an EEG-driven BMI by a paralyzed patient at the 2014 FIFA World Cup (see Regalado 2014) does not provide much confidence that a prosthetic device will be available in the short-term to improve conditions for the likes of Stephen Hawking (but see Pandarinath et al. 2017). The challenge for BMI, therefore, is to enhance the information transfer rate such that discussions about information theory become moot.
References
Baranauskas G (2014) What limits the performance of current invasive brain machine interfaces. Frontiers Systems Neuroscience doi:10.3389/fnsys.2014.00068.
Birbaumer N (2006) Breaking the silence: Brain-computer interfaces (BCI) for communication and motor control. Psychophysiology 43:517-532.
Chen X, Wang Y, Nakanishi M, Gao X, Jung T-P, Gao S (2015) High-speed spelling with a noninvasive brain-computer interface. Proceedings of the National Academy of Sciences USA doi:10.1073/pnas.1508080112.
De Lange C (2011) The man who saved Stephen Hawking’s voice. New Scientist, December issue.
Lebedev MA, Nicolelis MAL (2017) Brain-machine interfaces: from basic science to neuroprostheses and neurorehabilitation. Physiological Review 97:767-837.
Nicolas-Alonso LF, Gomez-Gil J (2012) Brain computer interfaces, a review. Sensors 12:1211-1279.
O’Doherty JE, Lebedev MA, Ifft PJ, Zhuang KZ, Shokur S, Bleuler H, Nicolelis MA (2011) Active tactile exploration using a brain-machine-brain interface. Nature 479:228-231.
Pandarinath C, Nuyujukian P, Blabe CH, Sorice BL, Saab J, Willett FR, Hochberg LR, Shenoy KV, Henderson JM (2017) High performance communication by people with paralysis using an intracortical brain-computer interface. eLife 6, e18554:1-27.
Regalado A (2014) The top technology failures of 2014. MIT Technology Review, Dec. 31.
Tehovnik EJ, Chen LL (2015) Brain control and information transfer. Experimental Brain Research 233:3335-3347.
Tehovnik EJ, Woods LC, Slocum WM (2013) Transfer of information by BMI. Neuroscience 255:134-146.
See recent study by Pandarinath et al. (2017), particularly their Table 1 that lists the information transfer rates of many studies.
Full reference:
Pandarinath C, Nuyujukian P, Blabe CH, Sorice BL, Saab J, Willett FR, Hochberg LR, Shenoy KV, Henderson JM (2017) High performance communication by people with paralysis using an intracortical brain-computer interface. eLife 6, e18554:1-27.
https://elifesciences.org/articles/18554
Fitts’ Law and Brain-machine Interfaces according to Willett et al. (2017):
Reaching movements typically obey Fitts’ law: MT = a + b log2 (D/R) where MT is movement time, D is target distance, R is target radius, and a & b are parameters. Fitts’ law describes two properties that would be ideal for brain-machine interfaces (BMIs): (1) that movement time is insensitive to the absolute scale of the task since the time depends on the ratio of D/R and (2) that movements have a large dynamic range of accuracy since movement time is logarithmically proportional to D/R. Movement times for BMI (based on motor cortex electrophysiological recordings from two tetraplegics performing a center-out task) were better described by the formula, MT = a + bD + cR(-2 pow), since the movement time increased as the target radius became smaller, independent of target distance. The mismatch between reaching movement and BMI-generated movement was determined to be due the signal-independent noise of the decoder for BMI which makes targets below a certain size very difficult to acquire in a timely manner. This would reduce the information transfer rate by BMI when using small targets.
For the complete article see: Willett FR, Murphy BA, Memberg WD, Blabe CH, Pandarinath C, et al. (2017) Signal-independent noise in intracortical brain-computer interfaces causes movement time properties inconsistent with Fitts’ law. J. Neural Eng. 14:026010.
- Badges
- Science topic