Wouldn’t it be super cool to possess a machine that can read other people’s thoughts and intentions? … No more secrets that could be kept from you – no more mind games that could be played with you…

---

Even if there are numerous ethical and practical concerns, the science fiction-like idea of a mind-reading device is an intriguing prospect. While already a mainstay of sci-fi movies – for example Doc Brown and his brain wave analyser in “Back to the Future” – is there actually any chance that neuroscience will be able to provide us with this seemingly futuristic technology?

Back to Reality!

With our current technology and scientific understanding, complex mind-reading is nothing more than a fictitious idea. However, within the field of vision research, a variety of studies have taken the tentative first steps towards inferring mental states from brain data.

In the following, we will review some of the mind-boggling results and discuss the methods employed. The studies discussed below share two common aspects:

1. They deal with the visual system.
2. Brain data is acquired via functional magnetic resonance imaging (fMRI).

What the f-MRI?

f-MRI is a popular neuroimaging technique that indirectly captures brain activity by detecting changes in oxygen in cerebral blood. Traditional fMRI studies investigate the brain’s response to manipulations of psychological functions. Hence, fMRI potentially informs us about the brain areas that are associated with specific mental processes.

Mind-reading, in this context, would constitute a reversed approach namely inferring mental processes from brain activity pattern. So why don’t we just use fMRI to read what people are thinking?

fMRI: Silver bullet to read minds?

Due to a plethora of reasons, mind-reading via fMRI is far more complicated than it might appear at first sight:

Firstly, naïve “reverse inference”, where one would attempt to cold read mental state based on brain activation is problematic. This results from the fact that every brain area is involved in a multitude of mental functions and, conversely, that every mental process engages various brain areas.

Perhaps of even more importance, many limitations are related to the measurement of brain activation by fMRI itself. Specifically:

• FMRI merely captures oxygen consumption in the blood as a proxy for neuronal activity – although, it has been proven repeatedly that the both are coupled.
• Obtained fMRI-data includes a lot of noise besides “meaningful” signal.
• Spatial and particularly temporal resolution of fMRI is not perfect.

Even if we could accurately measure neural activation patterns, it still wouldn’t necessarily imply that mind-reading is feasible. It is inarguable that our brains are among the most complex structures in the universe. How should we be able to read minds, if we do not even understand the full complexity of the human brain? With this question in the back of our minds, let us finally move on to some studies.

One part of the brain that has been studied extensively is the visual system, which makes it the perfect candidate for “model-based” reverse inference. In this field, a sizeable number of studies has been published during the last years, some of which have attracted great media attention.

Probably most impressive is one landmark study from 2011 of Jack Gallant’s Lab at UC Berkley. Using fMRI and sophisticated computer algorithms, researchers reconstructed short movies clips from brain activity. You can see the fascinating results below in the short 30s clip!

But wait … how was this even possible?

Put simply, reconstruction was the result of a 2-step procedure: encoding and decoding. During the experiment, participants watched several hours of movie footage while lying in the scanner. Importantly, two separate data sets were obtained for all subjects: training and test data.

Encoding

With the training data, predictive encoding models were generated for all participants matching brain activity patterns with shapes, movements and edges of the movies on the screen. The obtained models were then used to predict brain activity for short (1s) video clips that were not shown in the experiment. Hence, a large library of movie clips and their associated (modelled) brain activity was compiled – and this for every subject individually.

Decoding            

Subsequently, the recorded test data were compared to the library of predicted brain activations. For every short movie-reconstruction, several clips were taken from the library whose predicted activity most closely resembled what was recorded in the scanner. To complete reconstruction, these were then averaged and superimposed: Et voilà, this is what you saw in the video!

In summary, both encoding and decoding rest on the fact that stimuli characteristics are represented by specific patterns of activity. Encoding predicts activity for unseen stimuli. Decoding, conversely uses recorded activity to predict stimuli characteristics. Click here if you want to find out more about encoding and decoding in f-MRI!

The 2011 movie-study is a great demonstration of dynamic visual reconstruction. “Dynamic” implies that visual input & reconstruction was not only limited to static pictures, as in previous publications. To mention other interesting findings, a study from 2014 demonstrated remarkable reconstruction of faces.  – Click here if you want to take a look at the beautiful reconstruction!

Besides reconstruction, earlier approaches of visual decoding concerned themselves with more basic instances of image classification or identification.

On to classification and identification of images!

Image classification refers to determining the category of an observed stimulus. For instance, it is possible to predict with high accuracy if a person looked at a face or a house, since both concepts evoke specific patterns of activity. However, for this to work, stimulus specific activation patterns obviously need to be available. Therefore, classification only works for predefined categories of interest.

As the name suggests, image identification entails the attempt to identify the image that had been shown to an observer out of a large set of potential images. In this case, the decision (= identification) relies on comparing recorded activation (while the target image was shown) with modelled activations (for every single potential image). One study, for instance, achieved successful identification of natural images, even when the set of potential images was very large (~1000 images that were not shown + the target image). However, identification sometimes failed, especially if target and selected image were similar in terms of visual features.

At this point it should be noted again that most visual decoding studies rely on developing individual models for each individual participant.

Classifying-Identifying mental images

Perhaps even more interesting than only determining what a person sees, classification and identification can furthermore be applied to imagined content. Researchers demonstrated that they could accurately predict which one of ten possible objects (e.g. a screwdriver) a person was imagining. Notably, this approach did not even require within-person modelling.

In another more recent fMRI study, participants were told to mentalize familiar pieces of art which then could be identified by a computer algorithm. If pursued further, it is conceivable that this could one day enable imagination-based online image searches, which of course is extremely exciting – especially for research on brain computer interfaces.

The future of decoding.

Ultimately, potential future applications for visual decoding and other “mind-reading” approaches are numerous. Ideas range from communication with coma patients over dream analysis to uses in the criminal justice system. In fact, these ideas are not only futuristic visions:

1. A study from 2010 investigated mental imagery (i.e. visualizing the act of spatial navigation or playing tennis) in patients with impaired consciousness. Remarkably, one subject, who seemed to be in an entirely vegetative state according to clinical testing, communicated simple yes or no answers by altering his brain activity through mental imagery.
2. In 2013, Japanese researchers attempted to reconstruct what people where dreaming based on recorded brain activity.
3. In 2008, India became the first country to convict a woman of murder based on EEG recordings. Obviously, this sparked great controversy in the scientific community due to ethical and validity related questions.

In conclusion, it should be noted that brain-reading based on neuroimaging data relies on purely correlational (probabilistic) models that in no way achieve 100% accuracy (i.e. images and categories are selected based on resemblance of modelled vs. recorded activities). Furthermore, the models usually require extensive “training” on individual data for every subject or allow for classification of only a very limited number of categories.

Thus, when it comes to the subject of mind-reading, it appears advisable to take all overly-enthusiastic statements from researchers and journalists (who probably aim to sell their scientific discoveries / news articles) with a grain of salt!