What comes to your mind when you think of language? It might be grammar or maybe literature. Or that one language you always wanted to learn but never found the time for (yet!). But lay back for a moment. Think about the wondrous fact that by reading these lines, the arbitrary symbols called letters create first words, then images and knowledge in your head. Impressive, isn’t it?

Have you ever wondered whether people also think in words or how meaning is stored in the brain? And if these “words” can be read? This week, we want to focus on how concepts about the world get into our heads, aka The Neuroscience of Semantics.

Painting pictures in your mind.

Are you in for a little experiment? Of course you are!

Kitty

Hooray, you just used your brain! Apart from the obvious (I’m sure there will be an article about reading at some point – please be patient) your brain just activated the area for living but non-human things. Also, it might have activated shape areas to remind you of cute round eyes and a fluffy tail, and the motor areas because you like to pet kittens. Maybe some olfactory areas for… you know, kitty litter or cat food cans, all depending on your prior experiences with the concept¹ [Fig. 1]. – Wait, but why?

Embodied Cognition.

The mind wide web
Although there are some defenders of the idea that everyone is born with a certain set of concepts in their minds (so-called Extreme Nativists²), current scientific evidence points not to a clearly bordered dictionary area, but to a complex neuronal semantic network system based on Neuroplasticity.

Quick recap: “Neurons that fire together wire together and neurons out of sync delink.”³ This key principle is called Hebb’s Axiom or correlation learning. Sounds complicated? — It really isn’t.

Fig 2: Co-activated areas form meaning concepts. “Eye” is a concrete word, subsuming many colors but similar shapes which have highly overlapping representations in their sensory-motor semantic feature neurons which fire together. “Beauty” is an abstract concept, where less features overlap so the relationship is weaker. Pulvermüller, 2013. (CC BY-NC-ND 4.0)

That means for semantics: Pronouncing a word goes hand in hand with activation in motor areas (facial muscles) but also with auditory and bodily perceptions (hearing yourself speak and feeling yourself move your lips) – all together, this leads to activation in the frontotemporal perisylvian cortex. But this is just the first part. You usually perceive the world through all your senses but some might be predominantly useful to distinguish one concept from the other.

Depending on the word’s meaning there are also category-specific semantic effects – activations close to sound- (superior temporal auditory cortex), smell- (piriform and interior insular olfactory cortex), taste- (anterior insular and frontal operculum), or action-related brain areas (motor cortex). Now, when the corresponding areas are often activated simultaneously (“fire together”) they become correlated and the network strengthens this association [Fig. 2]. But when you perceive something without receiving the expected perceptual stimulation (“out of sync”), the connection becomes weaker again.³

Depending on the theory, it is assumed that these lower level zones converge in either several task-dependent convergence zones, one single integration single hub, or first in bi-modal and later in single hubs^4,5,6 [Fig. 3].

 

Still following me? Stay tuned, I’m not continuing with more lists of brain areas.

When the body changes its mind
Some researchers postulate that our body influences cognition even on a very high level. An important figure in this discussion is George Lakoff who concluded that both mathematics⁷ and metaphors⁸ reflect an extremely body-centered representation of meaning (“feeling up” for “being in a good mood” that is linked to walking upright, f. ex.). So does this also work the other way around?

You might have heard of the prominent study⁹ claiming that holding a pen with your teeth in a way that occupies the same muscles as a smile, results in higher ratings of funniness of pictures. – This is called the facial feedback hypothesis. However, this year a huge replication study¹⁰ of 17 independent laboratories tried to confirm the findings using a meta-analysis but found no effect.  Just to be clear – I’m certainly not saying that it’s all nonsense — there could be many reasons for the discrepancy. But it does teach us once more not to blindly trust interesting results. You might want to check Strack’s response (author of the original study) though to make up your own mind.

How do we know where we know what we know?

Clinical studies
In clinical studies, the lack of a certain ability can help pinpoint locations in the brain and mechanisms behind that ability. For example, semantic dementia impairs meaning recognition, amongst other symptoms. When the disease spreads forward from the temporal pole to the frontotemporal cortex, we can see that the impairment location influences face- and color-related words significantly but not shape- or arm-related words.

Findings like these speak in favor of category-based circuits (– also like living vs. non-living things in the kitty example)¹¹ [Fig. 3]. The strong link between action and perception also shows in patients with Parkinson’s disease. As their motor functions become more limited, patients also show problems with the meaning of action-related verbs, but not with object-related nouns¹². That’s how we know they must have different representations in the brain.

Of course, much of our information also stems from brain imaging and computational models – such as in our next example.

Meaningful blobs on a screen

A recent fMRI study¹³ by Alexander Huth and colleagues from Gallant’s Lab (2016) paints a colorful (and interactive!) picture of the semantic representations of the brain. Despite the rather misleading title of the video “The brain dictionary” [Fig. 4] – as explained before, it should be rather a net or a map — the researchers could show once more the concept representation in clusters throughout the brain (tactile, visual, numeric, locational, abstract, temporal, professional, violent, communal, mental, emotional, social).

Their approach was a bit different than usual though. In their research, Huth studied seven participants who were passively listening to two hours of stories while they were being scanned in an MRI machine. Usually studies on semantics would rather have separate words as their material in order to distinguish them better. Here, the authors used this new approach to reflect the natural language of the real world.

Interestingly, it seems that the active locations were comparable across participants Therefore, Huth used the data to create “a probabilistic and generative model of areas tiling the cortex”. According to the study, with the data of six participants, the model could predict the activation in the seventh to a high degree.

Fig. 4. “The Brain Dictionary”. Video summary of the introduced study.

Mind reading and the Future

So, scientists may be just satisfied with only gaining more knowledge about the brain and about meaning encoding. But what else is this information good for, you might ask (except for bragging at parties of course – knowledge is attractive, right?).

Eventually, in the far future, it might be possible to use semantic maps for understanding so-called locked-in patients, who look like they’re in a coma because they can’t move at all, but are actually completely conscious. Training a computer on recognizing patterns of meaning in the brain could be a key to help those people communicate with the outside world.

Actually, no other than that same Gallant’s Lab¹⁴ has made an impressive attempt in that direction, researching the much better understood visual system. The researchers were able to teach the computational program neural responses to many different videos. Then, by matching the resulting model to the activation in the same individual but to newly introduced material, they were not only able to name but even reconstruct the videos (click it, it’s awesome!).¹⁴

However, it is important to point out that “The meanings might be stored in the same area, but the actual neurons would be idiosyncratic.”, as Matt Davis from the University of Cambridge, UK, commented about semantic mind reading in 2012.“You would have to scan a person as they thought their way through a dictionary”. This emphasizes that predictions can often only be made on an individual basis.

However, if Huth’s new finding¹³ holds true and the presented semantic map proofs to be comparable across individuals when derived from natural speech, we might be a step closer to helping locked-in patients – without the need to safe personal brain dictionaries on our future ID cards. But for now, as they mapped broader concepts, not single words, we would only get an overall idea about their thoughts – not a translation, yet.

Side-note:
As in all good science practice, the presented view is constantly challenged by integrating and opposing theories¹⁵ — Critics never sleep. If you’re curious about a more extensive overview and discussion of the current directions, take a look at this review¹⁶ by one of the key researchers in the field: Lawrence Barsalou.

References

Literature
¹ Not an actual example from a study but hopefully illustrates the idea!
² Pinker, S. (1997). How the mind works. New York, London: Norton.
³ Pulvermuller, F., & Fadiga, L. (2010). Active perception: sensorimotor circuits as a cortical basis for language. Nature reviews. Neuroscience, 11(5), 351–360. doi:10.1038/nrn2811
⁴ Patterson, K., Nestor, P. J., & Rogers, T. T. (2007). Where do you know what you know? The representation of semantic knowledge in the human brain. Nature reviews. Neuroscience, 8(12), 976–987. doi:10.1038/nrn2277
⁵ Barsalou, L. W., Kyle Simmons, W., Barbey, A. K., & Wilson, C. D. (2003). Grounding conceptual knowledge in modality-specific systems. Trends in Cognitive Sciences, 7(2), 84–91. doi:10.1016/S1364-6613(02)00029-3
⁶ Fernandino, L., Binder, J. R., Desai, R. H., Pendl, S. L., Humphries, C. J., Gross, W. L.,. . . Seidenberg, M. S. (2016). Concept Representation Reflects Multimodal Abstraction: A Framework for Embodied Semantics. Cerebral cortex (New York, N.Y. : 1991), 26(5), 2018–2034. doi:10.1093/cercor/bhv020
⁷ Lakoff, G., & Núñez, R. E. (2000). Where mathematics comes from: How the embodied mind brings mathematics into being (1st ed.). New York, NY: Basic Books.
⁸ Lakoff, G., & Johnson, M. (2003). Metaphors we live by. Chicago, Ill., London: University of Chicago Press. Retrieved from http://www.loc.gov/catdir/description/uchi051/80010783.html
⁹ Strack, F., Martin, L. L., & Stepper, S. (1988). Inhibiting and facilitating conditions of the human smile: A nonobtrusive test of the facial feedback hypothesis. Journal of Personality and Social Psychology, 54, 768-777.
¹⁰ E.-J. Wagenmakers, T. Beek, L. Dijkhoff, & Q. F. Gronau (Proposal). Registred Replication Report: Strack, Martin,  & Stepper (1988). Association for Psychological Science.
¹¹ Pulvermuller, F., Cooper-Pye, E., Dine, C., Hauk, O., Nestor, P. J., & Patterson, K. (2010). The word processing deficit in semantic dementia: all categories are equal, but some categories are more equal than others. Journal of cognitive neuroscience, 22(9), 2027–2041. doi:10.1162/jocn.2009.21339
¹² Boulenger, V., Mechtouff, L., Thobois, S., Broussolle, E., Jeannerod, M., & Nazir, T. A. (2008). Word processing in Parkinson’s disease is impaired for action verbs but not for concrete nouns. Neuropsychologia, 46(2), 743–756. doi:10.1016/j.neuropsychologia.2007.10.007
¹³ Huth, A. G., Heer, W. A. de, Griffiths, T. L., Theunissen, F. E., & Gallant, J. L. (2016). Natural speech reveals the semantic maps that tile human cerebral cortex. Nature, 532(7600), 453–458. doi:10.1038/nature17637
¹⁴ Nishimoto, S., Vu, A. T., Naselaris, T., Benjamini, Y., Yu, B., & Gallant, J. L. (2011). Reconstructing visual experiences from brain activity evoked by natural movies. Current biology : CB, 21(19), 1641–1646. doi:10.1016/j.cub.2011.08.031
¹⁵Leshinskaya, A., & Caramazza, A. (2016). For a cognitive neuroscience of concepts: Moving beyond the grounding issue. Psychonomic bulletin & review, 23(4), 991–1001. doi:10.3758/s13423-015-0870-z
¹⁶ Barsalou, L. W. (2016). On Staying Grounded and Avoiding Quixotic Dead Ends. Psychonomic bulletin & review, 23(4), 1122–1142. doi:10.3758/s13423-016-1028-3

Figures
¹ Boen, C. (2014). Varys in deep cat space [Online image]. Flickr. Retrieved on November 9, 2016 from http://bit.ly/2emFcD7.
² Pulvermüller, F. (2013). How neurons make meaning: brain mechanisms for embodied and abstract-symbolic semantics. Trends in Cognitive Sciences, 17(9), 458–470. doi:10.1016/j.tics.2013.06.004
³ Pulvermüller, F. (2013). How neurons make meaning: brain mechanisms for embodied and abstract-symbolic semantics. Trends in Cognitive Sciences, 17(9), 458–470. doi:10.1016/j.tics.2013.06.004
⁴ Nature Video. (2016, April 27). The brain dictionary [Video file]. Retrieved on November 9, 2016 from https://www.youtube.com/watch?v=k61nJkx5aDQ