Research Briefing: Does feedback in the brain rebuild our visual world?

Peripheral vision is useful. Whether you’re an astronomer who practices ‘averted vision’, or a footballer monitoring for defenders, or even just attending covertly to a weirdo on the Tube – in each situation, looking directly at what you’re attending to may not be the best course of action.

Evidence is now mounting that when we attend to objects in the periphery, critical information about them is transmitted, or ‘fed back’, to an unexpected part of the brain: a region that neuroscientists have traditionally believed represents only the ‘fovea’, our central visual field.

Exactly why this feedback of information occurs isn’t yet clear. But one possibility is that the visual system harnesses the resolving power of the foveal cortex to enhance our peripheral vision, much like a small-town police force sending important CCTV footage to the ‘big city’ for analysis. 

------------------------------------------------------------------------------------------------------------------------ 
Source article: Chambers, C.D., Allen, C.P.G., Maizey, L. & Williams, M.A. (in press). Is delayed foveal feedback critical for extra-foveal perception? Cortex. [Download here].

------------------------------------------------------------------------------------------------------------------------

When Aristotle stared up at the night sky, he noticed something interesting about the appearance of a very faint star: “to those who concentrated their gaze straight at it, the light would become dim, but to those who moved their vision slightly to the side, the light would grow stronger”. Centuries later, biologists would discover that the reason such faint stars became more visible was because peripheral regions of the retina contain a very high density of rods, a type of nerve cell that is particularly sensitive to subtle differences in brightness. Nowadays the practice of ‘averted vision’ is well known in optical astronomy, and is just one example of how useful peripheral vision can be.

Our peripheral vision is handy for detecting very faint or moving objects, but it has much lower acuity than the central part of the retina, called the fovea. The fovea is centred on the current point of gaze, and is about 2 degrees across. If you hold out your thumb at arm’s length, the fovea is about twice the size of your thumbnail.

The fovea is the business centre of the visual system. Even though it occupies only 1% of the retina, it takes up about 50% of the primary visual cortex in the brain. What this means is that the foveal part of visual cortex has a very high resolving power: at 10° degrees from the fovea, visual acuity drops by a factor of five, at 20° degrees by a factor of ten.

We can also think of these numbers in terms of the amount of the cortex devoted to each degree of the visual field. Neuroscientists call this the cortical magnification factor, which can be thought of as a neuronal ‘staff-to-customer ratio’. At the centre of the fovea, every mm of the cortex is devoted to just 0.15° of visual angle (premier service), whereas if we move 20° away from the centre, the same amount of cortex now has to deal with an area ten times larger (budget service).

For many years it has been believed, quite sensibly, that these parts of the visual cortex are hardwired to produce a fixed retinotopic map. This means that there is a fixed relationship between the location of a stimulus on the retina and the part of the visual cortex that represents that stimulus. Because of this correspondence, stimuli that occur beyond the fovea should not be represented in foveal retinotopic cortex.

A puzzle emerges

But in 2008, Mark Williams and colleagues made a surprising discovery. Using functional magnetic resonance imaging (fMRI), they found that when people paid attention covertly to stimuli in the periphery, brain activity in the foveal visual cortex contained information about the stimuli. Specifically, they showed that the pattern of brain activity in the foveal cortex could reveal whether two stimuli in the periphery were the same or different.

How is this possible? The foveal cortex should have been blind to the stimuli because they didn’t occur on the fovea. Williams and colleagues ruled out a series of obvious explanations for their results, before concluding that information was somehow being fed back from the periphery to the foveal cortex.

Key results from the study of Williams et al. (2008). The upper panel (a) shows the different brain regions of interest, or ‘ROIs’. Notice how the peripheral stimuli (yellow) clearly activated the visual cortex beyond the foveal representation (in red). The lower panel (b) shows the results of one experiment. Information about the peripheral stimuli is reflected by the difference between the black and grey bars. The information was statistically reliable only at the foveal location. Mark and colleagues replicated this result six times in the one paper.

When this study appeared my initial reaction was that it proposed an important rethink about the nature of ‘feedback’. In cognitive neuroscience, feedback is what happens when the higher and more complex regions of the brain, such as the frontal cortex, send signals back to lower-level regions, including the visual cortex.

Feedback occurs after information is fed forward into the system, creating a series of feedforward/feedback loops. The function of feedback is quite mysterious, but cognitive neuroscientists generally believe it involves tuning and pruning: sharpening sensory information and focusing brain resources on the most important events. Some researchers, such as Victor Lamme, have also posed the question of whether feedback may be the basis of consciousness.

There are many theories of how feedback works, but one idea they all share is that it influences existing neural representations. So, for instance, if I wanted to use my peripheral vision to identify an object, traditional theories would predict that feedback from higher brain regions helps by boosting the representation of that object, in the region of visual cortex that is hardwired to represent that specific peripheral location.

But Mark Williams and colleagues found evidence for a very different kind of feedback: one that doesn’t just tune existing brain activity but which instead constructs an entirely new representation, and at a location in the retinotopic map where, literally, nothing should be happening. 

This leads to us to ask whether feedback mechanisms are doing more than just tuning and pruning. Could feedback instead be rebuilding our visual world? There are reasons to think this might be a good idea. Feeding information to the foveal cortex could help enhance our peripheral vision by running the sensory data through the equivalent of a neuronal super computer: the most powerful and accurate processor of visual stimuli we have.

I thought Mark Williams’ study was convincing, and was deservedly published in one of the most competitive scientific journals. But Mark and I agreed that his study wasn’t entirely definitive. We saw two main problems. The first is a logical constraint of all brain-imaging studies: that because he measured brain activity, we don’t know whether the information at the fovea was necessary for perception. To conclude that foveal cortex is critical, we would need to disrupt that part of the brain and show an impairment of peripheral vision.

The second limitation is that fMRI measures a correlate of neural activity called the hemodynamic response, rather than the electrical neural activity itself. The hemodynamic response reflects changes in the supply of oxygenated blood to neurons, and takes several seconds to peak. But feedforward and feedback processes all occur within a split second, so with fMRI it is impossible to know whether the information in foveal cortex was feedback or feedforward.

To the lab

Mark and I decided that the technique of transcranial magnetic stimulation (TMS) could help overcome these drawbacks. TMS interrupts brain activity, so it is the natural complement to brain-imaging methods such as fMRI. TMS also has much better timing resolution than fMRI, so it could allow us to test not only whether the foveal retinotopic cortex is necessary for peripheral perception, but when it is needed in the rapid sequence of brain activity after a stimulus appears. If foveal cortex is needed relatively late in the timecourse of neural processes then we could say more conclusively that the information in foveal cortex is the product of feedback.

So a few summers ago, Mark came to Cardiff to get the study started. We collected a lot of data during that visit, aided by the fact that it was without doubt the most dismal August I have experienced in my life. It rained every day for two solid weeks. During that time I think Mark saw just three places: the house that me and my wife had just bought (in need of major renovation), the lab, and the pub. In terms of hours spent where, the pub ran a close second to the lab, which makes me ponder our place in Ed Yong’s world order.

In our first experiment we asked people to do basically the same task as in Mark’s previous study. Participants were shown a brief display containing two objects in the periphery, and we asked them to decide each time whether they were the same or different. On each ‘trial’ of this task, we gave a brief burst of TMS to the participant’s visual cortex, targeting the area that represents the fovea. The TMS could be applied at one of several different times relative to when the objects appeared, from about a seventh of a second beforehand (150 ms) to half a second afterward (500 ms).

Our first TMS experiment on foveal feedback. The upper panel (a) shows the sequence of stimuli presented to participants on a computer screen, and the possible times that TMS could occur (in milliseconds, 'ms'). The lower panel (b) shows the brain regions we targeted with TMS, in one of the eighteen participants. The ‘p-calcarine’ region was our estimate of the foveal cortex, at an anatomical area called the calcarine sulcus. This sulcus is shown by the black line. The ‘non-calcarine’ region was a control condition. See our paper for full details.

The results of this first experiment were interesting. When we disrupted the visual cortex early after the peripheral objects appeared, people’s ability to compare them was unaffected. But if we gave the TMS later, specifically 350-400 milliseconds (ms) after the stimuli appeared, then TMS reduced performance accuracy by about 50%. 

A timecourse of 350-400ms might sound pretty fast, but in the brain this is the equivalent of a lazy Sunday. All feedforward visual processing has happened by about 100ms, so we thought this pointed clearly to the activity of feedback mechanism.

We submitted our paper for publication but the reviewers were not entirely convinced. Their main concern was whether we had actually stimulated the foveal part of the visual cortex. We assumed that we had because we positioned the TMS based on each participant’s brain anatomy, as seen in their MRI scan. But actually, the reviewers were right – we hadn’t provided any definite confirmation that we had disrupted the foveal representation.

So we went back to the lab and ran a second experiment. This time our participants switched between comparing stimuli in the periphery or on the fovea. When the stimuli were on the fovea, early TMS impaired perception. This was reassuring because it confirmed that we were hitting the right spot, and it also replicated the well-established finding that TMS of the visual cortex impairs feedforward processing of foveal stimuli.

Crucially, our second experiment also replicated the same impairment of peripheral vision when TMS was given 350-400ms after the stimuli appeared. Considered in light of Mark’s previous fMRI study, we thought the two experiments now provided a more convincing case that feedback to the foveal visual cortex is important for peripheral vision. Our reviewers agreed and the paper was accepted for publication.

Even though our study was published, it is important to stress that no single study like ours is completely definitive. TMS is a powerful and useful technique, but it has limitations just as fMRI does. In this case, we still don’t know if the effects we saw were due only to disruption of the foveal visual cortex, or whether the brain activity caused by TMS could have spread to parts of the visual cortex that represent peripheral space. We think this is unlikely to have produced our results because any such spread would happen very quickly and should have caused a much earlier impairment of peripheral vision than we found. But that is ultimately a speculative argument, and the fact remains that we don’t know for sure. 

In that sense our study is representative of much cognitive neuroscience, building a converging body of evidence through a range of imperfect techniques. We now believe it is important for labs to independently replicate our findings – and I don’t mean “conceptually”. 

More questions than answers?

I think the notion that feedback can construct new representations is fascinating because it highlights the active and sophisticated nature of human vision. But so far, the two published studies on this phenomenon seem to raise more questions than they answer.

Here are four questions that leap to mind.

First, how do the foveal representations interact with incoming feedforward inputs? Can they all exist simultaneously in the same area of cortex without interfering with each other? How does the brain segregate them?

Second, if the foveal visual cortex is being used to rebuild representations of peripheral stimuli, why don’t people see images of the stimuli on their fovea? A clue may lie in work on mental imagery. Mark Stokes from Oxford University has shown that higher areas of visual cortex represent stimuli that are imagined, and that the patterns of activity between imagined and perceived stimuli show striking similarities. Yet when we imagine stimuli we obviously don’t see them as real objects. If we did then we would spend our lives hallucinating and confusing imagination with reality. Perhaps feedback of information from the periphery to the fovea works in a similar way to mental imagery.

Third, does the information in foveal cortex consist of a difference between peripheral representations, a sum, or a more complex calculation? Might constructive feedback ‘glue’ peripheral representations together in order to compare then? If so, this mechanism might lead to strange invisible illusions. For example – as shown in the figure below – it is possible that two peripheral objects could combine in foveal cortex to produce an illusory representation. The viewer would, in all likelihood, be completely unaware of such information, yet it might still be detected using fMRI or behavioural methods that probe unconscious processing.

Wild speculation. If foveal feedback takes information from the periphery and "moves" it to the foveal cortex, then it could bind stimuli together to produce representations that don't exist. This figure shows some examples of how this could arise. On the left, two triangles combine at the fovea to form the illusory representation of a star. In the middle, two halves of a letter combine to form the representation of "A". On the right, we see a scenario involving more complex objects. Is it possible that foveal feedback could inadvertently combine these stimuli to rebuild a representation that resembles a different category of object? People would not be aware of such representations, so the illusions would be invisible, but they might be detected using a priming task. Such ideas are purely speculative at this stage, but they raise interesting questions for future studies. 
 

Finally, we know that the visual system fills in gaps in our perception, which is why people are generally unaware of their own retinal blind spots. So another question is whether constructive feedback adds any new information to the mix, or instead whether it just fills in the gaps using a best guess.

I have no idea what the answers to these questions are, but we are planning more studies to find out.

____

* All comments and questions are welcome.
* Thanks to Mark Williams for comments on a previous draft of this post.

Research Briefings

At the recent Royal Institution debate on science journalism, I argued that scientists should be using their blogs to summarise key research findings for journalists and the public. You can watch me make this case here, running from 8.00min – 16.00min. I would also recommend watching the whole video to see the excellent talks by Ananyo Bhattacharya, Ed Yong, and Fiona Fox.

The aim of such blog articles shouldn’t be to court publicity or to merely regurgitate our peer-reviewed publications. Instead we should try to provide an overview that is tailored specifically for non-scientists, using minimal jargon and assuming no background knowledge. In keeping with Tim Radford's 7th commandment, we should also avoid insulting the reader's intelligence.

I’m going to call these blog articles “Research Briefings”. I plan to write a new research briefing for each of our peer-reviewed publications that has potential interest to those outside science. If the work is of sufficient public interest to warrant a general press release, I will also link to the blog article within the press release.

This month I'll be writing two research briefings for our forthcoming papers. In the first one I'll discuss a study in which we used transcranial magnetic stimulation to test whether 'feedback' in the brain rebuilds visual perception. In the second article, I'll talk about a series of experiments in which we found that training people to inhibit simple movements of the hands can reduce risky gambling behaviour.

Most of the work we produce is published behind paywalls but I will always make the PDF freely available within my articles. All of my published papers can also be freely downloaded here.

A final cautionary note about these articles. At the Ri debate, Nicola Davis of The Times’ Eureka magazine took the microphone to express concerns about scientists blogging. In her words:

“But, actually, with my journalist's hat on, my concern with that is that you have no idea about the credibility of what’s in that blog, and you have the problem as well that who shouts the loudest also gets more blog coverage. 

So you might have a story that has really very little scientific credibility getting a huge amount of blog posts being written, and written, and written about it. So I think blogging is great in one way…but I think it’s really a dangerous territory as well because there’s no kitemark on a blog.”

I think blogging is important, but I also think Nicola makes a good point. Therefore I want to stress the following caution for readers: my research briefings are not peer-reviewed and should be regarded as a supplement to the original peer-reviewed paper, and never as a substitute for it. So, caveat lector, and all that!