Automatic audiovisual integration in speech perception.

Maurizio Gentilucci; Luigi Cattaneo

doi:10.1007/s00221-005-0008-z

Home/Archive/PubMed

PubMed · 16034571

Automatic audiovisual integration in speech perception.

Maurizio Gentilucci·Luigi Cattaneo

View original source ↗Export metadata ↓

Abstract

Two experiments aimed to determine whether features of both the visual and acoustical inputs are always merged into the perceived representation of speech and whether this audiovisual integration is based on either cross-modal binding functions or on imitation. In a McGurk paradigm, observers were required to repeat aloud a string of phonemes uttered by an actor (acoustical presentation of phonemic string) whose mouth, in contrast, mimicked pronunciation of a different string (visual presentation). In a control experiment participants read the same printed strings of letters. This condition aimed to analyze the pattern of voice and the lip kinematics controlling for imitation. In the control experiment and in the congruent audiovisual presentation, i.e. when the articulation mouth gestures were congruent with the emission of the string of phones, the voice spectrum and the lip kinematics varied according to the pronounced strings of phonemes. In the McGurk paradigm the participants were unaware of the incongruence between visual and acoustical stimuli. The acoustical analysis of the participants' spoken responses showed three distinct patterns: the fusion of the two stimuli (the McGurk effect), repetition of the acoustically presented string of phonemes, and, less frequently, of the string of phonemes corresponding to the mouth gestures mimicked by the actor. However, the analysis of the latter two responses showed that the formant 2 of the participants' voice spectra always differed from the value recorded in the congruent audiovisual presentation. It approached the value of the formant 2 of the string of phonemes presented in the other modality, which was apparently ignored. The lip kinematics of the participants repeating the string of phonemes acoustically presented were influenced by the observation of the lip movements mimicked by the actor, but only when pronouncing a labial consonant. The data are discussed in favor of the hypothesis that features of both the visual and acoustical inputs always contribute to the representation of a string of phonemes and that cross-modal integration occurs by extracting mouth articulation features peculiar for the pronunciation of that string of phonemes.

Explore related subjects

Acoustic Stimulation Adult Female Humans Lipreading Male Photic Stimulation Psychoacoustics Sound Spectrography Speech Acoustics Speech Perception Speech Production Measurement Verbal Learning Visual Perception Vocabulary Comparative Study Journal Article Research Support, Non-U.S. Gov't

Keep this discovery

Explore connections, maps & timelines

BibTeX RIS

Maurizio Gentilucci, Luigi Cattaneo. 2005-10-29. Automatic audiovisual integration in speech perception.. https://doi.org/10.1007/s00221-005-0008-z

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related citations

PubMed2008

Sensory filtering phenomenology in PTSD.

Disrupted sensory filtering, or problems with suppressing irrelevant environmental sensory stimuli, has been reported in individuals with posttraumatic stress disorder (PTSD). However, the relationship of sensory filtering deficits to specific PTSD symptoms versus an association with general trauma exposure is unclear. These relationships were examined by administering self-report measures of trauma exposure, PTSD, and sensory gating phenomenology to undergraduate participants with PTSD (n=32), with trauma history but without PTSD (n=144), and with minimal trauma history (n=153). Subjects with PTSD reported greater filtering disruption than individuals in the trauma only and low trauma groups, who did not differ. Individuals endorsing reexperiencing and numbing symptoms, and females endorsing hypervigilance, reported disrupted sensory filtering phenomenology. These results suggest that impaired filtering differentiates between individuals with PTSD symptoms and asymptomatic individuals exposed to multiple traumas and low-trauma controls.

Acoustic Stimulation↗

PubMed2007

Hypoglycemia reduces the blood-oxygenation level dependent signal in primary auditory and visual cortex: a functional magnetic resonance imaging study.

Studies of the effects of hypoglycemia on the brain using neurocognitive testing have suggested that mainly complex functions subserved by secondary and tertiary cortex are affected by mild to moderate hypoglycemia and that intensively treated patients with Type I diabetes mellitus (T1DM) may have altered sensitivity to the central nervous system effects of hypoglycemia. Functional magnetic resonance imaging provides a sensitive, regionally-specific probe of possible neurophysiologic changes related to hypoglycemia in the brain. Eleven intensively-treated T1DM patients and 11 matched non-diabetic controls took part in a 2-day protocol in which functional magnetic resonance imaging (MRI) was used to measure changes in the patterns of brain activation produced by simple auditory and visual stimuli in different conditions. On one day, participants were euglycemic the entire time. On the other day, an initial 50-min euglycemic period was followed by a 50-min hypoglycemic period. Results indicated that hypoglycemia reduced the amplitude of the blood-oxygenation level dependent response in primary auditory and visual cortex to simple auditory and visual stimuli. The latency and duration of the transient hemodynamic response function were not affected. Responses to hypoglycemia were similar in diabetic and non-diabetic participants. These results suggest that mild to moderate hypoglycemia may alter the balance of blood flow and oxygen extraction when glucose levels are lowered. Intensively-treated T1DM, with its attendant frequent hypoglycemic episodes, did not seem to alter hypoglycemic responses in primary visual and auditory cortex.

Acoustic Stimulation↗

PubMed2007

Song discrimination learning in zebra finches induces highly divergent responses to novel songs.

Perceptual biases can shape the evolution of signal form. Understanding the origin and direction of such biases is therefore crucial for understanding signal evolution. Many animals learn about species-specific signals. Discrimination learning using simple stimuli varying in one dimension (e.g. amplitude, wavelength) can result in perceptual biases with preferences for specific novel stimuli, depending on the stimulus dimensions. We examine how this translates to discrimination learning involving complex communication signals; birdsongs. Zebra finches (Taeniopygia guttata) were trained to discriminate between two artificial songs, using a Go/No-Go procedure. The training songs in experiment 1 differed in the number of repeats of a particular element. The songs in experiment 2 differed in the position of an odd element in a series of repeated elements. We examined generalization patterns by presenting novel songs with more or fewer repeated elements (experiment 1), or with the odd element earlier or later in the repeated element sequence (experiment 2). Control birds were trained with only one song. The generalization curves obtained from (i) control birds, (ii) experimental birds in experiment 1, and (iii) experimental birds in experiment 2 showed large and systematic differences from each other. Birds in experiment 1, but not 2, responded more strongly to specific novel songs than to training songs, showing 'peak shift'. The outcome indicates that learning about communication signals may give rise to perceptual biases that may drive signal evolution.

Acoustic Stimulation↗