Newswise — Losing the ability to speak due to neurological damage can be incredibly isolating. But thanks to recent advancements in technology, there's hope on the horizon. Scientists have been working on neural speech prostheses, special devices that can help people who have trouble speaking by translating brain activity into speech.
In a recent study published in Nature Machine Intelligence, a team of NYU researchers led by Yao Wang — Professor of Electrical and Computer Engineering and Biomedical Engineering at NYU Tandon, as well as a member of NYU WIRELESS — and Adeen Flinker — Associate Professor of Biomedical Engineering at NYU Tandon and Neurology at NYU Grossman School of Medicine — and Tandon ECE Ph.D. student Xupeng Chen presented a significant advancement in the decoding of speech using neural architectures — turning signals recorded from the brain and transforming them into audible speech. Building upon previous research, their work introduces modifications that enhance decoding accuracy across a broader range of voices.
One key innovation lies in the adaptation of neural architectures to accommodate diverse speech patterns. Recent strides in machine learning and Brain-Computer Interface (BCI) systems have propelled the development of neural speech prostheses, offering hope to those affected by speech impairments.
One effective method for gathering data to develop such prostheses involves Electrocorticographic (ECoG) recordings obtained from epilepsy surgery patients. Implanted electrodes provide a rare opportunity to collect cortical data during speech with high precision, leading to promising results in speech decoding. Previously validated on five patients, their updated approach now has been validated over 48 individuals — an order of magnitude larger than in other similar work, ensuring a more robust and generalized decoding process.
Two significant challenges persist in decoding speech from neural signals. Firstly, the limited duration of training data contrasts with the extensive data required for deep learning models. Secondly, speech production variability, encompassing rate, intonation, and pitch variations, complicates model representation.
The NYU team’s approach uses a unique speech synthesizer developed in their previous research. This synthesizer translates a series of interpretable speech “parameters” including pitch, frequency, loudness, etc., into natural sounding speech. The developed system leverages neural network architectures to decode neural signals into speech parameters which the synthesizer uses to produce the intended speech.
The team developed an efficient neural network training pipeline that works effectively with limited training data, and compared the efficacy of different neural network architectures. The system can produce speech that is much closer to the actual voice of the study participants – a unique feature of this approach.
Perhaps most intriguing is the discovery regarding the right hemisphere's contribution to speech decoding. Traditionally the right hemisphere is overshadowed by the left hemisphere, which is predominantly associated with language functions. However, some of the participants only had electrodes implanted on their right hemisphere, providing the researchers no information about the left-hemisphere’s activities. Crucially, they were still able to use the information from the right hemisphere to produce accurate speech decoding. Not only does this reveal how speech is processed and produced by the brain across the two hemispheres, it also opens new possibilities for therapeutic interventions, particularly in addressing speech disorders like aphasia, following damage to the left hemisphere.
In addition to its scientific findings, the study offers an open-source neuro-decoding pipeline, facilitating collaboration and replication of results within the research community. This initiative promotes transparency and accelerates progress in the field of neural decoding.
The research’s implications for understanding the complexities of speech processing and potential therapeutic avenues loom large, and marks a significant milestone in unraveling the mysteries of the human mind's linguistic capabilities, paving the way for future breakthroughs in neuroengineering and clinical interventions. The NYU team is continuing to investigate neural decoding approaches under a new NSF award and ongoing NIH grants
This work was supported by National Science Foundation under its Collaborative Research in Computational Neuroscience Program as well as the NIH National Institute of Neurological Disorders and Stroke.
About the New York University Tandon School of Engineering
The NYU Tandon School of Engineering is home to a community of renowned faculty, undergraduate and graduate students united in a mission to understand and create technology that powers cities, enables worldwide communication, fights climate change, and builds healthier, safer, and more equitable real and digital worlds. The school’s culture centers on encouraging rigorous, interdisciplinary collaboration and research; fostering inclusivity, entrepreneurial thinking, and diverse perspectives; and creating innovative and accessible pathways for lifelong learning in STEM. NYU Tandon dates back to 1854, the founding year of both the New York University School of Civil Engineering and Architecture and the Brooklyn Collegiate and Polytechnic Institute. Located in the heart of Brooklyn, NYU Tandon is a vital part of New York University and its unparalleled global network. For more information, visit engineering.nyu.edu.
RSS