Speech recognition (automatic speech recognition (ASR), computer speech recognition, or speech-to-text (STT)) is a sub-field of computational linguistics concerned with methods and technologies that translate spoken language into text or other interpretable forms.
Speech recognition applications include voice user interfaces,
where the user speaks to a device, which "listens" and processes the
audio. Common voice applications include interpreting commands for
calling, call routing, home automation, and aircraft control. These
applications are called direct voice input. Productivity applications include searching audio recordings, creating transcripts, and dictation.
Speech recognition can be used to analyse speaker characteristics, such as identifying native language using pronunciation assessment.
Voice recognition (speaker identification) refers to identifying the speaker, rather than speech contents. Recognizing the speaker can simplify the task of translating speech in systems trained on a specific person's voice. It can also be used to authenticate the speaker as part of a security process.
History
Applications for speech recognition developed over many decades, with progress accelerated due to advances in deep learning and the use of big data. These advances are reflected in an increase in academic papers, and greater system adoption.
Key areas of growth include vocabulary size, more accurate
recognition for unfamiliar speakers (speaker independence), and faster
processing speed.
Pre-1970
Raj Reddy was the first person to work on continuous speech recognition, as a graduate student at Stanford University in the late 1960s. Previous systems required users to pause after each word. Reddy's system issued spoken commands for playing chess.
Around this time, Soviet researchers invented the dynamic time warping (DTW) algorithm and used it to create a recognizer capable of operating on a 200-word vocabulary. DTW processed speech by dividing it into short frames (e.g. 10 ms
segments) and treating each frame as a unit. Speaker independence,
however, remained unsolved.
1970–1990
- 1971 – DARPA
funded a five-year speech recognition research project, Speech
Understanding Research, seeking a minimum vocabulary size of 1,000
words. The project considered speech understanding a key to achieving progress in speech recognition, which was later disproved. BBN, IBM, Carnegie Mellon (CMU), and Stanford Research Institute participated.
- 1972 – The IEEE Acoustics, Speech, and Signal Processing group held a conference in Newton, Massachusetts.
- 1976 – The first ICASSP was held in Philadelphia, which became a major venue for publishing on speech recognition.
During the late 1960s, Leonard Baum developed the mathematics of Markov chains at the Institute for Defense Analysis. A decade later, at CMU, Raj Reddy's students James Baker and Janet M. Baker began using the hidden Markov model (HMM) for speech recognition. James Baker had learned about HMMs while at the Institute for Defense Analysis. HMMs enabled researchers to combine sources of knowledge, such as acoustics, language, and syntax, in a unified probabilistic model.
By the mid-1980s, Fred Jelinek's team at IBM created a voice-activated typewriter called Tangora, which could handle a 20,000-word vocabulary. Jelinek's statistical approach placed less emphasis on emulating human
brain processes in favor of statistical modelling. (Jelinek's group
independently discovered the application of HMMs to speech.) This was controversial among linguists since HMMs are too simplistic to account for many features of human languages. However, the HMM proved to be a highly useful way for modelling speech
and replaced dynamic time warping as the dominant speech recognition
algorithm in the 1980s.
- 1982 – Dragon Systems, founded by James and Janet M. Baker, was one of IBM's few competitors.
Practical speech recognition
The 1980s also saw the introduction of the n-gram language model.
- 1987 – The back-off model enabled language models to use multiple-length n-grams, and CSELT used HMM to recognize languages (in software and hardware, e.g. RIPAC).
At the end of the DARPA program in 1976, the best computer available to researchers was the PDP-10 with 4 MB of RAM. It could take up to 100 minutes to decode 30 seconds of speech.
Practical products included:
- 1984 – the Apricot Portable was released with up to 4096 words support, of which only 64 could be held in RAM at a time.
- 1987 – a recognizer from Kurzweil Applied Intelligence
- 1990 – Dragon Dictate, a consumer product released in 1990. AT&T deployed the Voice Recognition Call Processing service in 1992 to route telephone calls without a human operator. The technology was developed by Lawrence Rabiner and others at Bell Labs.
By the early 1990s, the vocabulary of the typical commercial speech
recognition system had exceeded the average human vocabulary. Reddy's former student, Xuedong Huang, developed the Sphinx-II
system at CMU. Sphinx-II was the first to do speaker-independent, large
vocabulary, continuous speech recognition, and it won DARPA's 1992
evaluation. Handling continuous speech with a large vocabulary was a
major milestone. Huang later founded the speech recognition group at Microsoft in 1993. Reddy's student Kai-Fu Lee joined Apple, where, in 1992, he helped develop the Casper speech interface prototype.
Lernout & Hauspie,
a Belgium-based speech recognition company, acquired other companies,
including Kurzweil Applied Intelligence in 1997 and Dragon Systems in
2000. L&H was used in Windows XP.
L&H was an industry leader until an accounting scandal destroyed it
in 2001. L&H speech technology was bought by ScanSoft, which became
Nuance in 2005. Apple licensed Nuance software for its digital assistant Siri.
2000s
In the 2000s, DARPA sponsored two speech recognition programs:
Effective Affordable Reusable Speech-to-Text (EARS) in 2002, followed by
Global Autonomous Language Exploitation (GALE) in 2005. Four teams participated in EARS: IBM; a team led by BBN with LIMSI and the University of Pittsburgh; Cambridge University; and a team composed of ICSI, SRI, and the University of Washington. EARS funded the collection of the Switchboard telephone speech corpus, which contained 260 hours of recorded conversations from over 500 speakers. The GALE program focused on Arabic and Mandarin broadcast news. Google's first effort at speech recognition came in 2007 after recruiting Nuance researchers. Its first product, GOOG-411, was a telephone-based directory service.
Since at least 2006, the U.S. National Security Agency has employed keyword spotting, allowing analysts to index large volumes of recorded conversations and identify speech containing "interesting" keywords. Other government research programs focused on intelligence applications, such as DARPA's EARS program and IARPA's Babel program.
In the early 2000s, speech recognition was dominated by hidden Markov models combined with feed-forward artificial neural networks (ANN). Later, speech recognition was taken over by long short-term memory (LSTM), a recurrent neural network (RNN) published by Sepp Hochreiter & Jürgen Schmidhuber in 1997. LSTM RNNs avoid the vanishing gradient problem and can learn "Very Deep Learning" tasks that require memories of events that happened thousands of discrete time steps earlier, which is important for speech.
Around 2007, LSTMs trained with Connectionist Temporal Classification (CTC) began to outperform. In 2015, Google reported a 49 percent error‑rate reduction in its speech recognition via CTC‑trained LSTM. Transformers, a type of neural network based solely on attention, were adopted in computer vision and language modelling and then to speech recognition.
Deep feed-forward (non-recurrent) networks for acoustic modelling were introduced in 2009 by Geoffrey Hinton and his students at the University of Toronto, and by Li Deng and colleagues at Microsoft Research. In contrast to the prioer incremental improvements, deep learning decreased error rates by 30%.
Both shallow and deep forms (e.g., recurrent nets) of ANNs had been explored since the 1980s. However, these methods never defeated non-uniform internal-handcrafting Gaussian mixture model/hidden Markov model (GMM-HMM) technology. Difficulties analyzed in the 1990s, included gradient diminishing and weak temporal correlation structure. All these difficulties combined with insufficient training data and
computing power. Most speech recognition pursued generative modelling
approaches until deep learning won the day. Hinton et al. and Deng et
al.
2010s
By early the 2010s, speech recognition was differentiated from speaker recognition, and speaker independence
was considered a major breakthrough. Until then, systems required a
"training" period for each voice.
In 2017, Microsoft researchers reached the human parity milestone
of transcribing conversational speech on the widely benchmarked
Switchboard task. Multiple deep learning models were used to improve
accuracy. The error rate was reported to be as low as 4 professional
human transcribers working together on the same benchmark.
Models, methods, and algorithms
Both acoustic modeling and language modeling
are important parts of statistically-based speech recognition
algorithms. Hidden Markov models (HMMs) are widely used in many systems.
Language modelling is also used in many other natural language
processing applications, such as document classification or statistical machine translation.
Hidden Markov models
Speech recognition systems are based on HMMs. These are statistical
models that output a sequence of symbols or quantities. HMMs are used in
speech recognition because a speech signal can be viewed as a piecewise
stationary signal or a short-time stationary signal. In a short time
scale (e.g. 10 milliseconds), speech can be approximated as a stationary process. Speech can be thought of as a Markov model for many stochastic purposes.
HMMs are popular because they can be trained automatically and
are simple and computationally feasible. An HMM outputs a sequence of n-dimensional real-valued vectors (where n is an integer such as 10), outputting one every 10 milliseconds. The vectors consist of cepstral coefficients, obtained by a Fourier transform of a short window of speech and decorrelating the spectrum using a cosine transform,
then taking the first (most significant) coefficients. The HMM tends to
have, in each state, a statistical distribution that is a mixture of
diagonal covariance Gaussians, which give a likelihood for each observed vector. Each word, or (for more general speech recognition systems), each phoneme,
has a different output distribution; an HMM for a sequence of words or
phonemes is made by concatenating the individual trained HMMs for the
separate words and phonemes.
Speech recognition systems use combinations of standard
techniques to improve results. A typical large-vocabulary system applies
context dependency for the phonemes (so that phonemes with different
left and right context have different realizations as HMM states). It
uses cepstral normalization to handle speaker and recording conditions. It might use vocal tract length normalization (VTLN) for male-female normalization and maximum likelihood linear regression
(MLLR) for more general adaptation. The features use delta and
delta-delta coefficients to capture speech dynamics, and in addition
might use heteroscedastic linear discriminant analysis (HLDA); or might use splicing and LDA-based projection, followed by HLDA or a global semi-tied covariance transform (also known as maximum likelihood linear transform
(MLLT)). Many systems use discriminative training techniques that
dispense with a purely statistical approach to HMM parameter estimation
and instead optimize some classification-related measure of the training
data. Examples are maximum mutual information (MMI), minimum classification error (MCE), and minimum phone error (MPE).
Dynamic time warping (DTW)-based speech recognition
Dynamic time warping was historically used for speech recognition, but was later displaced by HMM.
Dynamic time warping measures similarity between two sequences
that may vary in time or speed. For instance, similarities in walking
patterns could be detected, even if in one video a person was walking
slowly and in another was walking more quickly, or even if accelerations
and decelerations came during one observation. DTW has been applied to
video, audio, and graphics – any data that can be turned into a linear
representation can be analyzed with DTW.
This could handle speech at different speaking speeds. In
general, it allows an optimal match between two sequences (e.g., time
series) with certain restrictions. The sequences are "warped"
non-linearly to match each other. This sequence alignment method is
often used in the context of HMMs.
Neural networks
Neural networks
became interesting in the late 1980s before beginning to dominate in
the 2010s. Neural networks have been used in many aspects of speech
recognition, such as phoneme classification, phoneme classification through multi-objective evolutionary algorithms, isolated word recognition, audiovisual speech recognition, audiovisual speaker recognition, and speaker adaptation.
Neural networks make fewer explicit assumptions about feature
statistical properties than HMMs. When used to estimate the
probabilities of a speech segment, neural networks allow natural and
efficient discriminative training. However, in spite of their
effectiveness in classifying short-time units such as individual
phonemes and isolated words, early neural networks were rarely successful for continuous recognition
because of their limited ability to model temporal dependencies.
One approach was to use neural networks for feature transformation, or dimensionality reduction. However, more recently, LSTM and related recurrent neural networks (RNNs), Time Delay Neural Networks (TDNN's), and transformers demonstrated improved performance.
Deep feedforward and recurrent neural networks
Researchers are exploring deep neural networks (DNNs) and denoising autoencoders .A DNN is a type of artificial neural network that includes multiple hidden layers between the input and output. Like simpler neural networks, DNNs can model complex, non-linear
relationships. However, their deeper architecture allows them to build
more sophisticated representations that combine features from earlier
layers. This gives them a powerful ability to learn and recognize
complex patterns in speech data.
A major breakthrough in using DNNs for large vocabulary speech
recognition came in 2010. In a collaboration between industry and
academia, researchers used DNNs with large output layers based on
context-dependent HMM states that were created using decision trees. This approach significantly improved performanc.
A core idea behind deep learning is to eliminate the need for
manually designed features and instead learn directly from input data.
This was first demonstrated using deep autoencoders trained on raw
spectrograms or linear filter-bank features. These models outperformed traditional Mel-Cepstral features, which rely
on fixed transformations. More recently, researchers showed that
waveforms can achieve excellent results in large-scale speech
recognition.
End-to-end learning
Since 2014, much research has considered "end-to-end" ASR. Traditional phonetic-based (i.e., all HMM-based model) approaches required separate components and training for pronunciation, acoustic, and language.
End-to-end models learn from all the components at once. This
simplifies the training and deployment processes. For example, an n-gram language model
is required for all HMM-based systems, and a typical 2025-era n-gram
language model often takes gigabytes in memory, making them impractical
to deploy on mobile devices. Consequently, ASR systems from Google and Apple (as of 2017) deploy on servers and required a network connection to operate.
The first attempt at end-to-end ASR was the Connectionist Temporal Classification (CTC)-based system introduced by Alex Graves of Google DeepMind and Navdeep Jaitly of the University of Toronto in 2014. The model consisted of RNNs and a CTC layer. Jointly, the RNN-CTC model
learns the pronunciation and acoustic model together, however, it is
incapable of learning the language model due to conditional independence
assumptions, similar to an HMM. Consequently, CTC models can directly
learn to map speech acoustics to English characters, but the models make
many common spelling mistakes and must rely on a separate language
model to finalize transcripts. Later, Baidu's Deep Speech 2
(2015) expanded on this approach by replacing hand-engineered pipeline
components with a single end-to-end deep neural network trained on over
11,000 hours of English and 9,400 hours of Mandarin speech. The system
matched or exceeded human-level transcription accuracy on several
benchmarks and demonstrated that a single architecture could generalize
across two linguistically distinct languages
In 2016, the University of Oxford presented LipNet, the first end-to-end sentence-level lipreading model, using
spatiotemporal convolutions coupled with an RNN-CTC architecture,
surpassing human-level performance in a restricted dataset. A large-scale convolutional-RNN-CTC architecture was presented in 2018 by Google DeepMind, achieving 6 times better performance than human experts. In 2019, Nvidia launched two CNN-CTC ASR models, Jasper and QuarzNet, with an overall performance word error rate (WER) of 3%. Similar to other deep learning applications, transfer learning and domain adaptation
are important strategies for reusing and extending the capabilities of
deep learning models, particularly due to the small size of available
corpora in many languages and/or specific domains.
In 2018, researchers at MIT Media Lab announced preliminary work on AlterEgo, a device that uses electrodes to read the neuromuscular signals users make as they subvocalize. They trained a convolutional neural network to translate the electrode signals into words.
Attention-based models
Attention-based ASR models were introduced by Chan et al. of Carnegie Mellon University and Google Brain, and Bahdanau et al. of the University of Montreal in 2016. The model named "Listen, Attend and Spell" (LAS), literally "listens"
to the acoustic signal, pays "attention" to all parts of the signal and
"spells" out the transcript one character at a time. Unlike CTC-based
models, attention-based models require conditional-independence
assumptions and can learn all the components of a speech recognizer
directly. This means that during deployment, no a priori language model is required, making it less demanding for applications with limited memory.
Attention-based models immediately outperformed CTC models (with or without an external language model) and continued improving. Latent Sequence Decomposition (LSD) was proposed by Carnegie Mellon
University, MIT, and Google Brain to directly emit sub-word units that
are more natural than English characters. The University of Oxford
and Google DeepMind extended LAS to "Watch, Listen, Attend and Spell"
(WLAS) to handle lip reading and surpassed human-level performance.
Applications
In-car systems
Voice commands may be used to initiate phone calls, select radio
stations, or play music. Voice recognition capabilities vary across car
make and model. Some models offer natural-language speech recognition,
allowing the driver to use full sentences and common phrases in a
conversational style. With such systems, fixed commands are not
required.
Education
Automatic pronunciation assessment is the use of speech recognition to verify the correctness of speech, as distinguished from assessment by a person. Also called speech verification, pronunciation evaluation, and
pronunciation scoring, the main application of this technology is
computer-aided pronunciation teaching (CAPT) when combined with computer-aided instruction for computer-assisted language learning (CALL), speech remediation, or accent reduction. Pronunciation assessment does not determine unknown speech (as in dictation or automatic transcription) but instead, compares speech to a reference model for the words spoken, sometimes with inconsequential prosody such as intonation, pitch, tempo, rhythm, and stress. Pronunciation assessment is also used in reading tutoring, for example in products such as Microsoft Teams and Amira Learning. Pronunciation assessment can also be used to help diagnose and treat speech disorders such as apraxia.
Assessing intelligibility is essential for avoiding inaccuracies from accent bias, especially in high-stakes assessments, from words with multiple correct pronunciations, and from phoneme coding errors in digital pronunciation dictionaries. In 2022, researchers found that some newer speech to text systems, based on end-to-end reinforcement learning
to map audio signals directly into words, produce word and phrase
confidence scores closely correlated with listener intelligibility. In the Common European Framework of Reference for Languages
(CEFR) assessment criteria for "overall phonological control",
intelligibility outweighs formally correct pronunciation at all levels.
Health care
Medical documentation
In the health care sector, speech recognition can be implemented in
front-end or back-end medical documentation processes. In front-end
speech recognition, the provider dictates into a speech-recognition
engine, words are displayed as they are recognized, and the speaker is
responsible for editing and signing off on the document. In back-end or
deferred speech recognition the provider speaks into a digital dictation
system, the voice is routed through a speech-recognition machine, and a
draft document is routed along with the voice file to an editor, who
edits/finalizes the draft and final report.
A major issue is that the American Recovery and Reinvestment Act of 2009 (ARRA) provides substantial financial benefits to physicians who utilize an Electronic Health Record
(EHR) that complies with "Meaningful Use" standards. These standards
require that substantial data be maintained by the EHR. The use of
speech recognition is more naturally suited to the generation of
narrative text, as part of a radiology/pathology interpretation,
progress note or discharge summary; the ergonomic gains of using speech
recognition to enter structured discrete data (e.g., numeric values or
codes from a list or a controlled vocabulary) are relatively minimal for people who are sighted and who can operate a keyboard and mouse.
A more significant issue is that most EHRs have not been
expressly tailored to take advantage of voice-recognition capabilities. A
large part of a clinician's interaction with EHR involves navigation
through the user interface that is heavily dependent on keyboard and
mouse; voice-based navigation provides only modest ergonomic benefits.
By contrast, many highly customized systems for radiology or pathology
dictation implement voice "macros", where the use of certain phrases –
e.g., "normal report", will automatically fill in a large number of
default values and/or generate boilerplate, which vary with the type of
exam – e.g., a chest X-ray vs. a gastrointestinal contrast series for a
radiology system.
Therapeutic use
Prolonged use of speech recognition software in conjunction with word processors has shown benefits to short-term-memory restrengthening in brain AVM patients who have been treated with resection.
Further research needs to be conducted to determine cognitive benefits
for individuals whose AVMs have been treated using radiologic
techniques.
Military
Aircraft
Substantial efforts have been devoted to the test and evaluation of speech recognition in fighter aircraft. Of particular note have been the US programme in speech recognition for the Advanced Fighter Technology Integration (AFTI)/F-16 aircraft (F-16 VISTA), the programme in France for Mirage
aircraft, and UK programmes dealing with a variety of aircraft
platforms. In these programmes, speech recognizers have been operated
successfully, with applications including setting radio frequencies,
commanding an autopilot system, setting steer-point coordinates and
weapons release parameters, and controlling flight display.
Working with Swedish pilots flying the JAS-39 Gripen, Englund (2004) reported that recognition deteriorated with increasing g-loads.
The study concluded that adaptation greatly improved the results in all
cases and that the introduction of models for breathing was shown to
improve recognition scores significantly. Contrary to what might have
been expected, no effects of the broken English of the speakers were
found. Spontaneous speech caused problems for the recognizer, as might
have been expected. A restricted vocabulary, and above all, a proper
syntax, could thus be expected to improve recognition accuracy
substantially.
The Eurofighter Typhoon
employs a speaker-dependent system, requiring each pilot to create a
template. The system is not used for safety-critical or weapon-critical
tasks, such as weapon release or lowering of the undercarriage, but is
used for many cockpit functions. Voice commands are confirmed by visual
and/or aural feedback. The system is seen as a major benefit in the
reduction of pilot workload, and allows the pilot to assign targets with two voice commands or to a wingman with only five commands.
Speaker-independent systems are under test for the F-35 Lightning II (JSF) and the Alenia Aermacchi M-346 Master lead-in fighter trainer. These systems have produced word accuracy scores in excess of 98%.
Helicopters
The problems of achieving high recognition accuracy under stress and noise are particularly relevant in the helicopter
environment as well as in the fighter environment. The acoustic noise
problem is actually more severe in the helicopter environment, because
of the high noise levels, and because helicopter pilots, in general, do
not wear a facemask, which would reduce acoustic noise in the microphone. Substantial test and evaluation programmes, notably by the U.S. Army Avionics Research and Development Activity (AVRADA) and by the Royal Aerospace Establishment (RAE) in the UK. Work in France included speech recognition in the Puma helicopter. Voice applications include control of communication radios, navigation systems, and an automated target handover system.
The overriding issue for voice is the impact on pilot
effectiveness. Encouraging results are reported for the AVRADA tests,
although these represent only a feasibility demonstration in a test
environment. Much remains to be done both in speech recognition and in
overall speech technology in order to consistently achieve performance improvements in operational settings.
Air traffic control
Training for air traffic controllers
(ATC) represents an excellent application for speech recognition
systems. Many ATC training systems currently require a trainer to act as
a "pseudo-pilot", engaging in a voice dialog with the trainee
controller, which simulates the dialog that the controller would have
with real pilots. Speech recognition and synthesis
techniques offer the potential to eliminate the need for a person to
act as a pseudo-pilot, thus reducing training and support personnel.
In theory, air controller tasks are characterized by highly
structured speech as the primary output, reducing the difficulty of the
speech recognition task. In practice, this is rarely the case. FAA
document 7110.65 details the phrases that should be used by air traffic
controllers. While this document gives less than 150 examples of such
phrases, the number of phrases supported by one of the simulation
vendors speech recognition systems is in excess of 500,000.
The USAF, USMC, US Army, US Navy, and FAA as well as
international ATC training organizations such as the Royal Australian
Air Force and Civil Aviation Authorities in Italy, Brazil, and Canada
use ATC simulators with speech recognition.
People with disabilities
Speech recognition programs can provide many benefit to those with disabilities. For individuals who are deaf or hard of hearing, speech recognition software can be used to generate captions of conversations. Additionally, individuals who are blind (see blindness and education)
or have poor vision can benefit from listening to textual content, as
well as garner more functionality from a computer by issuing commands
with their voice.
The use of voice recognition software, in conjunction with a
digital audio recorder and a personal computer running word-processing
software, has proven useful for restoring damaged short-term memory
capacity in individuals who have suffered a stroke or have undergone a craniotomy.
Speech recognition has proven very useful for those who have
difficulty using their hands due to causes ranging from mild repetitive
stress injuries to disabilities that preclude the use of conventional
computer input devices. Individuals with physical disabilities can use
voice commands and transcription to navigate electronics hands-free. In fact, people who developed RSI from keyboard use became an early and urgent market for speech recognition. Speech recognition is used in deaf telephony, such as voicemail to text, relay services, and captioned telephone.
Individuals with learning disabilities who struggle with
thought-to-paper communication may benefit from the software, but the
product's fallibility remains a significant consideration for many. In addition, speech to text technology is only an effective aid for
those with intellectual disabilities if the proper training and
resources are provided (e.g. in the classroom setting).
This type of technology can help those with dyslexia, but the
potential benefits regarding other disabilities are still in question.
Mistakes made by the software hinder its effectiveness, since misheard
words take more time to fix.
Other domains
ASR is now commonplace in the field of telephony. In telephony systems, ASR is predominantly used in contact centers by integrating it with IVR systems.
It is becoming more widespread in computer gaming and simulation.
Despite the high level of integration with word processing in
general personal computing, in the field of document production, ASR has
not seen the expected increases in use.
The improvement of mobile processor speeds has made speech recognition practical in smartphones. Speech is used mostly as a part of a user interface, for creating predefined or custom speech commands.
The performance of speech recognition systems is usually evaluated in terms of accuracy and speed. Accuracy is usually rated with word error rate (WER), whereas speed is measured in elapsed time. Other measures of accuracy include Single Word Error Rate (SWER) and Command Success Rate (CSR).
Speech recognition is complicated by many properties of speech.
Vocalizations vary in terms of accent, pronunciation, articulation,
roughness, dialect, nasality, pitch, volume, and speed. Speech is
distorted by background noise, echoes, and recording characteristics.
Accuracy of speech recognition may vary with the following:
- Vocabulary size and confusability
- Speaker dependence versus independence
- Isolated, discontinuous, or continuous speech
- Task and language constraints
- Read versus spontaneous speech
- Adverse conditions
Accuracy
The accuracy of speech recognition may vary depending on the following factors:
- Error rates increase as the vocabulary size grows:
- e.g. the 10 digits "zero" to "nine" can be recognized
essentially perfectly, but vocabulary sizes of 200, 5000, or 100000 may
have error rates of 3%, 7%, or 45% respectively.
- Vocabulary is hard to recognize if it contains confusing letters:
- e.g. the 26 letters of the English alphabet
are difficult to discriminate because they are confusing words (most
notoriously, the E-set: "B, C, D, E, G, P, T, V, Z (when "Z" is
pronounced "zee" rather than "zed", depending on region); an 8% error
rate is considered good for this vocabulary.
- Speaker dependence vs. independence:
- A speaker-dependent system is intended for use by a single speaker.
- A speaker-independent system is intended for use by any speaker (more difficult).
- Isolated, discontinuous or continuous speech
- With isolated speech, single words are used, which is easier to recognize.
- With discontinuous speech, full sentences separated by silence are
used. The silence is easier to recognize similar to isolated speech.
- With continuous speech naturally spoken sentences are used, which are harder to recognize.
- Task and language constraints can inform the recognition
- The requesting application may dismiss the hypothesis "The apple is red."
- Constraints may be semantic; rejecting "The apple is angry."
- Syntactic; rejecting "Red is apple the."
- Constraints are often represented by grammar.
- Read vs. spontaneous speech
- When a person reads it's usually in a context that has been previously prepared.
- When a person speaks spontaneously, recognition must deal with
disfluencies such as "uh" and "um", false starts, incomplete sentences,
stuttering, coughing, and laughter) and limited vocabulary.
- Adverse conditions
- environmental noise (e.g., in a car or factory).
- Acoustic distortions (e.g. echoes, room acoustics)
Speech recognition is a multi-level pattern recognition task.
- Acoustic signals are structured into a hierarchy of units, e.g. phonemes, words, phrases, and sentences;
- Each level provides additional constraints; e.g., known word
pronunciations or legal word sequences, which can compensate for errors
or uncertainties at a lower level;
This hierarchy of constraints improves accuracy. By combining
decisions probabilistically at all lower levels, and making ultimate
decisions only at the highest level, speech recognition is broken into
several phases. Computationally, it is a problem in which a sound
pattern has to be recognized or classified into a category that
represents a meaning to a human. Every acoustic signal can be broken
into smaller sub-signals. As the more complex sound signal is divided,
different levels are created, where at the top level are complex sounds
made of simpler sounds on the lower level, etc. At the lowest level,
simple and more probabilistic rules apply. These sounds are put together
into more complex sounds on upper level, a new set of more
deterministic rules predicts what the complex sound represents. The
upper level of a deterministic rule should figure out the meaning of
complex expressions. In order to expand our knowledge about speech
recognition, we need to take into consideration neural networks. Neural
network approaches use the following steps:
- Digitize the speech – for telephone speech, 8000 samples per second are captured;
- Compute features of spectral-domain of the speech (with Fourier
transform); computed every 10ms, with one 10ms section called a frame;
Sound is produced by air (or some other medium) vibration. Sound creates a wave that has two measures: amplitude (strength), and frequency (vibrations per second). Accuracy can be computed with the help of WER, which is calculated by
aligning the recognized word and referenced word using dynamic string
alignment. The problem may occur while computing the WER due to the
difference between the sequence lengths of the recognized word and
referenced word.
The formula to compute the word error rate (WER) is:
where s is the number of substitutions, d is the number of deletions, i is the number of insertions, and n is the number of word references.
While computing, the word recognition rate (WRR) is used. The formula is:
where h is the number of correctly recognized words:
Security
Speech recognition can become a means of attack, theft, or accidental
operation. For example, activation words like "Alexa" spoken in an
audio or video broadcast can cause devices in homes and offices to start
listening for input inappropriately, or possibly take an unwanted
action. Voice-controlled devices may be accessible to unauthorized users.
Attackers may be able to gain access to personal information, like
calendars, address book contents, private messages, and documents. They
may also be able to impersonate the user to send messages or make online
purchases.
Two attacks have been demonstrated that use artificial sounds. One transmits ultrasound and attempts to send commands without people noticing. The other adds small, inaudible distortions to other speech or music
that are specially crafted to confuse the specific speech recognition
system into recognizing music as speech, or to make what sounds like one
command to a human sound like a different command to the system.
