CA1204865A - Adaptive automatic discrete utterance recognition - Google Patents

Abstract

ADAPTIVE AUTOMATIC DISCRETE UTTERANCE RECOGNITION
Abstract Adaptive training of a system for discrete utterance recognition during continuous speech permits single prototype utterances to be adapted to the needs of the talker, during operation, without tedious multiple recitation for training of prototypes. Initial training of the recognition system is by a single utterance (or simulation) of a prototype vocabulary.
Operation proceeds, so long as utterances are recognized, until an unrecognized utterance is detected. The system then prompts a choice of prototype vocabulary keyword candidates, which the talker may then choose and utter.

Description

ADAPTIVE ~UTOl~TIC DISCRETE ~TTER~CE ~ECOG-~!ITIO~

BACKGP~O~ D OF THE I~,V~~TIO

Field OL- ~he Invention This invention relates to automatic discrete utterance voice recognition systems, and par-ticularly relates to an adaptive automatic discre-te utterance recognition system which requires only a prototype vocabulary set to be establ~shed by multiple repetition techniques and permits subsequent talkers to interact with the voice recognition system in adaptive mode in which mode the new talker is required merely to retrain the system r'or that limited subset of the vocabularv set in which the system cannot perform recognition.

Description of the Prior Art Extant automatic discrete utterance voice recognition systems involve separate procedures for utterance prototype establishment in which multiple repetitions of each vocabulary item are taken from each talker. If a single repetition is taken the probability exists that the entire vocabulary prototype set will require re-establ-sh-ment if an inappropriate prototype representation occurs.

Typical of such procedures -- each talker repeating the prototype word list -- are the following:

.

s U. S. Patent 3,333,248, Greenberg et al, SELF-ADAPTI~E SYSTEMS, July 25, 1967. Greenbery et al shows a self-adaptive pattern recognizer which, after initial training, may be switched to the operate mode and remains in operate mode until a character is presented which results in a reject signal. At this time the operator must assist by placing the identification switch in the position which corresponds to the rejected pattern, and the operator must determine whether the rejected pattern is a siightly modified one of the initial sample patterns or a new sample pattern and must set the appropriate switches includins the switch to begin the training mode. In order to effectively update the self-adaptive circuit, a suffic ent n~nber of the initial sample patterns must be stored and represented to the self-adaptive circuit along with the rejected pattern.

U. S. Patent 3,369,077, French et al, PITCH
MODIFICATION OF AUDIO WAVEFORL~lS, February 13, 1968.
French et al shows a speech synthesizing system wherein pitch periods are adjusted according to a predetermined time base. ~

U. S. Patent 3,440,617, Lesti et al, SIGNAL
RESPONSIVE SYSTEMS, April 22, 1969. Lesti et al shows a technique for recognition, independent of amplitude and duration of t~e signals to be recognized, by segmenting the signal into a series -of componerlt signals. ~he system extra?ola~es and interpolates inputs which i~ has never be ore received to the response ~,Jhich ~,ost closel~
resembles ~-he signal. Data rniyh. be lost when it becomes replaced by ne-~ data. Les'i et al shows a technique in which newl~ coded samples are no-t discarded when the transmit buffer portion is already full but rather the oldest untransmitted coded sample is discarded to make room for storage of the new sample.

U. S. Patent 3,665,450, Leban, METHOD AND
MEANS FOR ENCODING AND DECODING IDEOGR~PHIC
CHARACTERS, May 23, 1972. Leban shows a -technique for handling ideographic characters.

U. S. Patent 3,718,768, Abramson et al, ~OICE
OR ANALOG COMMUNICATION SYSTEM EMPLOYING
ADAPTIVE ENCODI~G TECHNIQUES, February 27, 1973. Abramson et al shows a technique for transmitting comm~lnications to remote stations which can detect their own identification signals and have their own sampling rates.

U. S. Patent ~,069,393, `lartin et al, r`10RD
RECOG`.~ITION ~PPARATUS Ai~.D .'~!ETHOD, January 17, 1978. Martin et al shows a -technique for time normali~iny training words and ~"ord~ ror recognition. Martin et al deals with s?oken input training words and generates a correlation function, and with feature e~-traction.
During the training ;node, the equipment is trained with new vocabulary words, preferably spoken by the person who is to later use the machine. It is desirable to use multiple samples of the same tra~ning word to obtain a faithful average sample.

U. S. Patent 4,092,493, Rabiner et al, SPEECH
RECOGNITION SYSTEM, ~!ay 30, 1978. Rabiner et al shows a speech recognition system in which test signals are time aligned to the average voiced interval of ~^e?etitions of each speech segment having a previouslv generated voiced interval linear prediction characteristic.

U. S. Paten-t, 4,297,528, Beno, TRAINING
CIRCUIT FOR AUDIO SIGNAL RECOGNITION COMPUTER, October 27, 1981. Beno shows a training circuit technique in which each training pattern, to be accepted for merging, must match the previously merged pat-terns by a threshold amount. The threshold is automaticall~ varied as the number of pre-viously merged training patterns increases.

C. C. Tappert, A PRELI.~INARY IiNVrSTIGATIO~ CF
ADAPTIVE CONTROL IN THE INTERACTION 3ETr.EEN
SEGI~ENTATIO~ AMD SEGME~T CLASSIFICATIO~
AUTO.~TIC RECOGNITION OF CO~TINUOUS S EEC~, IEEE Trans. on Systems, ,lan, and Cybernetics, Vol. SMC2, No. 1, 1/72, P?. 6672. Ta?pert shows feedback control or the interaction or segmentation and segment classi ication i-.
continuous speech recognition.

C. C. Tappert, et 31, APPLICATION OF SEQ~E~TIAL
DECODING FOR CO~VERTING PHONETIC TO G~PHIC
REPRESENTATION IN AUTO~lATIC RECOGNITION OF
CONTINUOUS SPEECH (ARCS), IErE Trans. on Audio and Electroacoustics, Vol. Au-21, :~.o.
3, 6/73, pp. 225228. Tappert et al shoJs conversion of machine-contaminated phone~ic descriptions o~ speaker perrormance into standard orthosraphy. Distinction is made between speaker-and-machine dependent corrup-tion of phonetic input strings.

SU;~RY o.- r'lE I`iVE`lTIC`~

The invention provides an optimu~ techniqtle for prototype establishment involving ~erel~ an ini~ial single prototype statement of e~ch vocabul~ry item (from the first talker or electronic equivalentj and thereafter requiring no recitation of vocabulary items by either the first or subsequent talkers --e~cept for those vocabulary items in which the system has difficulty performing correct recognition.
Retraining for such misrecognized vocabulary items is integrated with the recognition procedure.

The object of the invention is to provide inexpensive, fast, unobtrusive adaptation of a speech recognition system for the special requirements of each talker.

~nother object o-f the invention is that the talker should not be very a~are that the system is being retrained.

Another object of the invention is for the system to provide prompts to lead the talker through the limited subset of the vocabulary prototype set in as unobtrusive a manner as possible and within the talker's conte~t.

Another object of the invention is to provide system retraining only for that limited subset of the vocabulary prototype set in which both of the following occur:

1) the talker utters a word from the prototype set; and

2) the system has difriculty performing recognition of the Jord spoken.

This permits the system to avoic retraining fo~
those words ~hich are not used.

BRIE~ DESCRIPTIO~ OF THE D~A~JI~GS

FIG. 1 is a block diagram of the adaptive automatic discrete utterance recogni~iGn system OL the invention. OL the svstem sho~,lr., the portion of FIG. 1 most significar.t to the invention is the subsystem bo~ mar.;ed "Ad2p~ive Training."

FIG. 2 is a detailed diagram of the Aaa?tive Training subsystem.

DESCRIPTION OF THE PREFERR~D EMBODI`1ENT

~IG. 1 is a block diagram of the adaptive automatic discrete ut~erance recognition system of the invention. Note that prior art voice recognition systems generally require each new talker to provide a multiple recitation of the prototype vocabulary for-subsequen~
recognition. If for any reason a talker needed to change the prototype vocabulary (e.g., if the talker developed a respiratory problem) the original procedure required repetition in its ent,rety. This invention eliminates, even for the first talker, the need for multiple recitation of the prototype vocabulary, and eliminates the need for any recitation at all of the entire prototype vocabulary by any talker.
Only if the system should have difficulty perrorming recognition of any individual vocabulary item is the talker prompted to utter that individual vocabulary item.

The method of this invention requires the following operational modes:

Mode l Initializing Mode A selected prototy?ical talker reci-es one prototype vocabulary keyword ut.erance for each of a prototype set of vocabulary items, the system converting each prototype vocabulary keyword utterance to a code pattern for subsequent use in recognition procedures. Where appropriate, a prototype set may be calculated and entered in coded Eorm without any recitation by any talker at all.

Mode 2 Normal Operation Mode A talker (which talker may be the proto-typical talker or a subsequer.t talker) provides voice input for recognition by the system so long as recognition proceeds satisfactorily. (Under conditions not including consistent recognition errors no further vocabulary training occurs.) .~9~2--0~ !~
~2~4~5 ~lode 3 ~dap~ive Retraininq :~lode Upon conditions of consis~ent recognition error for a sample vocabulary item, the system prompts the talker as re~uired for adaptive retraining.

Details o, the method will be subsequentl~
discussed under the heading "INVENTIVE ~THOD."

DETAIL~D DESCRIPTION OF TH~ DRAWINGS

FIG. 1 shows the adaptive automatic discrete utterance recognition system of the invention.
Host compu~er 1 is connected to user interface 2, which in turn connects to display unit 3, microphone 4 and loudspeaker 5. User interface 2 provides the proper interface for the user and the recognition system, which may take a number of forms such as that shown. In the form shown in FIG. 1, user data passes via line 6 to signal analysis and feature abstraction unit 7 for distribution of original training patterns (prototypes) along path-way 8 to prototype storage 9 during training mode -~and during recognition mode, feature patterns pass along pathway 10 to pattern recognition unit 11.
Information of a control nature passes between pattern recognition unit 11 and prototype storage ;25 9 via pathway 12; prototype patterns for recognition are supplied to pattern recognition mechanism 11 ;along pathway 13. Recognition information, in the form of utterance identification distance values, is provided via pathway 14 to decision mechanism 15.
Recognition result data passes via pathway 16 bac~
to user interface 2 for action.
.

s --10-- ,s Declsion information also ?asses be ;een decialon mechanism 15 and ada?ti~7e ~ralning unl-t 20 ~ia pathways 17 and 18 and -ro~ ada~tive t~aining uni 16 to prototype storage 9 Jia ~ath-"av 19. rl5 prevlousl~ pointed out the port Gn or FIG. 1 .~ost significant ~o the invention ls the su~s~stem (identified by reference eharacter 20) mar~ed Adaptive Training. Nodes (l) (2) and (3) are shown in FIG. l for ease in relating FIG. 1 with FIG. 2.

FIG. 2 is a detailed diagram o~ the Adaptive Training subsystem 20 of FIG. l. Keyword uni.
21 determines whether or not the utterance is a keyword. If no a notification is forwarde via pathway 22 to node (2), which connects to pathway 18 in both FIGS. l and 2. I yes a noti-fieation is forwarded to incrementer 23 via YES
line 24 to prompt a user identifieation of tne keyword by a sequenee of ehoiee numbers. Incremen-ter 23 inerements the eurrent keyword ehoice numberand provides the next choiee number to mechanism 25 for determination whether the keyword choiee number is equal to the eurrent ehoiee. If the ehoiee number is not the appropriate choice number the ehoice prompts eontinue by a signal to maximum identifieation meehanism 26 (CHOICE NO=CH~AX?j for eontinuing through the list of-ehoices. I- the ehoiee number is less than maximum a signal on NO
pathway 28 eontrols a prompt for the next ehoiee number. When the last ehoiee number is reaehed CHOICE NO=CH~AX box 26 signals YES via pathway 29 to invoke REPEAT ENTRY meehanism 30 for a new list of ehoices.

s -11- ,.
Once the utterance choice number is ~ete~mine~, as identified by "YES" path-~ay 31 to PRO~PT C~OICE
box 32, the user is provided t~ith a ~ro~.pt requesting that the user ~ake the u-tterance of cnolce. This utterance is the utterance to be used in certain situa-tions as the new protot~pe.

The utterance is processed accordiny to the pro-cessing technique selected for the utterance recognition system, and coded as utterance code INPUT 1. The prompt requires a second utterance, which is processed and coded as utterance code INPUT 2. These utterance code inputs are pro~-ided, along with a similarly coded prototype utterar.ce, to three matrix comparators 33, 34 and 35, ~hich make distance comparisons ~or t~e three codes.
Matrix comparator 33 provides distance code Dl relating the two new input utterances to each o'her.
~atri~ comparator 34 provides distance code D2 relating the prototype and the second ne~J inp~lt utterance to each other. Matrix comparator 35 provides distance code D3 relating the prototype and the ~irst new input utterance to each other.
The distance codes are compared by comparators 36 and 37. The function desired is to replace the prototype utterance by the utterance INPUT 2 where it is determined that INPUT 1 and INPUT 2 dirfer less from each other than they do ~rom the prototype.
If comparator 36 determines that INPUT 2 is closer to the prototype than to INPUT 1 (Dl is NOT less than D2) then mechanism 35 transmits on "NO" path~.~ay 38 to Node (2) in FIG. 1, signalling the decision mechanism to go on without altering the prototype, Similarly, i~ comparator 37 determines that INPUT 1 is closer to the prototype than to INPVT 2 (Dl is NOT
less than D3) then mechanism 37 ira~smits on "NO"
line 38 to Node (2) in FIG. 1, signalling the decision i ~) {; d ' ~
~4~
-12- s mechanism to go on with-~ut altering the ?ro~ot pe.

The function desired is to alter the Drotot~?e when Dl is less than D2 and Dl is also less than D3. Comparator 37 provides a "YES" out?ut on 'ine 39 to initiate P~OTOTYPE=I~PUT 2 action Srom mechan-lsm 40. Signal line 19 at Node (3) in FIG. 1 controls the action of replacing the prototype ~oca-bulary entry with INPUT 2.

This adaptive replacement of the prototy?e voca-bulary entry as a Sunction Gf the determination that the new utterance is acceptable permits a prototype vocabulary to be established once, then used insofar as acceptable, with unobtrusive adaptive alteration of a limited number of vocabulary words as they appear in the context of normal operation. When the prototype word is m~srecosnize~
for any reason (dialect, pronunciation or other difference between speakers -- or chanse ir. deliverv by the same talker), this system identifies the misrecognition, prompts a twin input of the utterance, compares the inputs to each other and to the proto-type, and in the situation where it is determined that the inputs fulfill the criteria, replace the prototype with one of the new inputs.

FIG. 2 details the procedures for adaptive retraining of prototype vocabulary items. During the recognition process if the talker produces a keyword mistake the adaptive retxaining routine is invoked. Under computer control the procedures are entered and a stack of vocabulary item choices -- related to the words just before the keyword mistake recognition --is sent to the adaptive training stage. Using this stack the system prompts the user to indicate which stack vocabulary item was ut-tered just prior ~,Z;~

to its identification as an unrecoynized utterance.
(This prompting can either be by audio response or character display from a gas panel, catnode ray tube, or the like.) If the word equivalent to S the misrecognized utterance is not contained in the stack, the user is prompted to recite the utterance again, and the recognition procedure continues.
Upon acknowledgment by the talker tha~ a stack voca~ulary item matches the utterance (INPUT 1), the user is prompted to provide a new prototype keyword candidate (INPUT 2) appropriate to that vocabulary item. The system calculates the matrix distances [INPUT 1 vs. INPUT 2=(Dl)], [PROTOTYPE
vs. INPUT 2=(D2)1, and [PROTOTYPE vs. INPUT l=(D3)], where IN~UT 1 is the misrecognized utterance and INPUT 2 is the prompted utterance of the same key-word. If D1 is less than D2, and if Dl is less than D3, then INPUT 2 becomes the new prototype keyword.
Otherwise the old prototype is retained. This procedure guarantees the best currentlv available prototype relative to the current user's (the user's current) speech characteristics. This procedure - is independent of the mode of determining the par-ticuiar matrix distance characteristics employed in the recognition process.

Claims (4)

INVENTIVE METHOD

The following steps are followed:

1. Prototypical talker recites the keyword set, which keyword set is stored in coded form as the prototype vocabulary keyword set.
Alternatively, the prototype vocabulary key-word set may be calculated and stored.

2. A subsequent talker utters a sample keyword for recognition.

3. If recognition occurs, proceed.

4. If recognition fails, enter adaptive retraining routine.

5. Set adaptive training stage with stack of recognition choice words related to the misrecognized keyword.

6. Prompt user to indicate which word of the stack was intended.

7. Check whether misrecognized keyword is in stack.

8. If 7 is negative, prompt talker to repeat misrecognized keyword.

9. Proceed with recognition procedure.

10. If 7 is positive, prompt talker to utter a new prototype candidate.

11. Calculate matrix distances:

INPUT 1 vs. INPUT 2=(D1) PROT vs. INPUT 2=(D2) PROT vs. INPUT 1=(D3).

12. Compare D1, D2, D3.

13. If D1<D2 AND D1<D3, then select INPUT 2 as new prototype and proceed.

The inventive method may be performed according to a number of variations without departing from the spirit and scope of the adaptive automatic discrete utterance recognition technique of this Patent Specification as pointed out in the following claims.

The embodiments of the invention in which an exclusive property or privilege is claimed are defined as follows:

1. A method of adaptive automatic discrete utterance recognition comprising:

a) providing the system with a single utterance set of prototype vocabulary keywords;

b) operating the system until a misrecognized utterance is detected;

c) prompting a prompted prototype vocabulary keyword candidate utterance;

d) calculating recognition distances for the unrecognized utterance, the prompted prototype vocabulary keyword candidate utterance, and the proto-type vocabulary keyword;

e) comparing the calculated recognition distances; and f) selectively accepting as prototype vocabulary keyword the prototype vocabulary keyword candidate utterance as a function of said step of comparing the calculated recognition distances.

2. A method for adaptive automatic discrete utterance voice recognition, characterized by the following steps:

a) providing a single recitation prototype vocabulary keyword set containing a multiplicity or coded prototype vocabulary items;

b) providing the prototype vocabulary keyword set for use by a subsequent talker;

c) proceeding by carrying out a sequence of vocabulary item recognitions until a recognition difficulty occurs on a particular vocabulary item utterance, which may be termed misrecognized utterance;

d) commencing a retraining routine by which the system prompts the talker to retrain the system by carrying out the following retraining steps:

1) isolating a stack of prototype voca-bulary keywords appropriate to the misrecognized utterance;

2) prompting the talker to choose a prototype vocabulary keyword from such stack of prototype vocabulary keywords;

3) prompting the talker to provide a prompted prototype vocabulary key-word associated with the accepted choice, as a prototype vocabulary keyword candidate utterance;

4) calculating distance relationships for the difference distance D1 between the misrecognized utterance and the prompted prototype vocabulary keyword candidate utterance, for the difference distance D2 between the prompted prototype vocabulary keyword candidate utterance and the prototype vocabulary keyword, and for the difference distance D3 between the misrecognized utterance and the prototype vocabulary keyword;

5) comparing the difference distances D1, D2 and D3 calculated in said calculating distance relationships step; and 6) selectively replacing the chosen prototype vocabulary keyword as a function of the result of said comparing the difference distances step; and e) returning to proceeding step (c).

3. The method of Claim 2, further characterized in that said comparing the difference distances step (i) makes comparisons D1<D2? and D1<D3?, and further characterized in that said selectively replacing the selected prototype vocabulary keyword step (j) is operative upon either negative result to return to step (d), and is operative upon positive result for both comparisons to activate replacement of the prototype vocabulary keyword by the coded equivalent of the prompted prototype vocabulary keyword candidate.

4. A system for speech recognition having operating means to recognize, during normal operation mode, utterances by a talker in accordance with a prototype vocabulary keyword set, -- characterized by --a) initializing mode control means operative during a first prototype vocabulary keyword initializing operation to establish as a criterion for recognition the coded equivalent of the prototype keyword utter-ance as a prototype vocabulary keyword;

b) mode control means operative during normal operation mode to identify a mis-recognized utterance situation and cause mode transfer to adaptive retraining mode;
and c) adaptive retraining mode control means operative to control, in sequence, the following:

1) means to inform the talker or a list of utterance choices for acceptance or rejection by the talker;

2) means responsive to the talker's rejection or acceptance respectively to inform the talker of a new list of choices, and repeat steps 1 and 2 until all lists are exhausted, or to proceed in retraining mode;

3) retraining mode means effective to prompt talker to utter the keyword associated with the accepted choice, as a prototype vocabulary keyword candidate utterance;