WO2006106415A1 - Method, device, and computer program product for multi-lingual speech recognition - Google Patents
Method, device, and computer program product for multi-lingual speech recognition Download PDFInfo
- Publication number
- WO2006106415A1 WO2006106415A1 PCT/IB2006/000797 IB2006000797W WO2006106415A1 WO 2006106415 A1 WO2006106415 A1 WO 2006106415A1 IB 2006000797 W IB2006000797 W IB 2006000797W WO 2006106415 A1 WO2006106415 A1 WO 2006106415A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- language
- alphabet
- character
- mapping table
- word
- Prior art date
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/10—Text processing
- G06F40/12—Use of codes for handling textual entities
- G06F40/126—Character encoding
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
- G06F40/263—Language identification
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
- G10L13/00—Speech synthesis; Text to speech systems
- G10L13/08—Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
- G10L15/00—Speech recognition
- G10L15/005—Language recognition
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
- G10L15/00—Speech recognition
- G10L15/08—Speech classification or search
- G10L15/18—Speech classification or search using natural language modelling
- G10L15/183—Speech classification or search using natural language modelling using context dependencies, e.g. language models
- G10L15/187—Phonemic context, e.g. pronunciation rules, phonotactical constraints or phoneme n-grams
Definitions
- the present invention relates generally to speech recognition methods and systems. More specifically, the present invention relates to a method, device, and computer program product for multi-lingual speech recognition.
- ASR Automatic Speech Recognition
- SDND speaker dependent name dialing
- Pronunciation modeling also called text-to- phoneme mapping (TTP)
- TTP text-to- phoneme mapping
- ML-SIND Multilingual Speaker-Independent Name Dialing
- Viikki et al. Speech Recognition in Mobile Communication Systems
- Viikki et al. Speech Recognition in Mobile Communication Systems
- Automatic language identification is an integral part of multilingual systems that use dynamic vocabularies. LID module detects the language of the vocabulary item.
- the language dependent pronunciation model is applied to obtain the phoneme sequence associated with the written form of the vocabulary item.
- the recognition model for each vocabulary item is constructed by concatenating the multilingual acoustic models. Using these basic modules the recognizer can, in principle, automatically cope with multilingual vocabulary items without any assistance from the user.
- the pronunciation of a given text in a particular language can usually be found in automatic speech recognition and text-to-speech systems. However, conventional systems are generally unable to find the pronunciations of the texts in any other language supported by the system. Other languages may be considered mismatched languages. It is common to have the mismatched languages due to some reasons, e.g. LID errors, non-native vocabulary items, N-Best or multiple pronunciation scheme, etc. It is not trivial to find the pronunciations of the given texts in mismatched languages because different languages have different alphabet sets and different pronunciation rules. For example, one cannot directly find English pronunciation of Russian text " ⁇ op ⁇ c" because of the different alphabet sets between English and Russian.
- exemplary embodiments include a symbol conversion table that maps the alphabet set of any language to the alphabet set of any other language.
- a text-to-phoneme module of a multi-lingual speech recognition system can provide pronunciations for words in the vocabulary of a language.
- Techniques to carry out the symbol mapping can include language-dependent and language- independent parts.
- One exemplary embodiment relates to a method of multi-lingual speech recognition. This method can include determining whether characters in a word are in a source list of a language-specific alphabet mapping table for a language, converting each character not in the source list according to a general alphabet mapping table and, where such conversion is performed, converting each converted character according to the language-specific alphabet mapping table for the language. The method further includes verifying that each character in the word is in a character set of the language, removing characters not in the character set of the language, and identifying a pronunciation of the word.
- the device can include a LID module that assigns language identifiers to words, a TTP module that applies language-specific TTP models to generate a multi-lingual phoneme sequence associated with words identified by the LID module, a multi-lingual acoustic modeling module that constructs a recognition model for each vocabulary entry according to phonetic transcription, and a processor.
- the processor executes programmed instructions to determine whether characters in a word are in a source list of a language-specific alphabet mapping table for a language, convert each character not in the source list according to a general alphabet mapping table, convert each converted character according to the language-specific alphabet mapping table, and remove characters not in the character set of the language.
- Another exemplary embodiment relates to a computer program product including computer code that determines whether characters in a word are in a source list of a language-specific alphabet mapping table for a language, computer code that searches a general alphabet mapping table for each character in the word that is not in the source list, computer code that converts each character not in the source list according to the general alphabet mapping table, computer code that converts each converted character according to the language-specific alphabet mapping table, and computer code that removes characters not in a character set of the language.
- the device can include means for assigning language identifiers to words, means for applying language-specific text-to-phoneme (TTP) models to generate a multi-lingual phoneme sequence associated with words identified by the assigning means, means for constructing a recognition model for each vocabulary entry according to phonetic transcription, and means for executing programmed instructions to determine whether characters in a word are in a source list of a language-specific alphabet mapping table for a language, convert each character not in the source list according to a general alphabet mapping table, convert each converted character according to the language-specific alphabet mapping table, and remove characters not in the character set of the language.
- TTP language-specific text-to-phoneme
- FIG. 1 is a general diagram of a multi-lingual speech recognition system in accordance with an exemplary embodiment.
- FIG. 2 is a diagram depicting operations performed in the text processing in the multi-lingual speech recognition system of Fig. 1 in accordance with an exemplary embodiment.
- Fig. 3 is a language-specific mapping table for French, Portuguese, and Spanish, in accordance with an exemplary embodiment.
- Fig. 4 is a language-specific mapping table for Romanian in accordance with an exemplary embodiment.
- Fig. 5 is a general mapping table for the multi-lingual speech recognition engine of Fig. 1 with French, Portuguese, Spanish, and English in accordance with an exemplary embodiment.
- Fig. 6 is a language-specific mapping table for Ukrainian in accordance with an exemplary embodiment.
- FIG. 7 is a diagram depicting a mobile station in accordance with an exemplary embodiment.
- Fig. 1 illustrates a multi-lingual speech recognition system 10 including an LID module 12, an on-line language-specific TTP module 14, and a multi-lingual acoustic modeling module 16.
- the LID module 12 assigns language tags or identifiers to each new word.
- the TTP module 14 applies language-specific TTP models to generate a multi-lingual phoneme sequence associated with the written form of the vocabulary item.
- the multi-lingual acoustic modeling module 16 constructs a recognition model for each vocabulary entry according to phonetic transcription.
- the speaker independent multi-lingual speech recognition (ML-ASR) system 10 operates on a vocabulary of words given in textual form.
- the words in the vocabulary may originate from multiple languages, and the language identifiers are not given by the user.
- the LID module 12 of the ML-ASR system 10 provides the language IDs for the words.
- the LID module 12 makes errors and the language IDs are not always correct.
- the LID module 12 was trained on four languages, including English, French, Portuguese, and Spanish.
- Table 1 below presents language identification rates for these languages.
- LID rates can be low. Due to such low LID rates, anN-best set of LID languages is utilized. If the N-best list is of suitable size, the correct language is one of the language IDs in the list and the correct pronunciation can be found for the word. In addition, some texts, for example many names, may belong to different languages, so multiple languages are used. Many loanwords also require proper handling of the mismatch languages. For the non-native speakers, it is common to speak the multi-lingual texts in their mother tongue that may be other than the matched language. [00022] Different languages have different alphabet sets.
- the multi-lingual speech recognition system 10 includes a mapping from the character set of any language to the character set of any other language. With this kind of mapping, the TTP module 14 can provide the pronunciations for the N-best list of languages of each word in the vocabulary.
- Fig. 2 illustrates operations performed in the processing of textual input for multi-lingual speech recognition. This processing includes the mapping between languages supported by the ML-ASR system to the alphabet of any other language supported by the ML-ASR system. Additional, fewer, or different operations may be performed depending on the particular embodiment.
- an N-best list of languages is provided for each word in a vocabulary.
- an alphabet mapping table is provided for each language supported by the ML-ASR system.
- a general alphabet mapping table composed of the characters in all the language-specific alphabet sets.
- a standard alphabet set is defined.
- normal Latin characters [a-z] are used.
- the Uh language-specific and the standard alphabet sets are denoted as LS t , and SS, respectively.
- SS (Si, s 2, , SM ⁇ ; where c ⁇ , and S k are the kth characters in the Uh language-specific and the standard alphabet sets, ni and Mare the sizes of the Uh language-dependent and the standard alphabet sets.
- a source character belongs to the union of all the characters in all the language specific alphabet sets LS 1 ; and a target character belongs to the standard alphabet set SS.
- the corresponding mapping is denoted as GeneralQ.
- a source character belongs to the union of the language specific alphabet set LS 1 and the standard alphabet set SS; and a target character belongs to the language specific alphabet set LS 1 only.
- the corresponding mapping is denoted as Language ,(.).
- mapping is applied
- mapping back to the alphabet set is achieved by the proper definition of the mapping Language ⁇ .). Accordingly, by applying the mappings above, the characters of the word after the mapping belong to the language specific alphabet whether the language matches the word or not.
- an operation 26 the characters of a word are processed one character at a time.
- the language specific alphabet mapping table is checked.
- a decision is made in operation 28 if the character is in the L_source list of the language specific alphabet mapping table. If it is in the Ljsource list, it is kept unchanged. If the character is not in the L_source list of the language specific alphabet mapping table, the general alphabet mapping table is checked in an operation 30 and the character is converted according to the general alphabet mapping table in an operation 32.
- Fig. 3 illustrates language dependent mapping tables for French, Portuguese and Spanish.
- the language dependent mapping tables define, how the non-standard characters outside the [a-z] range are treated for the language.
- the source identifies the character as is appears in a word
- the target identifies the character as it appears in the word after the application of the language specific mapping table.
- the language specific mapping table of English is empty, since the character set of US-English is composed of the standard character set only.
- Fig. 4 illustrates a language dependent mapping table for Bulgarian. If a character in the standard [a-z] character set is not included in the alphabet of the language, it needs to be indicated in the language dependent mapping table.
- the alphabet of Bulgarian does not include the standard characters q, w, and y.
- the symbol ⁇ NULL> in the Romanian table indicates that the letters are not in the alphabet of the language, and they are removed from the words when the words are processed and passed on to TTP.
- Fig. 5 illustrates a general language independent mapping table that maps all the non-standard characters outside the [a-z] range into the standard [a-z] range for the ML-ASR system supporting French, Portuguese, Spanish, and English.
- the table defines how the characters not included in the standard character set are mapped to the standard character set, unless they are present in the alphabet of the language.
- the languages with the Latin character set need to be supported as well. Therefore, the Cyrillic characters are included in the language dependent character tables.
- the standard set of characters needs to be mapped to the alphabet of the Cyrillic language. Due to these reasons, the language dependent character set of Ukrainian, for example, is illustrated in the table of Fig. 6. Note that in addition to the Cyrillic characters, the table provides also the mapping from the standard character set to the alphabet set of the language.
- mapping tables Given these mapping tables, the pronunciation of the given texts in any of the languages supported by the ML-ASR system can be found. Assume that there is a given text that needs to be converted into the language that has ID i. Then, all the letters of the text are mapped to the alphabet set of the given language according to the following exemplary logic.
- the logic is applied to all the letters in the text, and the result is the text that has been converted to the alphabet set of the language i.
- Fig. 7 illustrates a mobile station 30.
- the mobile station 30 can include a display 32, a text input module 34, and an LID system 36.
- the LID system 36 can include a mapping module 38 for mapping a word provided by the text input module 32 using the characters of the standard set 40.
- the LID system 36 further can include a language identifier (LID) module 42, an alphabet-scoring module 44, a plurality of language-dependent alphabet sets 46 and a decision module 48.
- the decision module 48 can include a processor with programmed instructions to perform the operations described above with reference to Figs. 2-6.
Abstract
A method of multi-lingual speech recognition can include determining whether characters in a word are in a source list of a language-specific alphabet mapping table for a language, converting each character not in the source list according to a general alphabet mapping table, converting each converted character according to the language-specific alphabet mapping table, verifying that each character in the word is in a character set of the language, removing characters not in the character set of the language, and identifying a pronunciation of the word.
Description
METHOD5 DEVICE, AND COMPUTER PROGRAM PRODUCT FOR MULTI-LINGUAL SPEECH RECOGNITION
FIELD OF THE INVENTION
[0001] The present invention relates generally to speech recognition methods and systems. More specifically, the present invention relates to a method, device, and computer program product for multi-lingual speech recognition.
BACKGROUND INFORMATION
[0002] This section is intended to provide a background or context. The description herein may include concepts that could be pursued, but are not necessarily ones that have been previously conceived or pursued. Therefore, unless otherwise indicated herein, what is described in this section is,not prior art to the claims in this application and is not admitted to be prior art by inclusion in this section.
[0003] Automatic Speech Recognition (ASR) technologies have been adopted in mobile phones and other hand-held communication devices. A speaker-trained name dialer is probably one of the most widely distributed ASR applications. In the speaker-trained name dialer, the user has to train the models for recognition, and it is known as a speaker dependent name dialing (SDND) application. Applications that rely on more advanced technology do not require the user to train any models for recognition. Instead, the recognition models are automatically generated based on the orthography of the multi-lingual words. Pronunciation modeling, also called text-to- phoneme mapping (TTP), based on orthography of the multi-lingual words is used, for example, in the Multilingual Speaker-Independent Name Dialing (ML-SIND) system, as disclosed in Viikki et al. ("Speaker- and Language-Independent Speech Recognition in Mobile Communication Systems", in Proceedings of International Conference on Acoustics, Speech, and Signal Processing, Salt Lake City, Utah, USA 2002). Due to globalization as well as the international nature of the markets and future applications in mobile phones, the demand for multilingual speech recognition systems is growing rapidly.
[0004] Automatic language identification (LID) is an integral part of multilingual systems that use dynamic vocabularies. LID module detects the language of the vocabulary item. Once the language has been determined, the language dependent pronunciation model is applied to obtain the phoneme sequence associated with the written form of the vocabulary item. Finally, the recognition model for each vocabulary item is constructed by concatenating the multilingual acoustic models. Using these basic modules the recognizer can, in principle, automatically cope with multilingual vocabulary items without any assistance from the user. The pronunciation of a given text in a particular language can usually be found in automatic speech recognition and text-to-speech systems. However, conventional systems are generally unable to find the pronunciations of the texts in any other language supported by the system. Other languages may be considered mismatched languages. It is common to have the mismatched languages due to some reasons, e.g. LID errors, non-native vocabulary items, N-Best or multiple pronunciation scheme, etc. It is not trivial to find the pronunciations of the given texts in mismatched languages because different languages have different alphabet sets and different pronunciation rules. For example, one cannot directly find English pronunciation of Russian text "δopπc" because of the different alphabet sets between English and Russian.
[0005] There is a need to handle multi-lingual textual input in multi-lingual automatic speech recognition systems. Further, there is a need to process multilingual automatic speech recognition such that TTP can be applied to find the pronunciation for textual input in any supported language.
SUMMARY OF THE INVENTION
[0006] In general, exemplary embodiments include a symbol conversion table that maps the alphabet set of any language to the alphabet set of any other language. As such, a text-to-phoneme module of a multi-lingual speech recognition system can provide pronunciations for words in the vocabulary of a language. Techniques to carry out the symbol mapping can include language-dependent and language- independent parts.
[0007] One exemplary embodiment relates to a method of multi-lingual speech recognition. This method can include determining whether characters in a word are in a source list of a language-specific alphabet mapping table for a language, converting each character not in the source list according to a general alphabet mapping table and, where such conversion is performed, converting each converted character according to the language-specific alphabet mapping table for the language. The method further includes verifying that each character in the word is in a character set of the language, removing characters not in the character set of the language, and identifying a pronunciation of the word.
[0008] Another exemplary embodiment relates to a device for multi-lingual speech recognition. The device can include a LID module that assigns language identifiers to words, a TTP module that applies language-specific TTP models to generate a multi-lingual phoneme sequence associated with words identified by the LID module, a multi-lingual acoustic modeling module that constructs a recognition model for each vocabulary entry according to phonetic transcription, and a processor. The processor executes programmed instructions to determine whether characters in a word are in a source list of a language-specific alphabet mapping table for a language, convert each character not in the source list according to a general alphabet mapping table, convert each converted character according to the language-specific alphabet mapping table, and remove characters not in the character set of the language.
[0009] Another exemplary embodiment relates to a computer program product including computer code that determines whether characters in a word are in a source list of a language-specific alphabet mapping table for a language, computer code that searches a general alphabet mapping table for each character in the word that is not in the source list, computer code that converts each character not in the source list according to the general alphabet mapping table, computer code that converts each converted character according to the language-specific alphabet mapping table, and computer code that removes characters not in a character set of the language.
[00010] Another exemplary embodiment relates to a device for multi-lingual speech recognition. The device can include means for assigning language identifiers to words, means for applying language-specific text-to-phoneme (TTP) models to generate a multi-lingual phoneme sequence associated with words identified by the
assigning means, means for constructing a recognition model for each vocabulary entry according to phonetic transcription, and means for executing programmed instructions to determine whether characters in a word are in a source list of a language-specific alphabet mapping table for a language, convert each character not in the source list according to a general alphabet mapping table, convert each converted character according to the language-specific alphabet mapping table, and remove characters not in the character set of the language.
BRIEF DESCRIPTION OF DRAWINGS
[00011] Fig. 1 is a general diagram of a multi-lingual speech recognition system in accordance with an exemplary embodiment.
[00012] Fig. 2 is a diagram depicting operations performed in the text processing in the multi-lingual speech recognition system of Fig. 1 in accordance with an exemplary embodiment.
[00013] Fig. 3 is a language-specific mapping table for French, Portuguese, and Spanish, in accordance with an exemplary embodiment.
[00014] Fig. 4 is a language-specific mapping table for Romanian in accordance with an exemplary embodiment.
[00015] Fig. 5 is a general mapping table for the multi-lingual speech recognition engine of Fig. 1 with French, Portuguese, Spanish, and English in accordance with an exemplary embodiment.
[00016] Fig. 6 is a language-specific mapping table for Ukrainian in accordance with an exemplary embodiment.
[00017] Fig. 7 is a diagram depicting a mobile station in accordance with an exemplary embodiment.
DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS
[00018] Fig. 1 illustrates a multi-lingual speech recognition system 10 including an LID module 12, an on-line language-specific TTP module 14, and a
multi-lingual acoustic modeling module 16. The LID module 12 assigns language tags or identifiers to each new word. The TTP module 14 applies language-specific TTP models to generate a multi-lingual phoneme sequence associated with the written form of the vocabulary item. The multi-lingual acoustic modeling module 16 constructs a recognition model for each vocabulary entry according to phonetic transcription.
[00019] The speaker independent multi-lingual speech recognition (ML-ASR) system 10 operates on a vocabulary of words given in textual form. The words in the vocabulary may originate from multiple languages, and the language identifiers are not given by the user. Instead, the LID module 12 of the ML-ASR system 10 provides the language IDs for the words. The LID module 12 makes errors and the language IDs are not always correct. By way of any example, the LID module 12 was trained on four languages, including English, French, Portuguese, and Spanish.
[00020] Table 1 below presents language identification rates for these languages.
Table 1
[00021] As seen from Table 1, LID rates can be low. Due to such low LID rates, anN-best set of LID languages is utilized. If the N-best list is of suitable size, the correct language is one of the language IDs in the list and the correct pronunciation can be found for the word. In addition, some texts, for example many names, may belong to different languages, so multiple languages are used. Many loanwords also require proper handling of the mismatch languages. For the non-native speakers, it is common to speak the multi-lingual texts in their mother tongue that may be other than the matched language.
[00022] Different languages have different alphabet sets. The multi-lingual speech recognition system 10 includes a mapping from the character set of any language to the character set of any other language. With this kind of mapping, the TTP module 14 can provide the pronunciations for the N-best list of languages of each word in the vocabulary.
[00023] Fig. 2 illustrates operations performed in the processing of textual input for multi-lingual speech recognition. This processing includes the mapping between languages supported by the ML-ASR system to the alphabet of any other language supported by the ML-ASR system. Additional, fewer, or different operations may be performed depending on the particular embodiment.
[00024] In an operation 22, an N-best list of languages is provided for each word in a vocabulary. In an operation 24, an alphabet mapping table is provided for each language supported by the ML-ASR system. In addition to the language specific alphabet mapping table, there is a general alphabet mapping table composed of the characters in all the language-specific alphabet sets. In the creation of the mapping tables, a standard alphabet set is defined. By way of example, normal Latin characters [a-z] are used. The Uh language-specific and the standard alphabet sets are denoted as LSt, and SS, respectively. As such,
SS=(Si, s 2, , SM}; where c^, and Sk are the kth characters in the Uh language-specific and the standard alphabet sets, ni and Mare the sizes of the Uh language-dependent and the standard alphabet sets.
[00025] Two kinds of alphabet mapping tables are created. One is the general alphabet mapping table defined for all the characters in all the language specific alphabet sets. Another one is the language specific alphabet mapping table for each language supported by the ML-ASR system. In the mappings tables, the character mapping pairs are defined from the source character (Ljsource) to the target character {L Jar get) as follows:
General
General.Ljsourcei GeneralLjargetj
General. L_sourcem General.L_Jargetm
Language,
Language,.L_source,i Language,.L_target,i
Language,.L_source,ιm Language ,.L Jar -get \m
In the general mapping table, a source character belongs to the union of all the characters in all the language specific alphabet sets LS1; and a target character belongs to the standard alphabet set SS. The corresponding mapping is denoted as GeneralQ.
N
General. L _source € [J LS1 ι=l
General. L _tar get e SS
[00026] In the ith language specific mapping table, a source character belongs to the union of the language specific alphabet set LS1 and the standard alphabet set SS; and a target character belongs to the language specific alphabet set LS1 only. The corresponding mapping is denoted as Language ,(.).
Language , .L _ source e LS1 [J SS Language , .L _ target e LS1
All the language specific characters in the language i are mapped to themselves, i.e. if L^source e LS, then
Language, .L _ target = Language , (Z, _ source) = Language, .L _ source
Taking any letter from a given word, for the matching language k, the word remains unchanged since the mapping applies to all the letters in the word.
[00027] For mismatch language i, if letter e LS1[JSS , then the language dependent mapping is applied:
Language, .L_target - Language, {letter)
N
For mismatch language j, if letter <£ LSj[J SS but ;letter e [J LS, , then the general
mapping is applied
General. L __tar get = General (letter)
After this mapping application, if General (letter) e LS1-(J1SS1 , then a second mapping, a language dependent mapping, is carried out to map the character back to the alphabet set LS1 of the f language:
[00028] The mapping back to the alphabet set is achieved by the proper definition of the mapping Language^.). Accordingly, by applying the mappings above, the characters of the word after the mapping belong to the language specific alphabet whether the language matches the word or not.
[00029] In an operation 26, the characters of a word are processed one character at a time. For each character, the language specific alphabet mapping table is checked. A decision is made in operation 28 if the character is in the L_source list of the language specific alphabet mapping table. If it is in the Ljsource list, it is kept unchanged. If the character is not in the L_source list of the language specific alphabet mapping table, the general alphabet mapping table is checked in an operation 30 and the character is converted according to the general alphabet mapping table in an operation 32.
[00030] After the application of the language specific and the general alphabet mapping tables, it is verified whether the character is included in the character set of the language in an operation 34. If the character is in the alphabet, it is kept in an operation 36. Otherwise, it is removed in an operation 38. A determination is made in operation 40 whether the current character is the last character of the word. If not, the next character is examined. If it is the last character, a pronunciation for the word is found using the TTP module.
[00031] Fig. 3 illustrates language dependent mapping tables for French, Portuguese and Spanish. The language dependent mapping tables define, how the non-standard characters outside the [a-z] range are treated for the language. The source identifies the character as is appears in a word, and the target identifies the character as it appears in the word after the application of the language specific mapping table. The language specific mapping table of English is empty, since the character set of US-English is composed of the standard character set only.
[00032] Fig. 4 illustrates a language dependent mapping table for Romanian. If a character in the standard [a-z] character set is not included in the alphabet of the language, it needs to be indicated in the language dependent mapping table. The alphabet of Romanian, for example, does not include the standard characters q, w, and y. The symbol <NULL> in the Romanian table indicates that the letters are not in the alphabet of the language, and they are removed from the words when the words are processed and passed on to TTP.
[00033] Fig. 5 illustrates a general language independent mapping table that maps all the non-standard characters outside the [a-z] range into the standard [a-z] range for the ML-ASR system supporting French, Portuguese, Spanish, and English. The table defines how the characters not included in the standard character set are mapped to the standard character set, unless they are present in the alphabet of the language.
[00034] In addition to the languages with the Latin character set, the languages with the Cyrillic character set need to be supported as well. Therefore, the Cyrillic characters are included in the language dependent character tables. In addition, due to the N-best approach, it is possible that a Latin name is identified as a Cyrillic word. Therefore, the standard set of characters needs to be mapped to the alphabet of the Cyrillic language. Due to these reasons, the language dependent character set of Ukrainian, for example, is illustrated in the table of Fig. 6. Note that in addition to the Cyrillic characters, the table provides also the mapping from the standard character set to the alphabet set of the language.
[00035] Given these mapping tables, the pronunciation of the given texts in any of the languages supported by the ML-ASR system can be found. Assume that there is a given text that needs to be converted into the language that has ID i. Then, all the letters of the text are mapped to the alphabet set of the given language according to the following exemplary logic.
If letter e LS1[JSS apply Language i(letter) based on the mapping table (e.g. one of the tables in Fig. 3) else
apply General (letter) based on the general mapping table (e.g. table in
Fig. 5) if General{letter) e LS1[J SS apply Language t[General (letter)] endif endif
The logic is applied to all the letters in the text, and the result is the text that has been converted to the alphabet set of the language i.
[00036] Fig. 7 illustrates a mobile station 30. The mobile station 30 can include a display 32, a text input module 34, and an LID system 36. The LID system 36 can include a mapping module 38 for mapping a word provided by the text input module 32 using the characters of the standard set 40. The LID system 36 further can include a language identifier (LID) module 42, an alphabet-scoring module 44, a plurality of language-dependent alphabet sets 46 and a decision module 48. The decision module 48 can include a processor with programmed instructions to perform the operations described above with reference to Figs. 2-6.
[00037] While several embodiments of the invention have been described, it is to be understood that modifications and changes will occur to those skilled in the art to which the invention pertains. Accordingly, the claims appended to this specification are intended to define the invention precisely.
Claims
L A method of multi-lingual speech recognition, the method comprising: determining whether characters in a word are in a source list of a language-specific alphabet mapping table for a language; converting each character not in the source list according to a general alphabet mapping table and, where such conversion is performed, converting each converted character according to the language-specific alphabet mapping table for the language; verifying that each character in the word is in a character set of the language; removing characters not in the character set of the language; and identifying a pronunciation of the word.
2. The method of claim 1, further comprising providing an N-best list of languages for each word in a vocabulary.
3. The method of claim 1 , wherein the language-specific alphabet mapping table is constructed of a language-dependent symbol mapping.
4. The method of claim 1 , wherein the general alphabet mapping table comprises characters in all language-specific alphabet sets.
5. The method of claim 4, wherein the general mapping table comprises source characters and target characters.
6. The method of claim 1 , wherein the pronunciation is identified using a text-to-phoneme (TTP) module.
7. The method of claim 1 , further comprising searching the general alphabet mapping table for each character in the word that is not in the source list.
8. The method of claim 1, further comprising determining whether a word is associated with a mismatched language.
9. The method of claim 8, further comprising applying the language- specific alphabet mapping table if a character in the word belongs to a union of language-specific alphabet set and a standard alphabet set.
10. The method of claim 8, further comprising applying the general alphabet mapping table if a character in the word does not belong to a union of language-specific alphabet set and a standard alphabet set but does belong in a language-specific alphabet set for another language.
11. A device for multi-lingual speech recognition, the device comprising:
a language identification (LID) module that assigns language identifiers to words;
a text-to-phoneme (TTP) module that applies language-specific TTP models to generate a multi-lingual phoneme sequence associated with words identified by the LID module;
a multi-lingual acoustic modeling module that constructs a recognition model for each vocabulary entry according to phonetic transcription; and a processor that executes programmed instructions to determine whether characters in a word are in a source list of a language-specific alphabet mapping table for a language, convert each character not in the source list according to a general alphabet mapping table, convert each converted character according to the language-specific alphabet mapping table, and remove characters not in the character set of the language.
12. The device of claim 11, wherein the processor determines whether a word is associated with a mismatched language.
13. The device of claim 12, further comprising applying the language- specific alphabet mapping table if a character in the word associated with a mismatched language belongs to a union of language-specific alphabet set and a standard alphabet set.
14. The device of claim 12, further comprising applying the general alphabet mapping table if a character in the word associated with a mismatched language does not belong to a union of language-specific alphabet set and a standard alphabet set but does belong in a language-specific alphabet set for another language.
15. The device of claim 11 , comprising a hand-held device.
16. The device of claim 11 , comprising a mobile phone.
17. A computer program product comprising:
computer code that determines whether characters in a word are in a source list of a language-specific alphabet mapping table for a language;
computer code that searches a general alphabet mapping table for each character in the word that is not in the source list;
computer code that converts each character not in the source list according to the general alphabet mapping table;
computer code that converts each converted character according to the language-specific alphabet mapping table; and
computer code that removes characters not in a character set of the language.
18. The computer program product of claim 17, further comprising computer code that identifies a pronunciation of the word.
19. The computer program product of claim 17, further comprising computer code that identifies words with mismatched language identified.
20. The computer program product of claim 19, further comprising applying the language-specific alphabet mapping table if a character in the word associated with the mismatched language identified belongs to a union of language- specific alphabet set and a standard alphabet set.
21. The computer program product of claim 19, further comprising applying the general alphabet mapping table if a character in the word associated with the mismatched language identified does not belong to a union of language- specific alphabet set and a standard alphabet set but does belong in a language- specific alphabet set for another language.
22. A device for multi-lingual speech recognition, the device comprising:
means for assigning language identifiers to words;
means for applying language-specific text-to-phoneme (TTP) models to generate a multi-lingual phoneme sequence associated with words identified by the assigning means;
means for constructing a recognition model for each vocabulary entry according to phonetic transcription; and means for executing programmed instructions to determine whether characters in a word are in a source list of a language-specific alphabet mapping table for a language, convert each character not in the source list according to a general alphabet mapping table, convert each converted character according to the language- specific alphabet mapping table, and remove characters not in the character set of the language.
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US11/100,991 | 2005-04-07 | ||
US11/100,991 US7840399B2 (en) | 2005-04-07 | 2005-04-07 | Method, device, and computer program product for multi-lingual speech recognition |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2006106415A1 true WO2006106415A1 (en) | 2006-10-12 |
Family
ID=36613391
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/IB2006/000797 WO2006106415A1 (en) | 2005-04-07 | 2006-04-06 | Method, device, and computer program product for multi-lingual speech recognition |
Country Status (2)
Country | Link |
---|---|
US (1) | US7840399B2 (en) |
WO (1) | WO2006106415A1 (en) |
Families Citing this family (20)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
SG133419A1 (en) * | 2005-12-12 | 2007-07-30 | Creative Tech Ltd | A method and apparatus for accessing a digital file from a collection of digital files |
DE102006057159A1 (en) * | 2006-12-01 | 2008-06-05 | Deutsche Telekom Ag | Method for classifying spoken language in speech dialogue systems |
US8788256B2 (en) | 2009-02-17 | 2014-07-22 | Sony Computer Entertainment Inc. | Multiple language voice recognition |
US8442829B2 (en) | 2009-02-17 | 2013-05-14 | Sony Computer Entertainment Inc. | Automatic computation streaming partition for voice recognition on multiple processors with limited memory |
US8442833B2 (en) | 2009-02-17 | 2013-05-14 | Sony Computer Entertainment Inc. | Speech processing with source location estimation using signals from two or more microphones |
US8788266B2 (en) * | 2009-04-30 | 2014-07-22 | Nec Corporation | Language model creation device, language model creation method, and computer-readable storage medium |
US9104244B2 (en) * | 2009-06-05 | 2015-08-11 | Yahoo! Inc. | All-in-one Chinese character input method |
US8190420B2 (en) * | 2009-08-04 | 2012-05-29 | Autonomy Corporation Ltd. | Automatic spoken language identification based on phoneme sequence patterns |
DE112010005226T5 (en) | 2010-02-05 | 2012-11-08 | Mitsubishi Electric Corporation | Recognition dictionary generating device and speech recognition device |
US8442827B2 (en) * | 2010-06-18 | 2013-05-14 | At&T Intellectual Property I, L.P. | System and method for customized voice response |
US9431012B2 (en) | 2012-04-30 | 2016-08-30 | 2236008 Ontario Inc. | Post processing of natural language automatic speech recognition |
US9093076B2 (en) * | 2012-04-30 | 2015-07-28 | 2236008 Ontario Inc. | Multipass ASR controlling multiple applications |
KR20140114516A (en) * | 2013-03-15 | 2014-09-29 | 삼성전자주식회사 | A memory controller and a operating method of thereof |
CN105653517A (en) * | 2015-11-05 | 2016-06-08 | 乐视致新电子科技(天津)有限公司 | Recognition rate determining method and apparatus |
CN106844288B (en) | 2015-12-07 | 2022-03-22 | 创新先进技术有限公司 | Random character string generation method and device |
CN109145145A (en) | 2017-06-16 | 2019-01-04 | 阿里巴巴集团控股有限公司 | A kind of data-updating method, client and electronic equipment |
US10593321B2 (en) * | 2017-12-15 | 2020-03-17 | Mitsubishi Electric Research Laboratories, Inc. | Method and apparatus for multi-lingual end-to-end speech recognition |
CN108597493B (en) * | 2018-03-28 | 2019-04-12 | 孔繁泽 | The audio exchange method and audio exchange system of language semantic |
CN111737541B (en) * | 2020-06-30 | 2021-10-15 | 湖北亿咖通科技有限公司 | Semantic recognition and evaluation method supporting multiple languages |
CN112530402B (en) * | 2020-11-30 | 2024-01-12 | 深圳市优必选科技股份有限公司 | Speech synthesis method, speech synthesis device and intelligent equipment |
Citations (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20040078191A1 (en) * | 2002-10-22 | 2004-04-22 | Nokia Corporation | Scalable neural network-based language identification from written text |
Family Cites Families (18)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5649214A (en) * | 1994-09-20 | 1997-07-15 | Unisys Corporation | Method and apparatus for continued use of data encoded under a first coded character set while data is gradually transliterated to a second coded character set |
US5793381A (en) * | 1995-09-13 | 1998-08-11 | Apple Computer, Inc. | Unicode converter |
US5784071A (en) * | 1995-09-13 | 1998-07-21 | Apple Computer, Inc. | Context-based code convertor |
US5758314A (en) * | 1996-05-21 | 1998-05-26 | Sybase, Inc. | Client/server database system with methods for improved soundex processing in a heterogeneous language environment |
US5778213A (en) * | 1996-07-12 | 1998-07-07 | Microsoft Corporation | Multilingual storage and retrieval |
US6204782B1 (en) * | 1998-09-25 | 2001-03-20 | Apple Computer, Inc. | Unicode conversion into multiple encodings |
US6460015B1 (en) * | 1998-12-15 | 2002-10-01 | International Business Machines Corporation | Method, system and computer program product for automatic character transliteration in a text string object |
US6389385B1 (en) * | 1999-07-02 | 2002-05-14 | International Business Machines Corporation | System and method for translating source code |
GB2366940B (en) * | 2000-09-06 | 2004-08-11 | Ericsson Telefon Ab L M | Text language detection |
US20030033334A1 (en) * | 2001-07-13 | 2003-02-13 | International Business Machines Corporation | Method and system for ascertaining code sets associated with requests and responses in multi-lingual distributed environments |
US7043431B2 (en) * | 2001-08-31 | 2006-05-09 | Nokia Corporation | Multilingual speech recognition system using text derived recognition models |
JP4064748B2 (en) * | 2002-07-22 | 2008-03-19 | アルパイン株式会社 | VOICE GENERATION DEVICE, VOICE GENERATION METHOD, AND NAVIGATION DEVICE |
US7389474B2 (en) * | 2003-02-28 | 2008-06-17 | Microsoft Corporation | Language or script-based character set conversions per-application program |
US20050010392A1 (en) * | 2003-07-10 | 2005-01-13 | International Business Machines Corporation | Traditional Chinese / simplified Chinese character translator |
US7310605B2 (en) * | 2003-11-25 | 2007-12-18 | International Business Machines Corporation | Method and apparatus to transliterate text using a portable device |
US20050144003A1 (en) * | 2003-12-08 | 2005-06-30 | Nokia Corporation | Multi-lingual speech synthesis |
US7218252B2 (en) * | 2004-02-25 | 2007-05-15 | Computer Associates Think, Inc. | System and method for character conversion between character sets |
US7506254B2 (en) * | 2005-04-21 | 2009-03-17 | Google Inc. | Predictive conversion of user input |
-
2005
- 2005-04-07 US US11/100,991 patent/US7840399B2/en not_active Expired - Fee Related
-
2006
- 2006-04-06 WO PCT/IB2006/000797 patent/WO2006106415A1/en not_active Application Discontinuation
Patent Citations (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20040078191A1 (en) * | 2002-10-22 | 2004-04-22 | Nokia Corporation | Scalable neural network-based language identification from written text |
Non-Patent Citations (1)
Title |
---|
JILEI TIAN1 ET AL: "MULTILINGUAL PRONUNCIATION MODELING FOR IMPROVING MULTILINGUAL SPEECH RECOGNITION", ICSLP 2002 : 7TH INTERNATIONAL CONFERENCE ON SPOKEN LANGUAGE PROCESSING. DENVER, COLORADO, SEPT. 16 - 20, 2002, INTERNATIONAL CONFERENCE ON SPOKEN LANGUAGE PROCESSING. (ICSLP), ADELAIDE : CAUSAL PRODUCTIONS, AU, vol. VOL. 4 OF 4, 16 September 2002 (2002-09-16), pages 497, XP007011165, ISBN: 1-876346-40-X * |
Also Published As
Publication number | Publication date |
---|---|
US7840399B2 (en) | 2010-11-23 |
US20060229864A1 (en) | 2006-10-12 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US7840399B2 (en) | Method, device, and computer program product for multi-lingual speech recognition | |
US8990089B2 (en) | Text to speech synthesis for texts with foreign language inclusions | |
EP2571023B1 (en) | Machine translation-based multilingual human-machine dialog | |
Ramani et al. | A common attribute based unified HTS framework for speech synthesis in Indian languages | |
US6243680B1 (en) | Method and apparatus for obtaining a transcription of phrases through text and spoken utterances | |
US8498857B2 (en) | System and method for rapid prototyping of existing speech recognition solutions in different languages | |
US7472061B1 (en) | Systems and methods for building a native language phoneme lexicon having native pronunciations of non-native words derived from non-native pronunciations | |
US6910012B2 (en) | Method and system for speech recognition using phonetically similar word alternatives | |
EP0262938B1 (en) | Language translation system | |
US8954333B2 (en) | Apparatus, method, and computer program product for processing input speech | |
US8566076B2 (en) | System and method for applying bridging models for robust and efficient speech to speech translation | |
US20050267757A1 (en) | Handling of acronyms and digits in a speech recognition and text-to-speech engine | |
US20050033575A1 (en) | Operating method for an automated language recognizer intended for the speaker-independent language recognition of words in different languages and automated language recognizer | |
US20110307252A1 (en) | Using Utterance Classification in Telephony and Speech Recognition Applications | |
CN111508479B (en) | Voice recognition method, device, equipment and storage medium | |
US20150348543A1 (en) | Speech Recognition of Partial Proper Names by Natural Language Processing | |
EP4085451B1 (en) | Language-agnostic multilingual modeling using effective script normalization | |
CN113692616A (en) | Phoneme-based contextualization for cross-language speech recognition in an end-to-end model | |
US20190005950A1 (en) | Intention estimation device and intention estimation method | |
WO2023045186A1 (en) | Intention recognition method and apparatus, and electronic device and storage medium | |
Gao et al. | Zero-Shot Cross-Lingual Phonetic Recognition with External Language Embedding. | |
CN110852075A (en) | Voice transcription method and device for automatically adding punctuation marks and readable storage medium | |
Ananthakrishnan et al. | Automatic diacritization of Arabic transcripts for automatic speech recognition | |
Patel et al. | Development of Large Vocabulary Speech Recognition System with Keyword Search for Manipuri. | |
CN111933116B (en) | Speech recognition model training method, system, mobile terminal and storage medium |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application | ||
NENP | Non-entry into the national phase |
Ref country code: DE |
|
WWW | Wipo information: withdrawn in national office |
Country of ref document: DE |
|
NENP | Non-entry into the national phase |
Ref country code: RU |
|
WWW | Wipo information: withdrawn in national office |
Country of ref document: RU |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 06727429 Country of ref document: EP Kind code of ref document: A1 |
|
WWW | Wipo information: withdrawn in national office |
Ref document number: 6727429 Country of ref document: EP |