US20040167771A1 - Method and system for reducing lexical ambiguity - Google Patents
Method and system for reducing lexical ambiguity Download PDFInfo
- Publication number
- US20040167771A1 US20040167771A1 US10/781,223 US78122304A US2004167771A1 US 20040167771 A1 US20040167771 A1 US 20040167771A1 US 78122304 A US78122304 A US 78122304A US 2004167771 A1 US2004167771 A1 US 2004167771A1
- Authority
- US
- United States
- Prior art keywords
- lexical
- tokens
- cost
- paths
- graph
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
- G06F40/279—Recognition of textual entities
- G06F40/284—Lexical analysis, e.g. tokenisation or collocates
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/40—Processing or translation of natural language
- G06F40/42—Data-driven translation
- G06F40/44—Statistical methods, e.g. probability models
Definitions
- the present invention relates to language translation systems. More particularly, the present invention relates to a method for reducing lexical ambiguity.
- a typical language translation system functions by using natural language process.
- Natural language processing is generally concerned with the attempt to recognize a large pattern or sentence by decomposing it into small subpatterns according to linguistic rules.
- a natural language processing system uses considerable knowledge about the structure of the language, including what the words are, how words combine to form sentences, what the words mean, and how word meanings contribute to sentence meanings.
- linguistic behavior cannot be completely accounted for without also taking into account another aspect of what makes humans intelligent—their general world knowledge and their reasoning abilities. For example, to answer questions, to participate in a conversation, or to create and understand written language, a person not only must have knowledge about the structure of the language being used, but also must know about the world in general and the conversational setting in particular.
- phonetic and phonological knowledge concerns how words are related to sounds that realize them.
- Morphological knowledge concerns how words are constructed from more basic units called morphemes.
- Syntactic knowledge concerns how words can be put together to form correct sentences and determines what structural role each word plays in the sentence and what phrases are subparts of what other phrases.
- Typical syntactic representations of language are based on the notion of context-free grammars, which represent sentence structure in terms of what phrases are subparts of other phrases. This syntactic information is often presented in a tree form.
- Semantic knowledge concerns what words mean and how these meanings combine in sentences to form sentence meanings. This is the study of context-independent meaning—the meaning a sentence has regardless of the context in which it is used.
- the representation of the context-independent meaning of a sentence is called its logical form.
- the logical form encodes possible word senses and identifies the semantic relationships between the words and phrases.
- Natural language processing systems further include interpretation processes that map from one representation to the other. For instance, the process that maps a sentence to its syntactic structure and logical form is called parsing, and it is performed by a component called a parser.
- the parser uses knowledge about word and word meaning, the lexicon, and a set of rules defining the legal structures, the grammar, in order to assign a syntactic structure and a logical form to an input sentence.
- a context-free grammar of a language is a four-tuple comprising nonterminal vocabularies, terminal vocabularies, a finite set of production rules, and a starting symbol for all productions.
- the nonterminal and terminal vocabularies are disjoint.
- the set of terminal symbols is called the vocabulary of the language.
- Pragmatic knowledge concerns how sentences are used in different situations and how use affects the interpretation of the sentence.
- a natural language processor receives an input sentence, lexically separates the words in the sentence, syntactically determines the types of words, semantically understands the words, pragmatically determines the type of response to generate, and generates the response.
- the natural language processor employs many types of knowledge and stores different types of knowledge in different knowledge structures that separate the knowledge into organized types.
- lexical ambiguity may hinge on the fact that a particularly word has more than one meaning.
- the word bank can be used to denote either a place where monetary exchange and handling takes place or the land close river, the bank of the river.
- a word or a small group of words may also have two or more related meanings. That is, the adjective bright may be used as a synonym for “shining” (e.g., “The stars are bright tonight”) or as a synonym for “smart” (e.g., “She must be very bright if she made an “A” on the test”).
- POS Part-of-speech
- one word may have more than one POS tag depending on the context of POS within the sentence.
- the word table for example, can be a verb in some contexts (e.g., “He will table the motion”) and a noun in others (e.g., “The table is ready”).
- multiword expressions in many languages, including the English language, is yet another factor contributing to lexical ambiguity. That is, depending on the context, a group of words, such as “white house”, can be treated as a multiword expression (e.g., “I want to visit the White House”) or as separate words (e.g., “He lives in a white house across the street”).
- Another current approach to deal with lexical ambiguity recognizes all the possible words in a Japanese sentence and then finds possible connections between adjacent words.
- the recognition of all the words is done using a morpheme dictionary.
- the morpheme dictionary defines Japanese morphemes with the names of POS tags.
- the connectivity is defined using a connection-pair grammar.
- the connection-pair grammar defines pairs of sets of morphemes that may occur adjacently in a sentence.
- Various costs are then applied to the morphemes to compare all possible segmentations of the input sentence. These various costs correspond to the likelihood of observing a word as a certain part of speech and to the likelihood of observing two words in adjacent positions.
- the segmentation that has the lowest corresponding cost is selected from all the possible segmentations of the input sentence for further processing.
- the segmentation selected based upon the lowest costs may not correspond to the correct meaning of the input sentence. Since the syntactic parser is better equipped to recognize the correct meaning of the input sentence, making a selection before the parsing operation may result in loss of pertinent information. Consequently, this approach may lead to inaccurate results in producing a response to an input sentence; especially in producing a response to a longer or more complicated sentence.
- the techniques currently used to deal with lexical ambiguity in an English sentence have problems similar to those identified above. Unlike Japanese sentences, English sentences do not need to be segmented as the individual words form the segments.
- Another approach analyzes all POS tags for each word in an English input sentence and finds the most likely POS tag for each word using lexical and statistical probabilities.
- some probabilities may be hard to estimate. No matter how much text is analyzed for the estimation, there will always be a large volume of words that appear only a few times. Thus, relying strictly on probabilities may not result in an accurate interpretation, especially in dealing with a long or complex sentence in which a word's meaning is dependent upon the context of the word within the sentence.
- the syntactic parser is better equipped to recognize the correct meaning of the input sentence, making a selection before the parsing operation may result in loss of pertinent information.
- a method and system for reducing lexical ambiguity in an input stream are described.
- the input stream is broken into tokens.
- the tokens are used to create a connection graph comprising a number of paths. Each of the paths is assigned a cost. At least one best path is defined based upon a corresponding cost to generate an output graph.
- the generated output graph is provided to reduce lexical ambiguity.
- FIG. 1 is a block diagram of one embodiment for an architecture of a computer system
- FIG. 2 a is a block diagram of one embodiment for a natural language translation system
- FIGS. 2 b , 2 c , and 2 d are exemplary diagrams of structures used by the natural language translation system of FIG. 2 a;
- FIG. 3 is a diagram of one embodiment for a lexical ambiguity module
- FIG. 4 is a flow diagram of one embodiment for reducing lexical ambiguity in a natural language translation system:
- FIG. 5 a illustrates an exemplary connection graph
- FIG. 5 b illustrates an exemplary path in a connection graph
- FIG. 6 is a flow diagram of one embodiment for segmentation of an input stream
- FIG. 7 is a flow diagram of one embodiment for reducing lexical ambiguity in an input English expression
- FIG. 8 is a flow diagram of one embodiment for reducing lexical ambiguity in an input Japanese expression
- FIG. 9 illustrates an exemplary connection of tokens in an input Japanese sentence.
- a method and system for reducing lexical ambiguity in an input stream are described.
- the input stream is broken into tokens.
- the tokens are used to create a connection graph comprising a number of paths. Each of the paths is assigned a cost. At least one best path is defined based upon a corresponding cost to generate an output graph.
- the generated output graph is provided to reduce lexical ambiguity.
- the present invention also relates to apparatus for performing the operations herein.
- This apparatus may be specially constructed for the required purposes, or it may comprise a general purpose computer selectively activated or reconfigured by a computer program stored in the computer.
- a computer program may be stored in a computer readable storage medium, such as, but is not limited to, any type of disk including floppy disks, optical disks, CD-ROMs, and magnetic-optical disks, read-only memories (ROMs), random access memories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, or any type of media suitable for storing electronic instructions, and each coupled to a computer system bus.
- Lexical ambiguity is a recognized problem in natural language processing.
- the problem with lexical ambiguity arises when a natural language processor needs to extract information from an input sentence for the subsequent syntactic parsing. Extracting information becomes problematic in character-based languages which do not have separators such as spaces between words in a sentence.
- a word may have different part of speech (POS) tags depending on the context of the POS within the sentence.
- POS part of speech
- certain groups of words can either be treated as multiword expressions or as separate words depending on the context.
- the lexical ambiguity reduction module provides a method for reducing lexical ambiguity in an input sentence which increases the segmentation of the input sentence and supports POS tagging and multiword processing.
- the lexical ambiguity module provides a graph which is passed to a syntactic analysis module for subsequent processing.
- an efficient method of reducing lexical ambiguity is provided which allows the language processing system to produce an accurate interpretation of the input sentence without unreasonably burdening the operation of the syntactic analysis module.
- FIG. 1 is a block diagram of one embodiment for an architecture of a computer system 100 .
- computer system 100 includes system bus 101 that allows for communication among processor 102 , digital signal processor 108 , memory 104 , and non-volatile storage device 107 .
- System bus 101 may also receive inputs from keyboard 122 , pointing device 123 , and speech signal input device 125 .
- System bus 101 provides outputs to display device 121 , hard copy device 124 , and output device 126 (such as, for example, an audio speaker).
- Memory 104 may include, for example, read only memory (ROM), random access memory (RAM), flash memory, or any combination of the above.
- computer system 100 may be controlled by operating system software which includes a file management system, such as, for example, a disk operating system, which is part of the operating system software.
- the file management system may be stored in non-volatile storage device 107 and may be configured to cause processor 102 to execute the various functions required by the operating system to input and output data and to store data in memory 104 and on non-volatile storage device 107 .
- FIG. 2 a is a block diagram of one embodiment for a natural language translation system 200 .
- natural language translation system 200 includes five modules, supporting databases, and associated grammars to quickly and accurately translate text between source and target languages.
- Input text may be directly input into natural language translation system 200 (for example, as with a person typing sentences into a computer using keyboard 122 ).
- input text to natural language translation system 200 may be the output of another system, such as, for example, output from a speech recognition system (for example, speech input device 125 ), or from an optical character recognition system (not shown).
- a grammar is generally a set of context-free rules that define the valid phrase structures in the source or target language, with each context-free rule associated with one or more statements (the “rule body”) that perform tests and manipulations on the linguistic representations (feature structures).
- NP noun phase
- VP verb phrase
- All rule bodies utilized by the grammars of language translation system 200 are in the form of computer-executable routines produced by defining the grammar in terms of a grammar programming language (GPL) and passing appropriate rule bodies ( 209 , 215 , 219 , and 225 ) through a GPL compiler 240 .
- the output of the GPL compiler 240 may be in the form of directly executable code, or may be in the form of standard computer programming language statements (such as, for example, C, C++, Pascal, or Lisp) which are then input into the corresponding programming language compiler to produce executable code.
- the compiled grammars include a specific function for each context-free rule. The specific function performs all the processing required by the rule and its associated rule body.
- the interfaces between the compiled grammars and the modules enable a single language translation system 200 to perform translation between multiple natural languages, and to perform more than one translation simultaneously.
- a morphological analysis module 206 takes text input 202 and uses a source language dictionary 204 to decompose the words into morphemes by identifying root forms, grammatical categories, thesaurus information, and other lexical features of the words.
- the morphological analysis module 206 builds a “feature structure” for each word.
- Feature structures are well known in the art as linguistic data structures that contain feature-value pairs for strings, symbols, and numbers that appear in a natural language sentence. Each feature of a word is mapped to the appropriate value through a function commonly designated as:
- the Feature Structure 2 may be referred to as a “disjunctive” feature structure as it represents two mutually exclusive feature structures that are valid for the word. It will be appreciated that the grammatical category is not the only feature of these two words and that morphological analysis module 206 outputs full feature structures.
- the example feature structures are simplified for the sake of clarity in explanation and are also frequently represented by a shorthand notation, e.g., [want] or NP[].
- lexical ambiguity reduction module 210 may segment the words in character-based languages that do not utilize spaces through a database of lexical connector feature rules 208 .
- Lexical connector feature rules 208 are created from GPL grammar statements as described above. Each possible combination of adjacent segmented words are assigned a lexical cost.
- Dictionary 204 defines combinations of words (“multiwords”).
- Lexical ambiguity reduction module 210 evaluates each feature structures that contains a part-of-speech (POS) ambiguity, such as the feature structure for the word “wants” which is tagged as both a noun and a verb. The various possible POS tags are assigned a lexical cost. Lexical ambiguity reduction module 210 weighs the cost assigned to each word in the sentence and selects those feature structures that have the lowest cost.
- POS part-of-speech
- syntactic analysis module 216 uses parsing grammar 212 to create a syntax parse tree for the sentence.
- Parsing grammar 212 contains the source language context-free grammar rules in the form of a parsing table and the associated rule bodies in executable code. Each leaf of the syntax parse tree is a feature structure for one of the words in the sentence.
- an intermediate feature structure for each branch (parent) node in the syntax parse tree is built by combining its child nodes as specified in one or more of the context-free grammar rules.
- the rule body for each potentially applicable context-free grammar rule manipulates the various feature structures at the child nodes and determines whether the associated context-free rule could create a valid phrase from the possible combinations.
- a rule body may cause a thesaurus 214 to be queried as part of the manipulation. It will be appreciated that the feature structure that results from applying the context-free grammar rules may be nested (i.e., contain multiple feature structures from each child node).
- Syntactic analysis module 216 may create the syntax parse tree shown in FIG.
- the feature structure for the sentence in the source language is passed to transfer module 222 .
- the feature structure represents the analysis of the source input and may contain a number of nested linguistic representations (referred herein as sub-structures or slots).
- Transfer module 222 uses transfer grammar 218 to match source language slots of the input with source language slots in example database 220 .
- Example database 220 contains feature structure pairs in the source language and a target language. For example, database 220 may contain matching feature structures in English and Japanese.
- Transfer grammar 218 consists of a set of rewrite rules with a context-free component and a GPL rule body. The context-free parts of the rules are used to create a transfer generation tree.
- Transfer module 222 uses the GPL rule bodies within transfer grammar 218 to match the input source sub-structures or slots to the source sub-structures or slots in example database 220 . If a good match is found (in one embodiment, a low overall match cost), transfer module 222 checks if all sub-structures or slots of the input feature structure have found a match. If a match for a sub-structure is not found, the sub-structure is used as input to transfer module 222 . A transfer generation tree of the form shown in FIG. 2 c is used to break the sub-structure into multiple sub-structures. The new input may be a part of the original, source feature structure or a new feature sub-structure that is constructed from sections of different slots.
- Transfer module 222 uses the input feature structure (or sub-structure) in the source language as the starting symbol to build transfer generation tree 260 .
- Root 261 is a symbol-node (s-node) and is labeled with the starting symbol of the feature structure.
- the transfer grammar determines which transfer grammar rules are applicable to the feature structure at the root 261 , and creates child rule-node(s) (r-node) 263 depending from root 261 .
- r-nodes 263 are the rule numbers within transfer grammar 218 that may be validly applied to the input feature structure. Transfer grammar 218 rules added to tree 260 are applied to the s-nodes 265 .
- a child rule-node (r-node) 265 is added to tree 260 . If the application fails, the s-node 261 is tagged as “dead” for sub-sequent removal. Transfer grammar 218 then creates a new s-node 265 for each r-node 263 . Again, the applicable rules are found for each s-node 265 and applied. The process is repeated until all sub-features within the feature structure have been expanded. Transfer generation tree 260 is then pruned to remove any “dead” nodes and corresponding sub-trees. If root 261 is tagged as “dead,” the generation fails.
- the resulting transfer generation tree 260 is used by transfer module 222 to match the feature structure against the example database 220 .
- the feature structures and sub-structures in the target language associated with a match are substituted for the corresponding feature structures and sub-structures matched in the source language.
- Transfer module 222 recursively applies the transfer rules contained within transfer grammar 218 from the top-most transfer rules until all meaningful sub-features or constituents within the input source feature structure are transferred to the target sub-structures.
- the transfer module 222 will consult the thesaurus 214 when required to do so by a transfer rule.
- Transfer module 222 outputs a feature structure in the target language.
- the feature structure for the sentence in the target language is passed to a morphological and syntactical generation module 228 , where it is used as the root node for a syntactical generation tree, an example of which is shown in FIG. 2 d .
- the syntactical generation tree is built in the same fashion as the transfer generation tree, with context-free rules in a generation grammar 224 as its r-nodes 273 .
- the generation grammar 224 copies information to each s-node 275 , 279 .
- the transfer module 226 in which multiple sub-transfers created multiple transfer generation trees, only one syntactical generation tree is created by the morphological and syntactical generation module 228 .
- Any s-node that is not a leaf node 279 is used to generate the next level of r-nodes.
- the current branch of the tree is complete and the morphological and syntactical generation module 228 traverses back up the tree to find the next s-node that is not a leaf node.
- the thesaurus 214 is consulted when necessary during the generation of the tree.
- the transfer generation tree is complete when all the lowest level s-node are leaf nodes.
- the leaf nodes contain output feature structures representing the words in one or more translations of the input sentence.
- the sequence of output feature structures that represents the best sentence is converted into output text 230 by the morphological and syntactical generation module 228 using the dictionary 226 .
- all output feature structures for all sentences may be converted into the output text 230 .
- FIG. 3 is a diagram of one embodiment for lexical ambiguity reduction module 210 of FIG. 2 a .
- lexical ambiguity reduction module 210 comprises tokenizer 306 , segmentation and POS engine 320 and grammar programming language (GPL) compiler 312 . It will be recognized by one skilled in the art that a wide variety of other engines other than that discussed above may be used by the lexical ambiguity reduction module 300 without loss of generality.
- GPL grammar programming language
- tokenizer 306 receives input string 302 comprising a sequence of words and breaks it into individual tokens 308 .
- a token may comprise, for example, a full word, a reduced word, a number, a symbol, or a punctuation character.
- Tokenizer 306 examines the local context of the word or character within the sentence or phrase, or the current character and its immediate neighbors. Tokenizer 306 may use a small set of tokenization rules 304 . In one example of an English language sentence, tokenizer 306 may make a break at the following places with the corresponding effect:
- space character space, return, tab, End-of-Sentence (EOS)
- segmentation and POS engine 320 receives tokens 308 and performs one or more of its assigned functions, such as, for example, segmentation, POS tagging, and multiword processing. Each of the named functions is described in more detail below.
- lexical dictionary 316 comprises lexical entries in the format of feature structures. Each lexical entry stored in lexical dictionary 316 may have a corresponding POS information. In alternate embodiments, a wide variety of other lexical information may be stored in lexical dictionary 316 . In one embodiment, lexical dictionary 316 may also contain a multiword dictionary used in the multiword processing as described below. Alternatively, multiword information may be stored in a separate dictionary.
- Lexical functions 314 represent lexical grammar rules 310 .
- lexical grammar rules 314 result from pre-compiling lexical grammar rules 310 using GPL compiler 312 .
- lexical functions related to tokens 308 may be selected from lexical functions 314 .
- lexical grammar rules related to tokens 308 may be selected from lexical grammar rules 310 and compiled by GPL compiler 312 to generate lexical functions related to tokens 308 .
- Lexical grammar rules 310 may be written in GPL.
- lexical grammar rules 310 comprise Japanese lexical grammar rules.
- lexical grammar rules 310 may comprise various grammar rules of any other language and may be represented by a wide variety of other programming languages or structures.
- the Japanese grammar rules may include rules defining connectivity relation of tokens.
- GPL compiler 312 compiles rules selected from lexical grammar rules 310 to generate lexical functions 314 .
- lexical functions 314 may include calls to feature structure library routines which allow flexibility in developing lexical grammar rules. This flexibility becomes especially important when complex and space consuming rules are involved, such as lexical connector rules defining connectivity relation of tokens.
- segmentation and POS engine 320 may perform POS tagging. Alternatively, POS tagging may be performed simultaneously with the segmentation process. In another embodiment, POS tagging may be performed without performing segmentation (for example, in word-based natural languages). Segmentation and POS engine 320 performs POS tagging by consulting lexical dictionary 316 and assigning all possible POS tags to each segmented word of the input sentence 302 . In one embodiment, segmentation and POS engine 320 searches lexical dictionary 316 for every segmented word of input sentence 302 . As described above, lexical dictionary 316 may comprise lexical entries for words in the format of feature structures. Once the segmented word is found, segmentation and POS engine 320 retrieves all corresponding POS tags contained within the feature structure of this word.
- multiword processing is also performed to define multiword expressions in the input sentence.
- Segmentation and POS engine 320 performs multiword processing by consulting a multiword dictionary which may be included in the lexical dictionary 316 or contained in a separate dictionary. The multiword processing is described in more detail below.
- segmentation and POS engine 320 creates a connection graph comprising a plurality of paths defined by all possible segmentations of input sentence 302 and/or various POS tags assigned to each segmented word in input sentence 302 . Multiword expressions may also be reflected in the connection graph. The content of the connection graph and the process of its creation are explained below. Segmentation and POS engine 320 compares the paths in the connection graph. In one embodiment, the comparison is done using lexical cost file 318 which contains various lexical cost information. The information in lexical cost file 318 may include, for example, lexical costs, unigram costs, bigram costs and connector costs.
- Lexical costs correspond to the probability of observing a certain word as a certain part of speech. For example, the probability of observing word “bank” as a noun may be higher than the probability of observing word “bank” as a verb.
- Unigram cost or POS costs correspond to the probability of observing a particular part of speech, regardless of what the particular word is or what the surrounding parts of speech are. For example, the probability of observing a noun within any sentence may be higher than the probability of observing a determiner. Bigram costs correspond to the probability of observing a sequence of two particular parts of speech together, regardless of what the words are.
- the probability of observing a determiner followed by a noun may be higher than a probability of observing a noun followed by a determiner.
- Connector costs correspond to the probability of observing two particular words in adjacent positions.
- word 1 and word 2 may be created starting from a certain position depending on their lengths.
- word 1 is created by combining six characters
- word 2 is created by combining eight characters, which include the same six characters plus two characters immediately following the six characters.
- Word 3 in our example is a word which ends immediately before word 1 and word 2 start.
- the connector costs may reflect that the probability of observing word 3 in a position adjacent to word 2 may be higher than the probability of observing word 3 adjacent to word 1, or vice versa.
- Lexical cost information may be stored in a database or it may be divided with one portion being stored along with lexical grammar rules 310 and another portion being stored with POS information in lexical dictionary 316 . In alternate embodiments, a variety of means for storing the lexical cost information may be used.
- segmentation and POS engine 320 selects the best paths within the connection graph that have lower costs. The best paths are used to generate output graph 322 which is provided to syntactic analysis module 216 of FIG. 2 a for further processing. Output graph 322 contains the information needed by syntactic analysis module 216 for making an accurate final interpretation of the input sentence.
- the operation of syntactic analysis module 216 is simplified because only pertinent information (i.e., lexical feature structures for best paths as opposed to all possible paths) is passed to syntactic analysis module 216 .
- the present invention may provide an accurate response to an input sentence, without consuming unreasonable amount of memory and processing time.
- FIG. 4 is a flow diagram of one embodiment for reducing lexical ambiguity in a natural language translation system.
- an input stream is passed to lexical ambiguity reduction module 300 of FIG. 3.
- the input stream may be, for example, a full sentence, a reduced sentence, a word, a number, a symbol, or a punctuation character.
- the input stream is broken into tokens. In one embodiment, the input stream is broken into at least two tokens. The number of tokens varies depending upon the language, length and complexity of the input stream, and applicable tokenization rules 304 , as described above.
- connection graph may be created by finding all possible connections between tokens (i.e. performing segmentation of the input stream). The process of segmentation is described in more detail below. Regardless of whether the input stream requires segmentation, POS tagging and/or multiword processing may need to be performed. As described above, POS tagging involves finding all possible POS tags for each word in the input stream by consulting a lexical dictionary.
- Multiword processing involves defining all possible multiword expressions in the input stream using a multiword dictionary.
- the multiword dictionary comprises multiword expressions (“multiwords”) in the format of feature structures.
- multiwords multiword expressions
- Valid feature structures for the combination may include: white ⁇ [ root : white cat : adj ] ( Feature ⁇ ⁇ Structure ⁇ ⁇ 4 ) house ⁇ [ [ root : house cat : noun ] OR [ root : house cat : verb ] ] ( Feature ⁇ ⁇ Structure ⁇ ⁇ 5 )
- An equally valid feature structure for the combination may be: White ⁇ ⁇ House ⁇ [ root : White House cat : proper noun ] ( Feature ⁇ ⁇ Structure ⁇ ⁇ 6 )
- connection graph comprises a set of nodes and a set of arcs.
- a node corresponds to a separator between two words.
- An arc corresponds to a token and connects two nodes.
- An arc may be labeled with a corresponding part of speech tag.
- An example of a connection graph is shown in FIG. 5 a .
- each of the plurality of paths is assigned a cost, as shown in processing block 410 .
- Each arc comprising the path has a cost associated with it.
- each node may also have a cost associated with it.
- these costs may be obtained from lexical cost file 318 and may include, for example, lexical costs, unigram costs, bigram costs, connector costs.
- the cost assigned to each path results from summing all costs defined for every arc and, if applicable, every node in this path. The process of calculating the cost for each path will be described in more detail below.
- At processing block 412 at least one best path is selected from the plurality of paths based upon a corresponding cost. In one embodiment, costs of all possible paths are weighed and those with lower costs are selected to generate an output graph. The selection of paths is described in more detail below.
- the output graph comprising the best paths is provided to syntactic analysis module 216 for further processing. In the examples described, selection of the best paths reduces lexical ambiguity in the input stream before the syntactic analysis module 216 begins its parsing operation, thereby simplifying the parsing process.
- lexical ambiguity reduction module 210 provides syntactic analysis module 216 with all the information it may need for producing an accurate interpretation of the input stream.
- FIG. 5 a illustrates an exemplary connection graph for the input expression “I want to visit the White House.”
- each pair of nodes 2 through 16 are connected by arcs 22 through 42 .
- Arcs 22 through 42 are labeled with corresponding part of speech tags.
- the word “visit” 50 is separated by nodes 8 and 10 . Because the word “visit” 50 may have at least two part of speech tags, such as, for example, a verb and a noun, nodes 8 and 10 are connected by at least two arcs.
- arc 30 corresponds to a verb (“v”) and arc 32 corresponds to a noun (“n”).
- the word “House” 54 is separated by nodes 14 and 16 which are connected by arc 38 , representing a verb tag, and arc 40 , representing a noun tag.
- the word “House” 54 is a part of a multiword “White House” 56 , which is a proper noun, as shown in Feature Structure 6.
- arc 42 is created connecting nodes 12 and 16 to represent the multiword expression with a POS tag of a proper noun.
- Possible combinations of arcs and nodes define a plurality of paths in the connection graph. The number of possible paths may vary depending on how many arcs represent each word in the input stream. If each word in the input stream has only one arc representing it, then the connection graph comprises only one path.
- FIG. 5 b illustrates an exemplary path of one of the twelve possible paths of FIG. 5 a .
- the exemplary path consists of the combination of arcs 22 , 24 , 28 , 30 , 34 and 42 and corresponding nodes.
- FIG. 6 is a flow diagram of one embodiment for segmenting an input stream.
- the segmentation process is used in character-based languages, e.g. the Japanese language, which do not have separators such as spaces between words.
- a task of the segmentation process is to recognize all the possible words (or segments) in the given input stream and find possible connections between adjacent words.
- tokens are received.
- at least two tokens are received.
- lexical functions may be selected from a collection of lexical functions 314 .
- lexical functions 314 result from pre-compiling lexical grammar rules 310 using a GPL compiler.
- Lexical grammar rules 310 may be written in GPL and may define connectivity relation of tokens.
- Lexical functions 314 may call feature structure library routines.
- the output of GPL compiler 312 may be in the form of directly executable code or may be in the form of standard computer programming language statements. Either approach provides a flexible method to develop grammar rules which becomes especially important for rules defining connectivity relation of tokens in character-based languages due to manipulation of large amount of data involved in presentation of these rules.
- segments are created from the tokens based upon the lexical functions and lexical dictionary.
- the created segments define all possible segmentations of the input stream.
- the creation of the segments may include finding various combinations of the tokens and then determining all possible connections between these various combinations. That is, the lexical information retrieved from lexical dictionary 316 may be used to define which tokens may be combined. Based upon all possible combinations, a number of lexical items (segments) may be created, in which every lexical item results from combining one or more tokens of the input stream. Then, lexical dictionary 316 and the lexical functions maw be used to determine which adjacent segments may be connected. The segments that have valid connections define all possible segmentations of the input stream.
- connection graph is generated from these segments.
- the connection graph represents all possible segmentations of the input stream and is subsequently processed by segmentation and POS engine 320 to generate an output graph.
- the time consuming segmentation process may be performed efficiently, thereby improving the overall performance of the translation system.
- FIG. 7 is a flow diagram of one embodiment for reducing lexical ambiguity in an input English sentence.
- tokenization of an input English sentence is performed by breaking the input English sentence into tokens. The number of tokens resulting from tokenizing an English sentence varies depending on the lengths and complexity of the sentence.
- a connection graph is created using the tokens. As described above, the connection graph comprises a set of nodes and a set of arcs.
- each word in the sentence comprises at least one token.
- an arc is added in the connection graph to represent every additional POS tag found. Every arc is labeled with a corresponding POS tag.
- FIG. 5 a example shows all the arcs defined for every word in the input sentence “I want to visit the White House.” The elements of the connection graph are described in more details above.
- multiword expressions are defined by consulting a multiword dictionary. As described above, an arc is added to define each multiword in the sentence. Based upon all possible POS tags and multiwords in the Sentence, a plurality of paths is defined in the connection graph. Each path represents a combination of arcs and nodes in the connection graph. In the example shown in FIG. 5 a , twelve different paths may be defined in the connection graph based on all possible combinations of the arcs and nodes. The FIG. 5 b example illustrates one of the twelve possible paths which consists of the combination of arcs 22 , 24 , 28 , 30 , 34 and 42 and corresponding nodes.
- each path in the connection graph is assigned a cost.
- This cost is a total of overall costs calculated for all the arcs contained in the path.
- the overall cost calculated for all the arcs in the path includes a lexical cost, a POS (or unigram cost) and a bigram cost.
- the lexical cost assigned to arc 24 may be lower than the cost assigned to arc 26 because the word “want” may be used more often as a verb than as a noun.
- the unigram cost or POS cost corresponds to the probability of observing this particular part of speech, regardless of what the word is or what the surrounding parts of speech are.
- the unigram cost assigned to arc 24 may be higher than the unigram cost assigned to arc 26 because verbs in general may be considered to be used more often than nouns.
- the bigram cost corresponds to the probability of observing a sequence of two particular parts of speech, regardless of what the words are.
- the bigram cost is assigned to each pair of connected arcs.
- the bigram cost assigned to the combination of arcs 22 and 24 may be lower than the bigram cost assigned to the combination of arcs 22 and 26 because the sequence of a pronoun and a verb may be more probable than the sequence of a pronoun and a noun.
- the total cost assigned to each path includes lexical costs assigned to each arc in the path, unigram costs assigned to each arc in the graph and bigram costs assigned to each pair of arcs in the graph.
- a path comprises an arc defining a multiword expression (e.g., arc 42 in FIG. 5 a or 5 b )
- the cost for this arc is derived from the multiword entry in the multiword dictionary.
- the n best paths are selected from all the paths in the connection graph. The selection is based upon a cost assigned to each path.
- the number (“n”) of best paths selected may be predefined based upon a variety of factors, such as, for example, a desired level of accuracy, the complexity of information being processed, or time constraints associated with the process.
- the number of best paths may be determined by segmentation and POS engine 320 during operation based upon various factors.
- the number of best paths may be varied depending upon a certain percentage defined to limit costs of selected best paths. For example, this percentage may be set to 10%.
- path 1 has a cost of 10
- path 2 has a cost of 11.8
- path 3 has a cost of 12.2
- path 4 has a cost of 14
- only paths 1 and 2 are selected as best paths because costs of paths 3 and 4 exceed the cost of path 1 for more than 10%.
- a variety of methods for determining the number of best paths may be used. The selected n best paths are then used to generate an output graph 718 as described above.
- FIG. 8 is a flow diagram of one embodiment for reducing lexical ambiguity in an input Japanese sentence.
- tokenization of an input Japanese sentence is performed by breaking the input Japanese sentence into tokens. Because a typical Japanese sentence does not have separators such as spaces between words, each Japanese character in the sentence may correspond to a token.
- the tokens are combined in all possible combinations to define a variety of lexical entries (segments) in the sentence using lexical dictionary 316 .
- FIG. 9 illustrates an exemplary connection of tokens in an input Japanese sentence. Referring to FIG. 9, tokens 50 through 80 are combined in various ways. Combinations of tokens are made to match any entry in lexical dictionary 316 .
- token 50 by itself may have a matching lexical entry in lexical dictionary 316 , or a combination of tokens 50 and 52 may have a matching lexical entry in lexical dictionary 316 . All combinations that have matching entries in lexical dictionary 316 are analyzed to define all the possible lexical entries (segments) in the input sentence.
- the combination of tokens 58 and 60 may define segment 20 .
- the combination of the same tokens 58 and 60 along with a token 62 may result in segment 22 .
- the combination of tokens 58 and 60 may be a part of segment 24 .
- the variety of segments are connected using lexical dictionary 316 and lexical functions to define possible segmentations of the input sentence.
- the lexical functions are associated with the segments being processed and are selected from the entire collection of lexical functions 314 .
- Lexical functions 314 result from compiling lexical grammar rules using GPL compiler 312 .
- Selected lexical functions define connectivity relation between lexical feature structures of the input sentence.
- all possible connections for each lexical feature structure may be defined using features LEX-TO and LEX-FROM assigned to the lexical feature structures of the input sentence.
- features LEX-TO and LEX-FROM may define all possible parts of speech that can be connected to this segment. That is, the feature LEX-FROM may define all parts of speech that may precede this segment and the feature LEX-TO may define all parts of speech that can immediately follow this segment. If any value in LEX-TO TO and LEX-FROM features of adjacent segments matches, then these two segments may be connected.
- lexical feature structure 26 defined by arc 5 has a value “noun-part” in its LEX-TO feature. The same value is contained in a LEX-FROM feature of lexical feature structure 28 defined by arc 7 . Thus, a valid connection can be made between these two adjacent segments.
- Each segment may be connected to more than one preceding segment and to more than one following segment.
- segment 24 may be connected to at least two preceding segments (e.g., segments 30 and 32 ) if any value in its LEX-FROM feature matches with any value in a LEX-TO feature of every preceding segment.
- segment 24 may be connected to at least two following segments (e.g., segments 26 and 34 ) if any value in its LEX-TO feature matches with any value in a LEX-FROM feature of every following segment.
- segments that do not have either preceding or following connections are ignored. The rest of the segments may be used to define all possible segmentations of the input sentence.
- each segment is assigned all POS tags found for this lexical entry in lexical dictionary 316 .
- FIG. 9 shows sample POS tags assigned to arcs 1 trough 11 .
- the segments and corresponding POS tags may be used to create a connection graph.
- the process of creating a connection graph for a Japanese sentence may be the same as the process of creating a connection graph for an English sentence.
- the connection graph comprises a set of nodes and a set of arcs. Each arc corresponds to a POS tag of a segment.
- Various combinations of arcs and nodes define a plurality of paths in the connection graph.
- each path is assigned a cost.
- this cost is a total of all costs calculated for every arc and node contained in the path.
- the cost calculated for every arc in the path may include a lexical cost and a POS (or unigram) cost.
- the segmentation process may involve a connector cost which is assigned to each node in the path.
- the connector cost corresponds to the probability of observing two types of words in adjacent positions. That is, each of all possible connections made between adjacent segments may carry a connector cost associated with this particular connection.
- the cost assigned to each path may include lexical costs assigned to each arc in the paths, unigram costs assigned to each arc in the graph and connector costs assigned to each node in the graph. In alternate embodiments, any other way of calculating a cost for a path may be used.
- n best paths are selected from all the paths within the connection graph. In one embodiment, the selection is based upon a cost assigned to each path. The number of best paths is determined as described above. The selected n best paths are used to generate an output graph which is passed to syntactic analysis module 216 .
- a method and system for reducing lexical ambiguity in an input stream have been described.
- the method breaks the input stream into tokens and creates a connection graph using the tokens. If needed, the method may perform segmentation of the input stream, POS tagging or multiword processing. Results received in any of the above processes are used to define a plurality of paths in the connection graph.
- the method assigns a cost to each of the plurality of paths. Based upon assigned costs, at least one best path is selected from the plurality of paths.
- the method uses the at least one best path to generate an output graph.
- the output graph is passed to a syntactic analysis module to reduce lexical ambiguity.
Abstract
A method and system for reducing lexical ambiguity in an input stream are described. In one embodiment, the input stream is broken into tokens. The tokens are used to create a connection graph comprising a number of paths. Each of the paths is assigned a cost. At least one best path is defined based upon a corresponding cost to generate an output graph. The generated output graph is provided to reduce lexical ambiguity.
Description
- 1. Field of the Invention
- The present invention relates to language translation systems. More particularly, the present invention relates to a method for reducing lexical ambiguity.
- 2. Background Information
- With the continuing growth of multinational business dealings where the global economy brings together business people of all nationalities and with the ease and frequency of today's travel between countries, the demand for a machine-aided interpersonal communication system that provides accurate near real-time language translation, whether in spoken or written form, is a compelling need. This system would relieve users of the need to possess specialized linguistic or translational knowledge.
- A typical language translation system functions by using natural language process. Natural language processing is generally concerned with the attempt to recognize a large pattern or sentence by decomposing it into small subpatterns according to linguistic rules. A natural language processing system uses considerable knowledge about the structure of the language, including what the words are, how words combine to form sentences, what the words mean, and how word meanings contribute to sentence meanings. However, linguistic behavior cannot be completely accounted for without also taking into account another aspect of what makes humans intelligent—their general world knowledge and their reasoning abilities. For example, to answer questions, to participate in a conversation, or to create and understand written language, a person not only must have knowledge about the structure of the language being used, but also must know about the world in general and the conversational setting in particular. Specifically, phonetic and phonological knowledge concerns how words are related to sounds that realize them. Morphological knowledge concerns how words are constructed from more basic units called morphemes. Syntactic knowledge concerns how words can be put together to form correct sentences and determines what structural role each word plays in the sentence and what phrases are subparts of what other phrases. Typical syntactic representations of language are based on the notion of context-free grammars, which represent sentence structure in terms of what phrases are subparts of other phrases. This syntactic information is often presented in a tree form. Semantic knowledge concerns what words mean and how these meanings combine in sentences to form sentence meanings. This is the study of context-independent meaning—the meaning a sentence has regardless of the context in which it is used. The representation of the context-independent meaning of a sentence is called its logical form. The logical form encodes possible word senses and identifies the semantic relationships between the words and phrases.
- Natural language processing systems further include interpretation processes that map from one representation to the other. For instance, the process that maps a sentence to its syntactic structure and logical form is called parsing, and it is performed by a component called a parser. The parser uses knowledge about word and word meaning, the lexicon, and a set of rules defining the legal structures, the grammar, in order to assign a syntactic structure and a logical form to an input sentence.
- Formally, a context-free grammar of a language is a four-tuple comprising nonterminal vocabularies, terminal vocabularies, a finite set of production rules, and a starting symbol for all productions. The nonterminal and terminal vocabularies are disjoint. The set of terminal symbols is called the vocabulary of the language. Pragmatic knowledge concerns how sentences are used in different situations and how use affects the interpretation of the sentence.
- A natural language processor receives an input sentence, lexically separates the words in the sentence, syntactically determines the types of words, semantically understands the words, pragmatically determines the type of response to generate, and generates the response. The natural language processor employs many types of knowledge and stores different types of knowledge in different knowledge structures that separate the knowledge into organized types.
- The complexity of the natural language process is increased due to lexical ambiguity of input sentences. Cases of lexical ambiguity may hinge on the fact that a particularly word has more than one meaning. For example, the word bank can be used to denote either a place where monetary exchange and handling takes place or the land close river, the bank of the river. A word or a small group of words may also have two or more related meanings. That is, the adjective bright may be used as a synonym for “shining” (e.g., “The stars are bright tonight”) or as a synonym for “smart” (e.g., “She must be very bright if she made an “A” on the test”). In the field of spoken language translation, the problem is compounded by words that are not necessarily spelled the same but are pronounced the same and have different meanings. For example, the words night and knight are pronounced exactly the same although they are spelled differently, and they have very different meanings.
- Factors causing the lexical ambiguity vary from one language to another. In character-based languages, e.g. in the Japanese language, extracting information from an input sentence creates a serious problem because Japanese sentences do not have spaces between words. Part-of-speech (POS) tags are another factor causing lexical ambiguity. In many languages, including both word-based and character-based natural languages, one word may have more than one POS tag depending on the context of POS within the sentence. The word table, for example, can be a verb in some contexts (e.g., “He will table the motion”) and a noun in others (e.g., “The table is ready”). The existence of multiword expressions in many languages, including the English language, is yet another factor contributing to lexical ambiguity. That is, depending on the context, a group of words, such as “white house”, can be treated as a multiword expression (e.g., “I want to visit the White House”) or as separate words (e.g., “He lives in a white house across the street”).
- One current approach that deals with lexical ambiguity in a Japanese input sentence involves treating each Japanese character as a word and letting the parser group the characters using the parsing grammar. After the parser defines the words, the parser must try all POS tags found for each word and rule out the impossible tags. As a result, the parsing program is time consuming and requires a large amount of space for its operation. If a long or complicated sentence is involved, such a parser may not be able to perform the parsing at all.
- Another current approach to deal with lexical ambiguity recognizes all the possible words in a Japanese sentence and then finds possible connections between adjacent words. The recognition of all the words is done using a morpheme dictionary. The morpheme dictionary defines Japanese morphemes with the names of POS tags. The connectivity is defined using a connection-pair grammar. The connection-pair grammar defines pairs of sets of morphemes that may occur adjacently in a sentence. Various costs are then applied to the morphemes to compare all possible segmentations of the input sentence. These various costs correspond to the likelihood of observing a word as a certain part of speech and to the likelihood of observing two words in adjacent positions. In this approach, the segmentation that has the lowest corresponding cost is selected from all the possible segmentations of the input sentence for further processing. However, the segmentation selected based upon the lowest costs may not correspond to the correct meaning of the input sentence. Since the syntactic parser is better equipped to recognize the correct meaning of the input sentence, making a selection before the parsing operation may result in loss of pertinent information. Consequently, this approach may lead to inaccurate results in producing a response to an input sentence; especially in producing a response to a longer or more complicated sentence. The techniques currently used to deal with lexical ambiguity in an English sentence have problems similar to those identified above. Unlike Japanese sentences, English sentences do not need to be segmented as the individual words form the segments. However, multiple POS tags of a word present the same problem for English sentences as they do for Japanese sentences. As described above, one approach taken to deal with this problem requires the parser to try all POS tags found for each word and rule out the impossible tags. In this approach, the parsing program is very time consuming and requires a large amount of space for its operation. In addition, this approach may not be able to handle long and complicated sentences.
- Another approach analyzes all POS tags for each word in an English input sentence and finds the most likely POS tag for each word using lexical and statistical probabilities. However, some probabilities may be hard to estimate. No matter how much text is analyzed for the estimation, there will always be a large volume of words that appear only a few times. Thus, relying strictly on probabilities may not result in an accurate interpretation, especially in dealing with a long or complex sentence in which a word's meaning is dependent upon the context of the word within the sentence. As explained earlier, since the syntactic parser is better equipped to recognize the correct meaning of the input sentence, making a selection before the parsing operation may result in loss of pertinent information.
- Therefore, what is required is an efficient way of reducing lexical ambiguity which will provide an accurate interpretation of an input sentence without unreasonably burdening the operation of the syntactic parser.
- A method and system for reducing lexical ambiguity in an input stream are described. In one embodiment, the input stream is broken into tokens. The tokens are used to create a connection graph comprising a number of paths. Each of the paths is assigned a cost. At least one best path is defined based upon a corresponding cost to generate an output graph. The generated output graph is provided to reduce lexical ambiguity.
- The present invention is illustrated by way of example and may be better understood by referring to the following description in conjunction with the accompanying drawings, in which like references indicate similar elements and in which:
- FIG. 1 is a block diagram of one embodiment for an architecture of a computer system;
- FIG. 2a is a block diagram of one embodiment for a natural language translation system;
- FIGS. 2b, 2 c, and 2 dare exemplary diagrams of structures used by the natural language translation system of FIG. 2a;
- FIG. 3 is a diagram of one embodiment for a lexical ambiguity module;
- FIG. 4 is a flow diagram of one embodiment for reducing lexical ambiguity in a natural language translation system:
- FIG. 5aillustrates an exemplary connection graph;
- FIG. 5billustrates an exemplary path in a connection graph;
- FIG. 6 is a flow diagram of one embodiment for segmentation of an input stream;
- FIG. 7 is a flow diagram of one embodiment for reducing lexical ambiguity in an input English expression;
- FIG. 8 is a flow diagram of one embodiment for reducing lexical ambiguity in an input Japanese expression;
- FIG. 9 illustrates an exemplary connection of tokens in an input Japanese sentence.
- A method and system for reducing lexical ambiguity in an input stream are described. In one embodiment, the input stream is broken into tokens. The tokens are used to create a connection graph comprising a number of paths. Each of the paths is assigned a cost. At least one best path is defined based upon a corresponding cost to generate an output graph. The generated output graph is provided to reduce lexical ambiguity.
- In the following detailed description of the present invention, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be apparent to one skilled in the art that the present invention may be practiced without these specific details. In some instances, well-known structures and devices are shown in block diagram form, rather than in detail, in order to avoid obscuring the present invention.
- Some portions of the detailed descriptions that follow are presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is here, and generally, conceived to be a self-consistent sequence of processing blocks leading to a desired result. The processing blocks are those. requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.
- It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise as apparent from the following discussion, it is appreciated that throughout the description, discussions utilizing terms such as “processing” or “computing” or “calculating” or “determining” or “displaying” or the like, refer to the action and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission or display devices.
- The present invention also relates to apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes, or it may comprise a general purpose computer selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a computer readable storage medium, such as, but is not limited to, any type of disk including floppy disks, optical disks, CD-ROMs, and magnetic-optical disks, read-only memories (ROMs), random access memories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, or any type of media suitable for storing electronic instructions, and each coupled to a computer system bus.
- The algorithms and displays presented herein are not inherently related to any particular computer or other apparatus. Various general purpose systems may be used with programs in accordance with the teachings herein, or it may prove convenient to construct more specialized apparatus to perform the required method steps. The required structure for a variety of these systems will appear from the description below. In addition, the present invention is not described with reference to any particular programming language. It will be appreciated that a variety of programming languages may be used to implement the teachings of the invention as described herein.
- Lexical ambiguity is a recognized problem in natural language processing. The problem with lexical ambiguity arises when a natural language processor needs to extract information from an input sentence for the subsequent syntactic parsing. Extracting information becomes problematic in character-based languages which do not have separators such as spaces between words in a sentence. In addition, in many languages, a word may have different part of speech (POS) tags depending on the context of the POS within the sentence. In some languages, certain groups of words can either be treated as multiword expressions or as separate words depending on the context. In one embodiment, the lexical ambiguity reduction module provides a method for reducing lexical ambiguity in an input sentence which increases the segmentation of the input sentence and supports POS tagging and multiword processing. In this embodiment, the lexical ambiguity module provides a graph which is passed to a syntactic analysis module for subsequent processing. In one embodiment, an efficient method of reducing lexical ambiguity is provided which allows the language processing system to produce an accurate interpretation of the input sentence without unreasonably burdening the operation of the syntactic analysis module.
- FIG. 1 is a block diagram of one embodiment for an architecture of a
computer system 100. Referring to FIG. 1,computer system 100 includessystem bus 101 that allows for communication amongprocessor 102,digital signal processor 108,memory 104, andnon-volatile storage device 107.System bus 101 may also receive inputs fromkeyboard 122, pointingdevice 123, and speechsignal input device 125.System bus 101 provides outputs to displaydevice 121,hard copy device 124, and output device 126 (such as, for example, an audio speaker).Memory 104 may include, for example, read only memory (ROM), random access memory (RAM), flash memory, or any combination of the above. - It will be appreciated that
computer system 100 may be controlled by operating system software which includes a file management system, such as, for example, a disk operating system, which is part of the operating system software. The file management system may be stored innon-volatile storage device 107 and may be configured to causeprocessor 102 to execute the various functions required by the operating system to input and output data and to store data inmemory 104 and onnon-volatile storage device 107. - FIG. 2ais a block diagram of one embodiment for a natural
language translation system 200. Referring to FIG. 2a, naturallanguage translation system 200 includes five modules, supporting databases, and associated grammars to quickly and accurately translate text between source and target languages. Input text may be directly input into natural language translation system 200 (for example, as with a person typing sentences into a computer using keyboard 122). Alternatively, input text to naturallanguage translation system 200 may be the output of another system, such as, for example, output from a speech recognition system (for example, speech input device 125), or from an optical character recognition system (not shown). - An English sentence “He wants to go to the White House” is used throughout this section as example text input to describe the functioning of the
system 200. The individual units in a sentence are referred to herein as “words” but the naturallanguage translation system 200 is not limited to only word-based natural languages, having equal applicability to translation of character-based languages as well. Except where the differences in processing word-based and character-based languages are specified, the term “word” is intended to encompass both words and characters. - In the following description, a grammar is generally a set of context-free rules that define the valid phrase structures in the source or target language, with each context-free rule associated with one or more statements (the “rule body”) that perform tests and manipulations on the linguistic representations (feature structures). Thus, an English sentence may be combined from a noun phase (NP) and a verb phrase (VP), but the subject and verb forms must agree, e.g., “He want to go to the White House” is a valid phrase structure but an improper English sentence. All rule bodies utilized by the grammars of
language translation system 200 are in the form of computer-executable routines produced by defining the grammar in terms of a grammar programming language (GPL) and passing appropriate rule bodies (209, 215,219, and 225) through aGPL compiler 240. The output of theGPL compiler 240 may be in the form of directly executable code, or may be in the form of standard computer programming language statements (such as, for example, C, C++, Pascal, or Lisp) which are then input into the corresponding programming language compiler to produce executable code. In either case, the compiled grammars include a specific function for each context-free rule. The specific function performs all the processing required by the rule and its associated rule body. Furthermore, the interfaces between the compiled grammars and the modules enable a singlelanguage translation system 200 to perform translation between multiple natural languages, and to perform more than one translation simultaneously. - A morphological analysis module206 takes
text input 202 and uses asource language dictionary 204 to decompose the words into morphemes by identifying root forms, grammatical categories, thesaurus information, and other lexical features of the words. The morphological analysis module 206 builds a “feature structure” for each word. Feature structures are well known in the art as linguistic data structures that contain feature-value pairs for strings, symbols, and numbers that appear in a natural language sentence. Each feature of a word is mapped to the appropriate value through a function commonly designated as: -
- The
Feature Structure 2 may be referred to as a “disjunctive” feature structure as it represents two mutually exclusive feature structures that are valid for the word. It will be appreciated that the grammatical category is not the only feature of these two words and that morphological analysis module 206 outputs full feature structures. The example feature structures are simplified for the sake of clarity in explanation and are also frequently represented by a shorthand notation, e.g., [want] or NP[]. - The feature structures built by morphological analysis module206 are input into lexical
ambiguity reduction module 210. In one embodiment, lexicalambiguity reduction module 210 may segment the words in character-based languages that do not utilize spaces through a database of lexical connector feature rules 208. Lexicalconnector feature rules 208 are created from GPL grammar statements as described above. Each possible combination of adjacent segmented words are assigned a lexical cost.Dictionary 204 defines combinations of words (“multiwords”). Lexicalambiguity reduction module 210 evaluates each feature structures that contains a part-of-speech (POS) ambiguity, such as the feature structure for the word “wants” which is tagged as both a noun and a verb. The various possible POS tags are assigned a lexical cost. Lexicalambiguity reduction module 210 weighs the cost assigned to each word in the sentence and selects those feature structures that have the lowest cost. - The feature structures chosen for the words by lexical
ambiguity reduction module 210 are passed tosyntactic analysis module 216.Syntactic analysis module 216 combines the chosen feature structures into a feature structure that represents the content of the input sentence. In one embodiment,syntactic analysis module 216uses parsing grammar 212 to create a syntax parse tree for the sentence. Parsinggrammar 212 contains the source language context-free grammar rules in the form of a parsing table and the associated rule bodies in executable code. Each leaf of the syntax parse tree is a feature structure for one of the words in the sentence. Once the leaves are created, an intermediate feature structure for each branch (parent) node in the syntax parse tree is built by combining its child nodes as specified in one or more of the context-free grammar rules. The rule body for each potentially applicable context-free grammar rule manipulates the various feature structures at the child nodes and determines whether the associated context-free rule could create a valid phrase from the possible combinations. A rule body may cause athesaurus 214 to be queried as part of the manipulation. It will be appreciated that the feature structure that results from applying the context-free grammar rules may be nested (i.e., contain multiple feature structures from each child node).Syntactic analysis module 216 may create the syntax parse tree shown in FIG. 2b for the example sentence from its constituent feature structures, with the following feature structure at the top (root) of the syntax parse tree to represent the full sentence: - It will be appreciated that both the syntax parse
tree 250 and theFeature Structure 3 are not exact representations but are simplified for purposes of ease in explanation. - The feature structure for the sentence in the source language is passed to transfer
module 222. The feature structure represents the analysis of the source input and may contain a number of nested linguistic representations (referred herein as sub-structures or slots).Transfer module 222 usestransfer grammar 218 to match source language slots of the input with source language slots inexample database 220.Example database 220 contains feature structure pairs in the source language and a target language. For example,database 220 may contain matching feature structures in English and Japanese.Transfer grammar 218 consists of a set of rewrite rules with a context-free component and a GPL rule body. The context-free parts of the rules are used to create a transfer generation tree. -
Transfer module 222 uses the GPL rule bodies withintransfer grammar 218 to match the input source sub-structures or slots to the source sub-structures or slots inexample database 220. If a good match is found (in one embodiment, a low overall match cost),transfer module 222 checks if all sub-structures or slots of the input feature structure have found a match. If a match for a sub-structure is not found, the sub-structure is used as input to transfermodule 222. A transfer generation tree of the form shown in FIG. 2c is used to break the sub-structure into multiple sub-structures. The new input may be a part of the original, source feature structure or a new feature sub-structure that is constructed from sections of different slots. -
Transfer module 222 uses the input feature structure (or sub-structure) in the source language as the starting symbol to buildtransfer generation tree 260.Root 261 is a symbol-node (s-node) and is labeled with the starting symbol of the feature structure. The transfer grammar determines which transfer grammar rules are applicable to the feature structure at theroot 261, and creates child rule-node(s) (r-node) 263 depending fromroot 261. In one embodiment, r-nodes 263 are the rule numbers withintransfer grammar 218 that may be validly applied to the input feature structure.Transfer grammar 218 rules added totree 260 are applied to the s-nodes 265. If the application of each rule succeeds, a child rule-node (r-node) 265 is added totree 260. If the application fails, the s-node 261 is tagged as “dead” for sub-sequent removal.Transfer grammar 218 then creates a new s-node 265 for each r-node 263. Again, the applicable rules are found for each s-node 265 and applied. The process is repeated until all sub-features within the feature structure have been expanded.Transfer generation tree 260 is then pruned to remove any “dead” nodes and corresponding sub-trees. Ifroot 261 is tagged as “dead,” the generation fails. Otherwise, the resultingtransfer generation tree 260 is used bytransfer module 222 to match the feature structure against theexample database 220. The feature structures and sub-structures in the target language associated with a match are substituted for the corresponding feature structures and sub-structures matched in the source language.Transfer module 222 recursively applies the transfer rules contained withintransfer grammar 218 from the top-most transfer rules until all meaningful sub-features or constituents within the input source feature structure are transferred to the target sub-structures. Thetransfer module 222 will consult thethesaurus 214 when required to do so by a transfer rule.Transfer module 222 outputs a feature structure in the target language. - The feature structure for the sentence in the target language is passed to a morphological and
syntactical generation module 228, where it is used as the root node for a syntactical generation tree, an example of which is shown in FIG. 2d. The syntactical generation tree is built in the same fashion as the transfer generation tree, with context-free rules in ageneration grammar 224 as its r-nodes 273. Thegeneration grammar 224 copies information to each s-node transfer module 226, in which multiple sub-transfers created multiple transfer generation trees, only one syntactical generation tree is created by the morphological andsyntactical generation module 228. Any s-node that is not aleaf node 279, i.e., associated with a feature structure for a word, is used to generate the next level of r-nodes. When all child s-nodes under an r-node are leaf nodes, the current branch of the tree is complete and the morphological andsyntactical generation module 228 traverses back up the tree to find the next s-node that is not a leaf node. Thethesaurus 214 is consulted when necessary during the generation of the tree. The transfer generation tree is complete when all the lowest level s-node are leaf nodes. - When the syntactical generation tree is complete, the leaf nodes contain output feature structures representing the words in one or more translations of the input sentence. The sequence of output feature structures that represents the best sentence is converted into output text230 by the morphological and
syntactical generation module 228 using thedictionary 226. Alternatively, all output feature structures for all sentences may be converted into the output text 230. - Lexical
ambiguity reduction module 210 of FIG. 2A will now be described in more detail. FIG. 3 is a diagram of one embodiment for lexicalambiguity reduction module 210 of FIG. 2a. Referring to FIG. 3, lexicalambiguity reduction module 210 comprisestokenizer 306, segmentation andPOS engine 320 and grammar programming language (GPL)compiler 312. It will be recognized by one skilled in the art that a wide variety of other engines other than that discussed above may be used by the lexical ambiguity reduction module 300 without loss of generality. - In one embodiment,
tokenizer 306 receivesinput string 302 comprising a sequence of words and breaks it intoindividual tokens 308. A token may comprise, for example, a full word, a reduced word, a number, a symbol, or a punctuation character. In a Japanese sentence, in which there are no spaces between words, each Japanese character may correspond to a token.Tokenizer 306 examines the local context of the word or character within the sentence or phrase, or the current character and its immediate neighbors.Tokenizer 306 may use a small set of tokenization rules 304. In one example of an English language sentence,tokenizer 306 may make a break at the following places with the corresponding effect: - space character (space, return, tab, End-of-Sentence (EOS));
- apostrophe+space character (“Doris'”→“Doris” “'”);
- apostrophe+“s” (“Peter's”→“Peter” “'s”);
- apostrophe+“re” (“they're”→“they” “'re”);
- apostrophe+“d” (“Peter'd”→“Peter” “'d”);
- apostrophe+“ve” (“Peter've”→“Peter” “'ve”);
- apostrophe+“ll” (“Peter'll”→“Peter” “'ll”);
- period+EOS (“Peter likes fish.”→“Peter” “likes” “fish” “.”);
- question mark (“Does Peter like fish?”→“does” “Peter” “like” “fish” “?”);
- exclamation mark (“Fish!”→“fish” “!”);
- comma (except between numbers) (“apples, oranges and bananas”→“apples” “,” “oranges” “and” “bananas”);
- dollar sign (“$30”→“$” “30”);
- percent sign (“30%”→“30” “%”);
- plus sign (“+80 ”→“+” “80”);
- minus sign (only when followed by a number) (“−3 ”→“−” “3”);
- semicolon (“fruits; apples, oranges and bananas”→“fruits” “;” “apples” “,” “oranges” “and” “bananas”);
- colon (except between numbers).
- In one embodiment, segmentation and
POS engine 320 receivestokens 308 and performs one or more of its assigned functions, such as, for example, segmentation, POS tagging, and multiword processing. Each of the named functions is described in more detail below. - During segmentation, segmentation and
POS engine 320 makes possible connections between tokens by consultinglexical dictionary 316 andlexical functions 314. In one embodiment,lexical dictionary 316 comprises lexical entries in the format of feature structures. Each lexical entry stored inlexical dictionary 316 may have a corresponding POS information. In alternate embodiments, a wide variety of other lexical information may be stored inlexical dictionary 316. In one embodiment,lexical dictionary 316 may also contain a multiword dictionary used in the multiword processing as described below. Alternatively, multiword information may be stored in a separate dictionary. -
Lexical functions 314 represent lexical grammar rules 310. In one embodiment,lexical grammar rules 314 result from pre-compilinglexical grammar rules 310 usingGPL compiler 312. In this embodiment, lexical functions related totokens 308 may be selected fromlexical functions 314. Alternatively, lexical grammar rules related totokens 308 may be selected fromlexical grammar rules 310 and compiled byGPL compiler 312 to generate lexical functions related totokens 308. - Lexical grammar rules310 may be written in GPL. In one embodiment,
lexical grammar rules 310 comprise Japanese lexical grammar rules. In an alternate embodiment,lexical grammar rules 310 may comprise various grammar rules of any other language and may be represented by a wide variety of other programming languages or structures. The Japanese grammar rules may include rules defining connectivity relation of tokens. - In one embodiment,
GPL compiler 312 compiles rules selected fromlexical grammar rules 310 to generatelexical functions 314. As described above,lexical functions 314 may include calls to feature structure library routines which allow flexibility in developing lexical grammar rules. This flexibility becomes especially important when complex and space consuming rules are involved, such as lexical connector rules defining connectivity relation of tokens. - After defining all possible connections of
tokens 308, segmentation andPOS engine 320 may perform POS tagging. Alternatively, POS tagging may be performed simultaneously with the segmentation process. In another embodiment, POS tagging may be performed without performing segmentation (for example, in word-based natural languages). Segmentation andPOS engine 320 performs POS tagging by consultinglexical dictionary 316 and assigning all possible POS tags to each segmented word of theinput sentence 302. In one embodiment, segmentation andPOS engine 320 searcheslexical dictionary 316 for every segmented word ofinput sentence 302. As described above,lexical dictionary 316 may comprise lexical entries for words in the format of feature structures. Once the segmented word is found, segmentation andPOS engine 320 retrieves all corresponding POS tags contained within the feature structure of this word. - In one embodiment, multiword processing is also performed to define multiword expressions in the input sentence. Segmentation and
POS engine 320 performs multiword processing by consulting a multiword dictionary which may be included in thelexical dictionary 316 or contained in a separate dictionary. The multiword processing is described in more detail below. - In one embodiment, segmentation and
POS engine 320 creates a connection graph comprising a plurality of paths defined by all possible segmentations ofinput sentence 302 and/or various POS tags assigned to each segmented word ininput sentence 302. Multiword expressions may also be reflected in the connection graph. The content of the connection graph and the process of its creation are explained below. Segmentation andPOS engine 320 compares the paths in the connection graph. In one embodiment, the comparison is done usinglexical cost file 318 which contains various lexical cost information. The information inlexical cost file 318 may include, for example, lexical costs, unigram costs, bigram costs and connector costs. - Lexical costs correspond to the probability of observing a certain word as a certain part of speech. For example, the probability of observing word “bank” as a noun may be higher than the probability of observing word “bank” as a verb. Unigram cost or POS costs correspond to the probability of observing a particular part of speech, regardless of what the particular word is or what the surrounding parts of speech are. For example, the probability of observing a noun within any sentence may be higher than the probability of observing a determiner. Bigram costs correspond to the probability of observing a sequence of two particular parts of speech together, regardless of what the words are. For example, the probability of observing a determiner followed by a noun may be higher than a probability of observing a noun followed by a determiner. Connector costs correspond to the probability of observing two particular words in adjacent positions. Consider a Japanese sentence, in which two different words,
word 1 andword 2, may be created starting from a certain position depending on their lengths. Let's say thatword 1 is created by combining six characters andword 2 is created by combining eight characters, which include the same six characters plus two characters immediately following the six characters.Word 3 in our example is a word which ends immediately beforeword 1 andword 2 start. In our example, the connector costs may reflect that the probability of observingword 3 in a position adjacent toword 2 may be higher than the probability of observingword 3 adjacent toword 1, or vice versa. - Lexical cost information may be stored in a database or it may be divided with one portion being stored along with
lexical grammar rules 310 and another portion being stored with POS information inlexical dictionary 316. In alternate embodiments, a variety of means for storing the lexical cost information may be used. - Based on the costs assigned to each path, segmentation and
POS engine 320 selects the best paths within the connection graph that have lower costs. The best paths are used to generateoutput graph 322 which is provided tosyntactic analysis module 216 of FIG. 2a for further processing.Output graph 322 contains the information needed bysyntactic analysis module 216 for making an accurate final interpretation of the input sentence. In addition, the operation ofsyntactic analysis module 216 is simplified because only pertinent information (i.e., lexical feature structures for best paths as opposed to all possible paths) is passed tosyntactic analysis module 216. Thus, the present invention may provide an accurate response to an input sentence, without consuming unreasonable amount of memory and processing time. - FIG. 4 is a flow diagram of one embodiment for reducing lexical ambiguity in a natural language translation system. Initially, at
processing block 404, an input stream is passed to lexical ambiguity reduction module 300 of FIG. 3. The input stream may be, for example, a full sentence, a reduced sentence, a word, a number, a symbol, or a punctuation character. Atprocessing block 406, the input stream is broken into tokens. In one embodiment, the input stream is broken into at least two tokens. The number of tokens varies depending upon the language, length and complexity of the input stream, andapplicable tokenization rules 304, as described above. - At
processing block 408, the tokens are used to create a connection graph. The connection graph may be created by finding all possible connections between tokens (i.e. performing segmentation of the input stream). The process of segmentation is described in more detail below. Regardless of whether the input stream requires segmentation, POS tagging and/or multiword processing may need to be performed. As described above, POS tagging involves finding all possible POS tags for each word in the input stream by consulting a lexical dictionary. - Multiword processing involves defining all possible multiword expressions in the input stream using a multiword dictionary. The multiword dictionary comprises multiword expressions (“multiwords”) in the format of feature structures. Consider the words “White House” in the sentence “I want to visit the White House.” Valid feature structures for the combination may include:
-
- If Feature Structure 4 and
Feature Structure 5 are found together in the multiword dictionary, then the combination “White House” is defined as a multiword andFeature Structure 6 is retrieved. - Referring again to processing block408 of FIG. 4, the connection graph comprises a set of nodes and a set of arcs. A node corresponds to a separator between two words. An arc corresponds to a token and connects two nodes. An arc may be labeled with a corresponding part of speech tag. An example of a connection graph is shown in FIG. 5a. After the connection graph with the plurality of paths is created, each of the plurality of paths is assigned a cost, as shown in
processing block 410. Each arc comprising the path has a cost associated with it. When processing character-based languages, e.g. the Japanese language, each node may also have a cost associated with it. As described above, these costs may be obtained fromlexical cost file 318 and may include, for example, lexical costs, unigram costs, bigram costs, connector costs. In one embodiment, the cost assigned to each path results from summing all costs defined for every arc and, if applicable, every node in this path. The process of calculating the cost for each path will be described in more detail below. - At
processing block 412, at least one best path is selected from the plurality of paths based upon a corresponding cost. In one embodiment, costs of all possible paths are weighed and those with lower costs are selected to generate an output graph. The selection of paths is described in more detail below. Atprocessing block 414, the output graph comprising the best paths is provided tosyntactic analysis module 216 for further processing. In the examples described, selection of the best paths reduces lexical ambiguity in the input stream before thesyntactic analysis module 216 begins its parsing operation, thereby simplifying the parsing process. In one embodiment, lexicalambiguity reduction module 210 providessyntactic analysis module 216 with all the information it may need for producing an accurate interpretation of the input stream. - FIG. 5a illustrates an exemplary connection graph for the input expression “I want to visit the White House.” Specifically, each pair of
nodes 2 through 16 are connected byarcs 22 through 42.Arcs 22 through 42 are labeled with corresponding part of speech tags. For example, the word “visit” 50 is separated bynodes nodes arc 30 corresponds to a verb (“v”) andarc 32 corresponds to a noun (“n”). The word “House” 54 is separated bynodes arc 38, representing a verb tag, andarc 40, representing a noun tag. In addition, the word “House” 54 is a part of a multiword “White House” 56, which is a proper noun, as shown inFeature Structure 6. As a result,arc 42 is created connectingnodes arcs - FIG. 6 is a flow diagram of one embodiment for segmenting an input stream. The segmentation process is used in character-based languages, e.g. the Japanese language, which do not have separators such as spaces between words. A task of the segmentation process is to recognize all the possible words (or segments) in the given input stream and find possible connections between adjacent words. Initially at
processing block 504, tokens are received. In one embodiment, at least two tokens are received. Atprocessing block 508, lexical functions may be selected from a collection oflexical functions 314. In one embodiment,lexical functions 314 result from pre-compilinglexical grammar rules 310 using a GPL compiler. Lexical grammar rules 310 may be written in GPL and may define connectivity relation of tokens.Lexical functions 314 may call feature structure library routines. As described above, the output ofGPL compiler 312 may be in the form of directly executable code or may be in the form of standard computer programming language statements. Either approach provides a flexible method to develop grammar rules which becomes especially important for rules defining connectivity relation of tokens in character-based languages due to manipulation of large amount of data involved in presentation of these rules. - At
processing block 512, segments are created from the tokens based upon the lexical functions and lexical dictionary. The created segments define all possible segmentations of the input stream. The creation of the segments may include finding various combinations of the tokens and then determining all possible connections between these various combinations. That is, the lexical information retrieved fromlexical dictionary 316 may be used to define which tokens may be combined. Based upon all possible combinations, a number of lexical items (segments) may be created, in which every lexical item results from combining one or more tokens of the input stream. Then,lexical dictionary 316 and the lexical functions maw be used to determine which adjacent segments may be connected. The segments that have valid connections define all possible segmentations of the input stream. - At
processing block 514, a connection graph is generated from these segments. The connection graph represents all possible segmentations of the input stream and is subsequently processed by segmentation andPOS engine 320 to generate an output graph. In one embodiment, the time consuming segmentation process may be performed efficiently, thereby improving the overall performance of the translation system. - FIG. 7 is a flow diagram of one embodiment for reducing lexical ambiguity in an input English sentence. At
processing block 704, tokenization of an input English sentence is performed by breaking the input English sentence into tokens. The number of tokens resulting from tokenizing an English sentence varies depending on the lengths and complexity of the sentence. Atprocessing block 706, a connection graph is created using the tokens. As described above, the connection graph comprises a set of nodes and a set of arcs. - At
processing block 708, all possible POS tags are defined for each word in the sentence by consultinglexical dictionary 316. In one embodiment, each word in the sentence comprises at least one token. When more than one POS tag is found in the lexical dictionary for a word, an arc is added in the connection graph to represent every additional POS tag found. Every arc is labeled with a corresponding POS tag. The FIG. 5a example shows all the arcs defined for every word in the input sentence “I want to visit the White House.” The elements of the connection graph are described in more details above. - Referring to FIG. 7, at
processing block 710, multiword expressions are defined by consulting a multiword dictionary. As described above, an arc is added to define each multiword in the sentence. Based upon all possible POS tags and multiwords in the Sentence, a plurality of paths is defined in the connection graph. Each path represents a combination of arcs and nodes in the connection graph. In the example shown in FIG. 5a, twelve different paths may be defined in the connection graph based on all possible combinations of the arcs and nodes. The FIG. 5b example illustrates one of the twelve possible paths which consists of the combination ofarcs - Referring to FIG. 7, at
processing block 714, each path in the connection graph is assigned a cost. This cost is a total of overall costs calculated for all the arcs contained in the path. In one embodiment, the overall cost calculated for all the arcs in the path includes a lexical cost, a POS (or unigram cost) and a bigram cost. In the example shown in FIG. 5a, the lexical cost assigned toarc 24 may be lower than the cost assigned toarc 26 because the word “want” may be used more often as a verb than as a noun. The unigram cost or POS cost corresponds to the probability of observing this particular part of speech, regardless of what the word is or what the surrounding parts of speech are. For example, the unigram cost assigned toarc 24 may be higher than the unigram cost assigned toarc 26 because verbs in general may be considered to be used more often than nouns. The bigram cost corresponds to the probability of observing a sequence of two particular parts of speech, regardless of what the words are. The bigram cost is assigned to each pair of connected arcs. For example, the bigram cost assigned to the combination ofarcs arcs arc 42 in FIG. 5a or 5 b), the cost for this arc is derived from the multiword entry in the multiword dictionary. - Referring to FIG. 7, at
processing block 716, the n best paths are selected from all the paths in the connection graph. The selection is based upon a cost assigned to each path. The number (“n”) of best paths selected may be predefined based upon a variety of factors, such as, for example, a desired level of accuracy, the complexity of information being processed, or time constraints associated with the process. In an alternate embodiment, the number of best paths may be determined by segmentation andPOS engine 320 during operation based upon various factors. In another embodiment, the number of best paths may be varied depending upon a certain percentage defined to limit costs of selected best paths. For example, this percentage may be set to 10%. If the lowest among the costs assigned to paths in the connection graph equals to 20, then only the paths with costs not exceeding the lowest costs for more than ten percent may be selected as best paths. For example, ifpath 1 has a cost of 10,path 2 has a cost of 11.8,path 3 has a cost of 12.2, and path 4 has a cost of 14, onlypaths paths 3 and 4 exceed the cost ofpath 1 for more than 10%. In alternate embodiments, a variety of methods for determining the number of best paths may be used. The selected n best paths are then used to generate anoutput graph 718 as described above. - FIG. 8 is a flow diagram of one embodiment for reducing lexical ambiguity in an input Japanese sentence. At
processing block 804, tokenization of an input Japanese sentence is performed by breaking the input Japanese sentence into tokens. Because a typical Japanese sentence does not have separators such as spaces between words, each Japanese character in the sentence may correspond to a token. Atprocessing block 806, the tokens are combined in all possible combinations to define a variety of lexical entries (segments) in the sentence usinglexical dictionary 316. FIG. 9 illustrates an exemplary connection of tokens in an input Japanese sentence. Referring to FIG. 9,tokens 50 through 80 are combined in various ways. Combinations of tokens are made to match any entry inlexical dictionary 316. For example, token 50 by itself may have a matching lexical entry inlexical dictionary 316, or a combination oftokens lexical dictionary 316. All combinations that have matching entries inlexical dictionary 316 are analyzed to define all the possible lexical entries (segments) in the input sentence. For example, the combination oftokens segment 20. In addition, the combination of thesame tokens segment 22. Furthermore, the combination oftokens segment 24. - Referring to FIG. 8, at
processing block 808, the variety of segments are connected usinglexical dictionary 316 and lexical functions to define possible segmentations of the input sentence. As described above, in one embodiment, the lexical functions are associated with the segments being processed and are selected from the entire collection oflexical functions 314.Lexical functions 314 result from compiling lexical grammar rules usingGPL compiler 312. Selected lexical functions define connectivity relation between lexical feature structures of the input sentence. In one embodiment, based uponlexical functions 314 andlexical dictionary 316, all possible connections for each lexical feature structure may be defined using features LEX-TO and LEX-FROM assigned to the lexical feature structures of the input sentence. For every lexical feature structure, features LEX-TO and LEX-FROM may define all possible parts of speech that can be connected to this segment. That is, the feature LEX-FROM may define all parts of speech that may precede this segment and the feature LEX-TO may define all parts of speech that can immediately follow this segment. If any value in LEX-TO TO and LEX-FROM features of adjacent segments matches, then these two segments may be connected. As shown in FIG. 9,lexical feature structure 26 defined byarc 5 has a value “noun-part” in its LEX-TO feature. The same value is contained in a LEX-FROM feature oflexical feature structure 28 defined byarc 7. Thus, a valid connection can be made between these two adjacent segments. Each segment may be connected to more than one preceding segment and to more than one following segment. For example,segment 24 may be connected to at least two preceding segments (e.g.,segments 30 and 32) if any value in its LEX-FROM feature matches with any value in a LEX-TO feature of every preceding segment. In addition,segment 24 may be connected to at least two following segments (e.g.,segments 26 and 34) if any value in its LEX-TO feature matches with any value in a LEX-FROM feature of every following segment. In one embodiment, segments that do not have either preceding or following connections are ignored. The rest of the segments may be used to define all possible segmentations of the input sentence. - Referring to FIG. 8, at
processing block 810, each segment is assigned all POS tags found for this lexical entry inlexical dictionary 316. FIG. 9 shows sample POS tags assigned toarcs 1trough 11. - Referring to FIG. 8, at
processing block 812, the segments and corresponding POS tags may be used to create a connection graph. In one embodiment, the process of creating a connection graph for a Japanese sentence may be the same as the process of creating a connection graph for an English sentence. As described above, the connection graph comprises a set of nodes and a set of arcs. Each arc corresponds to a POS tag of a segment. Various combinations of arcs and nodes define a plurality of paths in the connection graph. - At
processing block 814, each path is assigned a cost. As described above, this cost is a total of all costs calculated for every arc and node contained in the path. In a Japanese sentence, the cost calculated for every arc in the path may include a lexical cost and a POS (or unigram) cost. In addition, the segmentation process may involve a connector cost which is assigned to each node in the path. The connector cost corresponds to the probability of observing two types of words in adjacent positions. That is, each of all possible connections made between adjacent segments may carry a connector cost associated with this particular connection. Thus, in one embodiment, the cost assigned to each path may include lexical costs assigned to each arc in the paths, unigram costs assigned to each arc in the graph and connector costs assigned to each node in the graph. In alternate embodiments, any other way of calculating a cost for a path may be used. - At
processing block 816, n best paths are selected from all the paths within the connection graph. In one embodiment, the selection is based upon a cost assigned to each path. The number of best paths is determined as described above. The selected n best paths are used to generate an output graph which is passed tosyntactic analysis module 216. - A method and system for reducing lexical ambiguity in an input stream have been described. The method breaks the input stream into tokens and creates a connection graph using the tokens. If needed, the method may perform segmentation of the input stream, POS tagging or multiword processing. Results received in any of the above processes are used to define a plurality of paths in the connection graph. The method assigns a cost to each of the plurality of paths. Based upon assigned costs, at least one best path is selected from the plurality of paths. The method uses the at least one best path to generate an output graph. The output graph is passed to a syntactic analysis module to reduce lexical ambiguity. With the present invention, an efficient way of reducing lexical ambiguity is provided which produces an accurate interpretation of the input stream without unreasonably burdening the operation of the syntactic analysis module.
- Several variations in the implementation of the method for reducing lexical ambiguity have been described. The specific arrangements and methods described here are illustrative of the principles of this invention. Numerous modifications in form and detail may be made by those skilled in the art without departing from the true spirit and scope of the invention. Although this invention has been shown in relation to a particular embodiment, it should not be considered so limited. Rather it is limited only by the appended claims.
Claims (49)
1. A method for reducing lexical ambiguity in an input stream, comprising:
breaking the input stream into at least two tokens;
creating a connection graph using the at least two tokens, the connection graph comprising a plurality of paths;
assigning a cost to each of the plurality of paths;
defining at least one best path based upon a corresponding cost to generate an output graph; and
providing the output graph to a syntactic analysis module to reduce lexical ambiguity.
2. The method of claim 1 wherein a number of the at least one best path is either predefined or determined programmatically.
3. The method of claim 1 wherein creating a connection graph using the at least two tokens comprises:
compiling lexical grammar rules to generate lexical functions, the lexical grammar rules being written in a grammar programming language;
creating a plurality of segments from the at least two tokens based upon lexical information and the lexical functions, and
defining the plurality of paths using the plurality of segments.
4. The method of claim 1 wherein creating a connection graph using the at least two tokens comprises assigning at least one part of speech tag to at least one of the at least two tokens using lexical information.
5. The method of claim 1 wherein creating a connection graph using the at least two tokens comprises recognizing a multiword expression in the input stream using multiword information.
6. The method of claim 1 wherein the connection graph comprises a set of nodes and a set of arcs.
7. The method of claim 6 wherein each of the plurality of paths comprises a combination of nodes and arcs.
8. The method of claim 1 wherein the cost comprises lexical cost, unigram cost, bigram cost and connector cost.
9. A method for providing segmentation of an input stream having at least two tokens, comprising:
creating a plurality of segments from the at least two tokens based upon lexical information and lexical functions; and
generating a connection graph using the plurality of segments.
10. The method of claim 9 further comprising compiling lexical grammar rules to generate the lexical functions, the lexical grammar rules being written in a grammar programming language.
11. The method of claim 10 wherein the lexical grammar rules define connectivity relation of tokens.
12. The method of claim 9 further comprising assigning at least one part of speech tag to at least one segment using a lexical dictionary.
13. The method of claim 12 further comprising:
defining a plurality of paths in the connection graph based upon part of speech tags and the segments;
assigning a cost to each of the plurality of paths; and
determining at least one best path based upon a corresponding cost to generate an output graph.
14. An apparatus for reducing lexical ambiguity in an input stream, comprising:
means for breaking the input stream into at least two tokens;
means for creating a connection graph using the at least two tokens, the connection graph comprising a plurality of paths;
means for assigning a cost to each of the plurality of paths;
means for defining at least one best path based upon a corresponding cost to generate an output graph; and
means for providing the output graph to a syntactic analysis module to reduce lexical ambiguity.
15. The apparatus of claim 14 wherein a number of the at least one best path is either predefined or determined programmatically.
16. The apparatus of claim 14 further comprising
means for compiling lexical grammar rules to generate lexical functions, the lexical grammar rules being written in a grammar programming language;
means for creating a plurality of segments from the at least two tokens based upon lexical information and the lexical functions, and
means for defining the plurality of paths using the plurality of segments.
17. The apparatus of claim 14 further comprising means for assigning at least one part of speech tag to at least one of the at least two tokens using lexical information.
18. The apparatus of claim 14 further comprising means for recognizing a multiword expression in the input stream using multiword information.
19. The apparatus of claim 14 wherein the connection graph comprises a set of nodes and a set of arcs.
20. The apparatus claim 19 wherein each of the plurality of paths comprises a combination of nodes and arcs.
21. The apparatus of claim 14 wherein the cost comprises lexical cost, unigram cost, bigram cost and connector cost.
22. An apparatus for providing segmentation of an input stream having at least two tokens, comprising:
means for creating a plurality of segments from the at least two tokens based upon lexical information and lexical functions; and
means for generating a connection graph using the plurality of segments.
23. The apparatus of claim 22 further comprising means for compiling lexical grammar, rules to generate the lexical functions, the lexical grammar rules being written in a grammar programming language.
24. The apparatus of claim 23 wherein the lexical grammar rules define connectivity relation of tokens.
25. The apparatus of claim 22 further comprising means for assigning at least one part of speech tag to at least one segment using a lexical dictionary.
26. The apparatus of claim 25 further comprising:
means for defining a plurality of paths in the connection graph based upon part of speech tags and the segments;
means for assigning a cost to each of the plurality of paths; and
means for determining at least one best path based upon a corresponding cost to generate an output graph.
27. An apparatus for reducing lexical ambiguity in an input stream, comprising:
a tokenizer for breaking the input stream into at least two tokens;
a token connector for creating a connection graph using the at least two tokens, the connection graph comprising a plurality of paths;
a cost assignor for assigning a cost to each of the plurality of paths;
a path calculator for defining at least one best path based upon a corresponding cost to generate an output graph; and
a graph provider for providing the output graph to a syntactic analysis module to reduce lexical ambiguity.
28. The apparatus of claim 27 wherein a number of the at least one best path is either predefined or determined programmatically.
29. The apparatus of claim 27 wherein the token connector comprises:
a grammar programming language (GPL) compiler for compiling lexical grammar rules to generate lexical functions, the lexical grammar rules being written in a general programming language;
a segmentation engine for creating a plurality of segments from the at least two tokens based upon lexical information and the lexical functions, and
a path designator for defining the plurality of paths using the plurality of segments.
30. The apparatus of claim 27 wherein the token connector comprises a part of speech tagger for assigning at least one part of speech tag to at least one of the at least two tokens using lexical information.
31. The apparatus of claim 27 wherein the token connector comprises a multiword recognizer for recognizing a multiword expression in the input stream using multiword information.
32. The apparatus of claim 27 wherein the connection graph comprises a set of nodes and a set of arcs.
33. The apparatus of claim 32 wherein each of the plurality of paths comprises a combination of nodes and arcs.
34. The apparatus of claim 27 wherein the cost comprises lexical cost, unigram cost, bigram cost and connector cost.
35. An apparatus for providing segmentation of an input stream having at least two tokens, comprising:
a segmentation engine for creating a plurality of segments from the at least two tokens based upon lexical information and lexical functions; and
a graph generator for generating a connection graph using the plurality of segments.
36. The apparatus of claim 35 further comprising a grammar programming language (GPL) compiler for compiling lexical grammar rules to generate the lexical functions, the lexical grammar rules being written in GPL.
37. The apparatus of claim 36 wherein the lexical grammar rules define connectivity relation of tokens.
38. The apparatus of claim 35 further comprising a part of speech tagger for assigning at least one part of speech tag to at least one segment using lexical information.
39. The apparatus of claim 38 further comprising:
a path designator for defining a plurality of paths in the connection graph based upon part of speech tags and the segments;
a cost assignor for assigning a cost to each of the plurality of paths; and
a path calculator for determining at least one best path based upon a corresponding cost to generate an output graph.
40. A system for reducing lexical ambiguity, comprising:
a processor;
an input coupled to the processor, the input capable of receiving an input stream, the processor configured to break the input stream into at least two tokens, create a connection graph comprising a plurality of paths using the at least two tokens, assign a cost to each of the plurality of paths, and define at least one best path based upon a corresponding cost to generate an output graph; and
an output coupled to the processor, the output capable of providing the output graph to a syntactic analysis module to reduce lexical ambiguity.
41. A system for providing segmentation of an input stream, comprising:
a processor;
an input coupled to the processor, the input capable of receiving an input stream having at lest two tokens, the processor configured to create a plurality of segments from the at least two tokens based upon lexical information and lexical functions, and generate a connection graph using the plurality of segments; and
an output coupled to the processor, the output capable of providing segmentation of the input stream.
42. A computer readable medium comprising instructions, which when executed on a processor, perform method for reducing lexical ambiguity in an input stream, comprising:
breaking an input stream into at least two tokens;
creating a connection graph using the at least one token, the connection graph comprising a plurality of paths;
assigning a cost to each of the plurality of paths;
defining at least one best path based upon a corresponding cost to generate an output graph; and
providing the output graph to a syntactic analysis module to reduce lexical ambiguity.
43. The computer readable medium of claim 42 wherein creating a connection graph further comprises providing segmentation of the input stream using lexical information and lexical functions.
44. The computer readable medium of claim 42 wherein creating a connection graph further comprises assigning at least one part of speech tag to at least one of the at least two tokens using lexical information.
45. The computer readable medium of claim 42 wherein creating a connection graph further comprises recognizing a multiword expression in the input stream using lexical information.
46. The computer readable medium of claim 42 wherein a number of the at least one best path is either predefined or determined programmatically.
47. A computer readable medium comprising instructions, which when executed on a processor, perform method for providing segmentation of an input stream having at least two tokens, comprising:
creating a plurality of segments from the at least two tokens based upon lexical information and lexical functions; and
generating a connection graph using the plurality of segments.
48. The computer readable medium of claim 47 further comprising compiling the lexical grammar rules to generate lexical functions, the lexical grammar rules being written in a grammar programming language.
49. A memory for storing data for access by an application program being executed on a data processing system, comprising:
a data structure stored in said memory, said data structure including information resident in a file used by said application program and including:
a plurality of packet structures used for the transmission of data, wherein each packet structure includes
a set of nodes,
a set of arcs connecting at least two of the set of nodes, and
a value data object for each of the set of arcs having a value that represents a corresponding part of speech tag.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US10/781,223 US20040167771A1 (en) | 1999-10-18 | 2004-02-17 | Method and system for reducing lexical ambiguity |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US09/420,517 US6721697B1 (en) | 1999-10-18 | 1999-10-18 | Method and system for reducing lexical ambiguity |
US10/781,223 US20040167771A1 (en) | 1999-10-18 | 2004-02-17 | Method and system for reducing lexical ambiguity |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US09/420,517 Division US6721697B1 (en) | 1999-10-18 | 1999-10-18 | Method and system for reducing lexical ambiguity |
Publications (1)
Publication Number | Publication Date |
---|---|
US20040167771A1 true US20040167771A1 (en) | 2004-08-26 |
Family
ID=23666795
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US09/420,517 Expired - Fee Related US6721697B1 (en) | 1999-10-18 | 1999-10-18 | Method and system for reducing lexical ambiguity |
US10/781,223 Abandoned US20040167771A1 (en) | 1999-10-18 | 2004-02-17 | Method and system for reducing lexical ambiguity |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US09/420,517 Expired - Fee Related US6721697B1 (en) | 1999-10-18 | 1999-10-18 | Method and system for reducing lexical ambiguity |
Country Status (3)
Country | Link |
---|---|
US (2) | US6721697B1 (en) |
AU (1) | AU2466901A (en) |
WO (1) | WO2001029697A1 (en) |
Cited By (43)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20060129393A1 (en) * | 2004-12-15 | 2006-06-15 | Electronics And Telecommunications Research Institute | System and method for synthesizing dialog-style speech using speech-act information |
US20060225057A1 (en) * | 2002-01-30 | 2006-10-05 | Geisinger Nile J | Method and system for creating programs using code having coupled syntactic and semantic relationship |
US20060277465A1 (en) * | 2005-06-07 | 2006-12-07 | Textual Analytics Solutions Pvt. Ltd. | System and method of textual information analytics |
US20060293874A1 (en) * | 2005-06-27 | 2006-12-28 | Microsoft Corporation | Translation and capture architecture for output of conversational utterances |
US20090018821A1 (en) * | 2006-02-27 | 2009-01-15 | Nec Corporation | Language processing device, language processing method, and language processing program |
US20090063131A1 (en) * | 2007-09-05 | 2009-03-05 | Modibo Soumare | Methods and systems for language representation |
US20090077113A1 (en) * | 2005-05-12 | 2009-03-19 | Kabire Fidaali | Device and method for semantic analysis of documents by construction of n-ary semantic trees |
US20090228263A1 (en) * | 2008-03-07 | 2009-09-10 | Kabushiki Kaisha Toshiba | Machine translating apparatus, method, and computer program product |
US20110231188A1 (en) * | 2005-08-31 | 2011-09-22 | Voicebox Technologies, Inc. | System and method for providing an acoustic grammar to dynamically sharpen speech interpretation |
US8073681B2 (en) * | 2006-10-16 | 2011-12-06 | Voicebox Technologies, Inc. | System and method for a cooperative conversational voice user interface |
US8112275B2 (en) | 2002-06-03 | 2012-02-07 | Voicebox Technologies, Inc. | System and method for user-specific speech recognition |
US8140335B2 (en) | 2007-12-11 | 2012-03-20 | Voicebox Technologies, Inc. | System and method for providing a natural language voice user interface in an integrated voice navigation services environment |
US8145489B2 (en) | 2007-02-06 | 2012-03-27 | Voicebox Technologies, Inc. | System and method for selecting and presenting advertisements based on natural language processing of voice-based input |
US8195468B2 (en) | 2005-08-29 | 2012-06-05 | Voicebox Technologies, Inc. | Mobile systems and methods of supporting natural language human-machine interactions |
US8326634B2 (en) | 2005-08-05 | 2012-12-04 | Voicebox Technologies, Inc. | Systems and methods for responding to natural language speech utterance |
US8326637B2 (en) | 2009-02-20 | 2012-12-04 | Voicebox Technologies, Inc. | System and method for processing multi-modal device interactions in a natural language voice services environment |
US8332224B2 (en) | 2005-08-10 | 2012-12-11 | Voicebox Technologies, Inc. | System and method of supporting adaptive misrecognition conversational speech |
US20120323951A1 (en) * | 2011-06-20 | 2012-12-20 | Alexandru Mihai Caruntu | Method and apparatus for providing contextual based searches |
US20130110497A1 (en) * | 2011-10-27 | 2013-05-02 | Microsoft Corporation | Functionality for Normalizing Linguistic Items |
US20130144597A1 (en) * | 2006-10-26 | 2013-06-06 | Mobile Technologies, Llc | Simultaneous translation of open domain lectures and speeches |
US8589161B2 (en) | 2008-05-27 | 2013-11-19 | Voicebox Technologies, Inc. | System and method for an integrated, multi-modal, multi-device natural language voice services environment |
US9031845B2 (en) | 2002-07-15 | 2015-05-12 | Nuance Communications, Inc. | Mobile systems and methods for responding to natural language speech utterance |
US20150161109A1 (en) * | 2012-01-13 | 2015-06-11 | Google Inc. | Reordering words for machine translation |
US9171541B2 (en) | 2009-11-10 | 2015-10-27 | Voicebox Technologies Corporation | System and method for hybrid processing in a natural language voice services environment |
US9305548B2 (en) | 2008-05-27 | 2016-04-05 | Voicebox Technologies Corporation | System and method for an integrated, multi-modal, multi-device natural language voice services environment |
US9436681B1 (en) * | 2013-07-16 | 2016-09-06 | Amazon Technologies, Inc. | Natural language translation techniques |
US9502025B2 (en) | 2009-11-10 | 2016-11-22 | Voicebox Technologies Corporation | System and method for providing a natural language content dedication service |
US9626703B2 (en) | 2014-09-16 | 2017-04-18 | Voicebox Technologies Corporation | Voice commerce |
US9747896B2 (en) | 2014-10-15 | 2017-08-29 | Voicebox Technologies Corporation | System and method for providing follow-up responses to prior natural language inputs of a user |
US9753918B2 (en) | 2008-04-15 | 2017-09-05 | Facebook, Inc. | Lexicon development via shared translation database |
WO2017176496A1 (en) * | 2016-04-08 | 2017-10-12 | Pearson Education, Inc. | System and method for automatic content aggregation generation |
US9898459B2 (en) | 2014-09-16 | 2018-02-20 | Voicebox Technologies Corporation | Integration of domain information into state transitions of a finite state transducer for natural language processing |
US10325215B2 (en) * | 2016-04-08 | 2019-06-18 | Pearson Education, Inc. | System and method for automatic content aggregation generation |
US10331784B2 (en) | 2016-07-29 | 2019-06-25 | Voicebox Technologies Corporation | System and method of disambiguating natural language processing requests |
US10431214B2 (en) | 2014-11-26 | 2019-10-01 | Voicebox Technologies Corporation | System and method of determining a domain and/or an action related to a natural language input |
US20200097549A1 (en) * | 2018-09-26 | 2020-03-26 | Asustek Computer Inc. | Semantic processing method, electronic device, and non-transitory computer readable recording medium |
US10614799B2 (en) | 2014-11-26 | 2020-04-07 | Voicebox Technologies Corporation | System and method of providing intent predictions for an utterance prior to a system detection of an end of the utterance |
US10642848B2 (en) | 2016-04-08 | 2020-05-05 | Pearson Education, Inc. | Personalized automatic content aggregation generation |
EP3522038A4 (en) * | 2016-09-28 | 2020-06-03 | Systran International Co. Ltd. | Method for translating characters and apparatus therefor |
US10789316B2 (en) | 2016-04-08 | 2020-09-29 | Pearson Education, Inc. | Personalized automatic content aggregation generation |
US10810357B1 (en) * | 2014-10-15 | 2020-10-20 | Slickjump, Inc. | System and method for selection of meaningful page elements with imprecise coordinate selection for relevant information identification and browsing |
US20210192138A1 (en) * | 2018-09-14 | 2021-06-24 | Beijing Bytedance Network Technology Co., Ltd. | Method and device for generating modified statement |
US11222185B2 (en) | 2006-10-26 | 2022-01-11 | Meta Platforms, Inc. | Lexicon development via shared translation database |
Families Citing this family (57)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7379862B1 (en) * | 1999-11-19 | 2008-05-27 | Microsoft Corporation | Method and apparatus for analyzing and debugging natural language parses |
US7146349B2 (en) * | 2000-11-06 | 2006-12-05 | International Business Machines Corporation | Network for describing multimedia information |
US20020091509A1 (en) * | 2001-01-02 | 2002-07-11 | Yacov Zoarez | Method and system for translating text |
US6859771B2 (en) * | 2001-04-23 | 2005-02-22 | Microsoft Corporation | System and method for identifying base noun phrases |
JP4947861B2 (en) * | 2001-09-25 | 2012-06-06 | キヤノン株式会社 | Natural language processing apparatus, control method therefor, and program |
US6963832B2 (en) * | 2001-10-09 | 2005-11-08 | Hewlett-Packard Development Company, L.P. | Meaning token dictionary for automatic speech recognition |
US7013262B2 (en) * | 2002-02-12 | 2006-03-14 | Sunflare Co., Ltd | System and method for accurate grammar analysis using a learners' model and part-of-speech tagged (POST) parser |
US6988063B2 (en) * | 2002-02-12 | 2006-01-17 | Sunflare Co., Ltd. | System and method for accurate grammar analysis using a part-of-speech tagged (POST) parser and learners' model |
US7493253B1 (en) * | 2002-07-12 | 2009-02-17 | Language And Computing, Inc. | Conceptual world representation natural language understanding system and method |
US7092567B2 (en) * | 2002-11-04 | 2006-08-15 | Matsushita Electric Industrial Co., Ltd. | Post-processing system and method for correcting machine recognized text |
US20040193557A1 (en) * | 2003-03-25 | 2004-09-30 | Olsen Jesse Dale | Systems and methods for reducing ambiguity of communications |
JP3768205B2 (en) * | 2003-05-30 | 2006-04-19 | 沖電気工業株式会社 | Morphological analyzer, morphological analysis method, and morphological analysis program |
US7328156B2 (en) * | 2003-07-17 | 2008-02-05 | International Business Machines Corporation | Computational linguistic statements for providing an autonomic computing environment |
US7430504B2 (en) * | 2004-03-02 | 2008-09-30 | Microsoft Corporation | Method and system for ranking words and concepts in a text using graph-based ranking |
US7908143B2 (en) * | 2004-04-28 | 2011-03-15 | International Business Machines Corporation | Dialog call-flow optimization |
FR2896603B1 (en) * | 2006-01-20 | 2008-05-02 | Thales Sa | METHOD AND DEVICE FOR EXTRACTING INFORMATION AND TRANSFORMING THEM INTO QUALITATIVE DATA OF A TEXTUAL DOCUMENT |
US9984071B2 (en) | 2006-10-10 | 2018-05-29 | Abbyy Production Llc | Language ambiguity detection of text |
US8214199B2 (en) * | 2006-10-10 | 2012-07-03 | Abbyy Software, Ltd. | Systems for translating sentences between languages using language-independent semantic structures and ratings of syntactic constructions |
US9633005B2 (en) | 2006-10-10 | 2017-04-25 | Abbyy Infopoisk Llc | Exhaustive automatic processing of textual information |
US9235573B2 (en) | 2006-10-10 | 2016-01-12 | Abbyy Infopoisk Llc | Universal difference measure |
US8548795B2 (en) * | 2006-10-10 | 2013-10-01 | Abbyy Software Ltd. | Method for translating documents from one language into another using a database of translations, a terminology dictionary, a translation dictionary, and a machine translation system |
US9645993B2 (en) | 2006-10-10 | 2017-05-09 | Abbyy Infopoisk Llc | Method and system for semantic searching |
US8195447B2 (en) | 2006-10-10 | 2012-06-05 | Abbyy Software Ltd. | Translating sentences between languages using language-independent semantic structures and ratings of syntactic constructions |
US8145473B2 (en) | 2006-10-10 | 2012-03-27 | Abbyy Software Ltd. | Deep model statistics method for machine translation |
US9047275B2 (en) | 2006-10-10 | 2015-06-02 | Abbyy Infopoisk Llc | Methods and systems for alignment of parallel text corpora |
US8959011B2 (en) | 2007-03-22 | 2015-02-17 | Abbyy Infopoisk Llc | Indicating and correcting errors in machine translation systems |
US9779079B2 (en) * | 2007-06-01 | 2017-10-03 | Xerox Corporation | Authoring system |
US8812296B2 (en) | 2007-06-27 | 2014-08-19 | Abbyy Infopoisk Llc | Method and system for natural language dictionary generation |
US8738360B2 (en) | 2008-06-06 | 2014-05-27 | Apple Inc. | Data detection of a character sequence having multiple possible data types |
US9262409B2 (en) | 2008-08-06 | 2016-02-16 | Abbyy Infopoisk Llc | Translation of a selected text fragment of a screen |
US8321848B2 (en) * | 2009-04-16 | 2012-11-27 | The Mathworks, Inc. | Method and system for syntax error repair in programming languages |
US9547642B2 (en) * | 2009-06-17 | 2017-01-17 | Empire Technology Development Llc | Voice to text to voice processing |
US9201965B1 (en) | 2009-09-30 | 2015-12-01 | Cisco Technology, Inc. | System and method for providing speech recognition using personal vocabulary in a network environment |
US8935274B1 (en) * | 2010-05-12 | 2015-01-13 | Cisco Technology, Inc | System and method for deriving user expertise based on data propagating in a network environment |
US8977538B2 (en) * | 2010-09-13 | 2015-03-10 | Richard Salisbury | Constructing and analyzing a word graph |
US9465795B2 (en) | 2010-12-17 | 2016-10-11 | Cisco Technology, Inc. | System and method for providing feeds based on activity in a network environment |
US8909624B2 (en) | 2011-05-31 | 2014-12-09 | Cisco Technology, Inc. | System and method for evaluating results of a search query in a network environment |
JP5799733B2 (en) * | 2011-10-12 | 2015-10-28 | 富士通株式会社 | Recognition device, recognition program, and recognition method |
CN103365834B (en) * | 2012-03-29 | 2017-08-18 | 富泰华工业(深圳)有限公司 | Language Ambiguity eliminates system and method |
US8989485B2 (en) | 2012-04-27 | 2015-03-24 | Abbyy Development Llc | Detecting a junction in a text line of CJK characters |
US8971630B2 (en) | 2012-04-27 | 2015-03-03 | Abbyy Development Llc | Fast CJK character recognition |
RU2592395C2 (en) | 2013-12-19 | 2016-07-20 | Общество с ограниченной ответственностью "Аби ИнфоПоиск" | Resolution semantic ambiguity by statistical analysis |
RU2586577C2 (en) * | 2014-01-15 | 2016-06-10 | Общество с ограниченной ответственностью "Аби ИнфоПоиск" | Filtering arcs parser graph |
RU2596600C2 (en) | 2014-09-02 | 2016-09-10 | Общество с ограниченной ответственностью "Аби Девелопмент" | Methods and systems for processing images of mathematical expressions |
CN104239294B (en) * | 2014-09-10 | 2017-06-06 | 华建宇通科技(北京)有限责任公司 | Hide the how tactful Tibetan language long sentence cutting method of Chinese translation system |
US9626358B2 (en) | 2014-11-26 | 2017-04-18 | Abbyy Infopoisk Llc | Creating ontologies by analyzing natural language texts |
US9990361B2 (en) * | 2015-10-08 | 2018-06-05 | Facebook, Inc. | Language independent representations |
US10586168B2 (en) | 2015-10-08 | 2020-03-10 | Facebook, Inc. | Deep translations |
US10769535B2 (en) * | 2016-05-13 | 2020-09-08 | Cognitive Scale, Inc. | Ingestion pipeline for universal cognitive graph |
US10331788B2 (en) | 2016-06-22 | 2019-06-25 | International Business Machines Corporation | Latent ambiguity handling in natural language processing |
US10169324B2 (en) * | 2016-12-08 | 2019-01-01 | Entit Software Llc | Universal lexical analyzers |
US10528665B2 (en) * | 2017-01-11 | 2020-01-07 | Satyanarayana Krishnamurthy | System and method for natural language generation |
CN107301170B (en) * | 2017-06-19 | 2020-12-22 | 北京百度网讯科技有限公司 | Method and device for segmenting sentences based on artificial intelligence |
US10901811B2 (en) | 2017-07-31 | 2021-01-26 | Splunk Inc. | Creating alerts associated with a data storage system based on natural language requests |
US11494395B2 (en) | 2017-07-31 | 2022-11-08 | Splunk Inc. | Creating dashboards for viewing data in a data storage system based on natural language requests |
US20190034555A1 (en) * | 2017-07-31 | 2019-01-31 | Splunk Inc. | Translating a natural language request to a domain specific language request based on multiple interpretation algorithms |
US10956670B2 (en) | 2018-03-03 | 2021-03-23 | Samurai Labs Sp. Z O.O. | System and method for detecting undesirable and potentially harmful online behavior |
Citations (38)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US4905138A (en) * | 1985-10-17 | 1990-02-27 | Westinghouse Electric Corp. | Meta-interpreter |
US4974191A (en) * | 1987-07-31 | 1990-11-27 | Syntellect Software Inc. | Adaptive natural language computer interface system |
US5033087A (en) * | 1989-03-14 | 1991-07-16 | International Business Machines Corp. | Method and apparatus for the automatic determination of phonological rules as for a continuous speech recognition system |
US5068789A (en) * | 1988-09-15 | 1991-11-26 | Oce-Nederland B.V. | Method and means for grammatically processing a natural language sentence |
US5083268A (en) * | 1986-10-15 | 1992-01-21 | Texas Instruments Incorporated | System and method for parsing natural language by unifying lexical features of words |
US5088038A (en) * | 1989-05-24 | 1992-02-11 | Kabushiki Kaisha Toshiba | Machine translation system and method of machine translation |
US5095432A (en) * | 1989-07-10 | 1992-03-10 | Harris Corporation | Data processing system implemented process and compiling technique for performing context-free parsing algorithm based on register vector grammar |
US5111398A (en) * | 1988-11-21 | 1992-05-05 | Xerox Corporation | Processing natural language text using autonomous punctuational structure |
US5268840A (en) * | 1992-04-30 | 1993-12-07 | Industrial Technology Research Institute | Method and system for morphologizing text |
US5418717A (en) * | 1990-08-27 | 1995-05-23 | Su; Keh-Yih | Multiple score language processing system |
US5426583A (en) * | 1993-02-02 | 1995-06-20 | Uribe-Echebarria Diaz De Mendibil; Gregorio | Automatic interlingual translation system |
US5510981A (en) * | 1993-10-28 | 1996-04-23 | International Business Machines Corporation | Language translation apparatus and method using context-based translation models |
US5528491A (en) * | 1992-08-31 | 1996-06-18 | Language Engineering Corporation | Apparatus and method for automated natural language translation |
US5535120A (en) * | 1990-12-31 | 1996-07-09 | Trans-Link International Corp. | Machine translation and telecommunications system using user ID data to select dictionaries |
US5642519A (en) * | 1994-04-29 | 1997-06-24 | Sun Microsystems, Inc. | Speech interpreter with a unified grammer compiler |
US5644755A (en) * | 1995-02-24 | 1997-07-01 | Compaq Computer Corporation | Processor with virtual system mode |
US5659765A (en) * | 1994-03-15 | 1997-08-19 | Toppan Printing Co., Ltd. | Machine translation system |
US5677835A (en) * | 1992-09-04 | 1997-10-14 | Caterpillar Inc. | Integrated authoring and translation system |
US5680511A (en) * | 1995-06-07 | 1997-10-21 | Dragon Systems, Inc. | Systems and methods for word recognition |
US5768603A (en) * | 1991-07-25 | 1998-06-16 | International Business Machines Corporation | Method and system for natural language translation |
US5799268A (en) * | 1994-09-28 | 1998-08-25 | Apple Computer, Inc. | Method for extracting knowledge from online documentation and creating a glossary, index, help database or the like |
US5806021A (en) * | 1995-10-30 | 1998-09-08 | International Business Machines Corporation | Automatic segmentation of continuous text using statistical approaches |
US5864788A (en) * | 1992-09-25 | 1999-01-26 | Sharp Kabushiki Kaisha | Translation machine having a function of deriving two or more syntaxes from one original sentence and giving precedence to a selected one of the syntaxes |
US5873056A (en) * | 1993-10-12 | 1999-02-16 | The Syracuse University | Natural language processing system for semantic vector representation which accounts for lexical ambiguity |
US5884247A (en) * | 1996-10-31 | 1999-03-16 | Dialect Corporation | Method and apparatus for automated language translation |
US5907821A (en) * | 1995-11-06 | 1999-05-25 | Hitachi, Ltd. | Method of computer-based automatic extraction of translation pairs of words from a bilingual text |
US5963894A (en) * | 1994-06-24 | 1999-10-05 | Microsoft Corporation | Method and system for bootstrapping statistical processing into a rule-based natural language parser |
US5966686A (en) * | 1996-06-28 | 1999-10-12 | Microsoft Corporation | Method and system for computing semantic logical forms from syntax trees |
US5983169A (en) * | 1995-11-13 | 1999-11-09 | Japan Science And Technology Corporation | Method for automated translation of conjunctive phrases in natural languages |
US6131082A (en) * | 1995-06-07 | 2000-10-10 | Int'l.Com, Inc. | Machine assisted translation tools utilizing an inverted index and list of letter n-grams |
US6161083A (en) * | 1996-05-02 | 2000-12-12 | Sony Corporation | Example-based translation method and system which calculates word similarity degrees, a priori probability, and transformation probability to determine the best example for translation |
US6173441B1 (en) * | 1998-10-16 | 2001-01-09 | Peter A. Klein | Method and system for compiling source code containing natural language instructions |
US6212500B1 (en) * | 1996-09-10 | 2001-04-03 | Siemens Aktiengesellschaft | Process for the multilingual use of a hidden markov sound model in a speech recognition system |
US6230153B1 (en) * | 1998-06-18 | 2001-05-08 | International Business Machines Corporation | Association rule ranker for web site emulation |
US6243669B1 (en) * | 1999-01-29 | 2001-06-05 | Sony Corporation | Method and apparatus for providing syntactic analysis and data structure for translation knowledge in example-based language translation |
US6285978B1 (en) * | 1998-09-24 | 2001-09-04 | International Business Machines Corporation | System and method for estimating accuracy of an automatic natural language translation |
US6301554B1 (en) * | 1999-09-23 | 2001-10-09 | Wordstream, Inc. | Language translation using a constrained grammar in the form of structured sentences formed according to pre-defined grammar templates |
US6330530B1 (en) * | 1999-10-18 | 2001-12-11 | Sony Corporation | Method and system for transforming a source language linguistic structure into a target language linguistic structure based on example linguistic feature structures |
Family Cites Families (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5644775A (en) | 1994-08-11 | 1997-07-01 | International Business Machines Corporation | Method and system for facilitating language translation using string-formatting libraries |
-
1999
- 1999-10-18 US US09/420,517 patent/US6721697B1/en not_active Expired - Fee Related
-
2000
- 2000-10-17 WO PCT/US2000/041256 patent/WO2001029697A1/en active Application Filing
- 2000-10-17 AU AU24669/01A patent/AU2466901A/en not_active Abandoned
-
2004
- 2004-02-17 US US10/781,223 patent/US20040167771A1/en not_active Abandoned
Patent Citations (38)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US4905138A (en) * | 1985-10-17 | 1990-02-27 | Westinghouse Electric Corp. | Meta-interpreter |
US5083268A (en) * | 1986-10-15 | 1992-01-21 | Texas Instruments Incorporated | System and method for parsing natural language by unifying lexical features of words |
US4974191A (en) * | 1987-07-31 | 1990-11-27 | Syntellect Software Inc. | Adaptive natural language computer interface system |
US5068789A (en) * | 1988-09-15 | 1991-11-26 | Oce-Nederland B.V. | Method and means for grammatically processing a natural language sentence |
US5111398A (en) * | 1988-11-21 | 1992-05-05 | Xerox Corporation | Processing natural language text using autonomous punctuational structure |
US5033087A (en) * | 1989-03-14 | 1991-07-16 | International Business Machines Corp. | Method and apparatus for the automatic determination of phonological rules as for a continuous speech recognition system |
US5088038A (en) * | 1989-05-24 | 1992-02-11 | Kabushiki Kaisha Toshiba | Machine translation system and method of machine translation |
US5095432A (en) * | 1989-07-10 | 1992-03-10 | Harris Corporation | Data processing system implemented process and compiling technique for performing context-free parsing algorithm based on register vector grammar |
US5418717A (en) * | 1990-08-27 | 1995-05-23 | Su; Keh-Yih | Multiple score language processing system |
US5535120A (en) * | 1990-12-31 | 1996-07-09 | Trans-Link International Corp. | Machine translation and telecommunications system using user ID data to select dictionaries |
US5768603A (en) * | 1991-07-25 | 1998-06-16 | International Business Machines Corporation | Method and system for natural language translation |
US5268840A (en) * | 1992-04-30 | 1993-12-07 | Industrial Technology Research Institute | Method and system for morphologizing text |
US5528491A (en) * | 1992-08-31 | 1996-06-18 | Language Engineering Corporation | Apparatus and method for automated natural language translation |
US5677835A (en) * | 1992-09-04 | 1997-10-14 | Caterpillar Inc. | Integrated authoring and translation system |
US5864788A (en) * | 1992-09-25 | 1999-01-26 | Sharp Kabushiki Kaisha | Translation machine having a function of deriving two or more syntaxes from one original sentence and giving precedence to a selected one of the syntaxes |
US5426583A (en) * | 1993-02-02 | 1995-06-20 | Uribe-Echebarria Diaz De Mendibil; Gregorio | Automatic interlingual translation system |
US5873056A (en) * | 1993-10-12 | 1999-02-16 | The Syracuse University | Natural language processing system for semantic vector representation which accounts for lexical ambiguity |
US5510981A (en) * | 1993-10-28 | 1996-04-23 | International Business Machines Corporation | Language translation apparatus and method using context-based translation models |
US5659765A (en) * | 1994-03-15 | 1997-08-19 | Toppan Printing Co., Ltd. | Machine translation system |
US5642519A (en) * | 1994-04-29 | 1997-06-24 | Sun Microsystems, Inc. | Speech interpreter with a unified grammer compiler |
US5963894A (en) * | 1994-06-24 | 1999-10-05 | Microsoft Corporation | Method and system for bootstrapping statistical processing into a rule-based natural language parser |
US5799268A (en) * | 1994-09-28 | 1998-08-25 | Apple Computer, Inc. | Method for extracting knowledge from online documentation and creating a glossary, index, help database or the like |
US5644755A (en) * | 1995-02-24 | 1997-07-01 | Compaq Computer Corporation | Processor with virtual system mode |
US5680511A (en) * | 1995-06-07 | 1997-10-21 | Dragon Systems, Inc. | Systems and methods for word recognition |
US6131082A (en) * | 1995-06-07 | 2000-10-10 | Int'l.Com, Inc. | Machine assisted translation tools utilizing an inverted index and list of letter n-grams |
US5806021A (en) * | 1995-10-30 | 1998-09-08 | International Business Machines Corporation | Automatic segmentation of continuous text using statistical approaches |
US5907821A (en) * | 1995-11-06 | 1999-05-25 | Hitachi, Ltd. | Method of computer-based automatic extraction of translation pairs of words from a bilingual text |
US5983169A (en) * | 1995-11-13 | 1999-11-09 | Japan Science And Technology Corporation | Method for automated translation of conjunctive phrases in natural languages |
US6161083A (en) * | 1996-05-02 | 2000-12-12 | Sony Corporation | Example-based translation method and system which calculates word similarity degrees, a priori probability, and transformation probability to determine the best example for translation |
US5966686A (en) * | 1996-06-28 | 1999-10-12 | Microsoft Corporation | Method and system for computing semantic logical forms from syntax trees |
US6212500B1 (en) * | 1996-09-10 | 2001-04-03 | Siemens Aktiengesellschaft | Process for the multilingual use of a hidden markov sound model in a speech recognition system |
US5884247A (en) * | 1996-10-31 | 1999-03-16 | Dialect Corporation | Method and apparatus for automated language translation |
US6230153B1 (en) * | 1998-06-18 | 2001-05-08 | International Business Machines Corporation | Association rule ranker for web site emulation |
US6285978B1 (en) * | 1998-09-24 | 2001-09-04 | International Business Machines Corporation | System and method for estimating accuracy of an automatic natural language translation |
US6173441B1 (en) * | 1998-10-16 | 2001-01-09 | Peter A. Klein | Method and system for compiling source code containing natural language instructions |
US6243669B1 (en) * | 1999-01-29 | 2001-06-05 | Sony Corporation | Method and apparatus for providing syntactic analysis and data structure for translation knowledge in example-based language translation |
US6301554B1 (en) * | 1999-09-23 | 2001-10-09 | Wordstream, Inc. | Language translation using a constrained grammar in the form of structured sentences formed according to pre-defined grammar templates |
US6330530B1 (en) * | 1999-10-18 | 2001-12-11 | Sony Corporation | Method and system for transforming a source language linguistic structure into a target language linguistic structure based on example linguistic feature structures |
Cited By (103)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20060225057A1 (en) * | 2002-01-30 | 2006-10-05 | Geisinger Nile J | Method and system for creating programs using code having coupled syntactic and semantic relationship |
US8112275B2 (en) | 2002-06-03 | 2012-02-07 | Voicebox Technologies, Inc. | System and method for user-specific speech recognition |
US8731929B2 (en) | 2002-06-03 | 2014-05-20 | Voicebox Technologies Corporation | Agent architecture for determining meanings of natural language utterances |
US8155962B2 (en) | 2002-06-03 | 2012-04-10 | Voicebox Technologies, Inc. | Method and system for asynchronously processing natural language utterances |
US8140327B2 (en) | 2002-06-03 | 2012-03-20 | Voicebox Technologies, Inc. | System and method for filtering and eliminating noise from natural language utterances to improve speech recognition and parsing |
US9031845B2 (en) | 2002-07-15 | 2015-05-12 | Nuance Communications, Inc. | Mobile systems and methods for responding to natural language speech utterance |
US20060129393A1 (en) * | 2004-12-15 | 2006-06-15 | Electronics And Telecommunications Research Institute | System and method for synthesizing dialog-style speech using speech-act information |
US20090077113A1 (en) * | 2005-05-12 | 2009-03-19 | Kabire Fidaali | Device and method for semantic analysis of documents by construction of n-ary semantic trees |
US7856438B2 (en) * | 2005-05-12 | 2010-12-21 | Kabire Fidaali | Device and method for semantic analysis of documents by construction of n-ary semantic trees |
US20060277465A1 (en) * | 2005-06-07 | 2006-12-07 | Textual Analytics Solutions Pvt. Ltd. | System and method of textual information analytics |
US7689557B2 (en) * | 2005-06-07 | 2010-03-30 | Madan Pandit | System and method of textual information analytics |
US7991607B2 (en) * | 2005-06-27 | 2011-08-02 | Microsoft Corporation | Translation and capture architecture for output of conversational utterances |
US20060293874A1 (en) * | 2005-06-27 | 2006-12-28 | Microsoft Corporation | Translation and capture architecture for output of conversational utterances |
US8849670B2 (en) | 2005-08-05 | 2014-09-30 | Voicebox Technologies Corporation | Systems and methods for responding to natural language speech utterance |
US8326634B2 (en) | 2005-08-05 | 2012-12-04 | Voicebox Technologies, Inc. | Systems and methods for responding to natural language speech utterance |
US9263039B2 (en) | 2005-08-05 | 2016-02-16 | Nuance Communications, Inc. | Systems and methods for responding to natural language speech utterance |
US8332224B2 (en) | 2005-08-10 | 2012-12-11 | Voicebox Technologies, Inc. | System and method of supporting adaptive misrecognition conversational speech |
US8620659B2 (en) | 2005-08-10 | 2013-12-31 | Voicebox Technologies, Inc. | System and method of supporting adaptive misrecognition in conversational speech |
US9626959B2 (en) | 2005-08-10 | 2017-04-18 | Nuance Communications, Inc. | System and method of supporting adaptive misrecognition in conversational speech |
US8447607B2 (en) | 2005-08-29 | 2013-05-21 | Voicebox Technologies, Inc. | Mobile systems and methods of supporting natural language human-machine interactions |
US8849652B2 (en) | 2005-08-29 | 2014-09-30 | Voicebox Technologies Corporation | Mobile systems and methods of supporting natural language human-machine interactions |
US9495957B2 (en) | 2005-08-29 | 2016-11-15 | Nuance Communications, Inc. | Mobile systems and methods of supporting natural language human-machine interactions |
US8195468B2 (en) | 2005-08-29 | 2012-06-05 | Voicebox Technologies, Inc. | Mobile systems and methods of supporting natural language human-machine interactions |
US8150694B2 (en) | 2005-08-31 | 2012-04-03 | Voicebox Technologies, Inc. | System and method for providing an acoustic grammar to dynamically sharpen speech interpretation |
US8069046B2 (en) | 2005-08-31 | 2011-11-29 | Voicebox Technologies, Inc. | Dynamic speech sharpening |
US20110231188A1 (en) * | 2005-08-31 | 2011-09-22 | Voicebox Technologies, Inc. | System and method for providing an acoustic grammar to dynamically sharpen speech interpretation |
US8301435B2 (en) * | 2006-02-27 | 2012-10-30 | Nec Corporation | Removing ambiguity when analyzing a sentence with a word having multiple meanings |
US20090018821A1 (en) * | 2006-02-27 | 2009-01-15 | Nec Corporation | Language processing device, language processing method, and language processing program |
US10510341B1 (en) | 2006-10-16 | 2019-12-17 | Vb Assets, Llc | System and method for a cooperative conversational voice user interface |
US10297249B2 (en) | 2006-10-16 | 2019-05-21 | Vb Assets, Llc | System and method for a cooperative conversational voice user interface |
US11222626B2 (en) | 2006-10-16 | 2022-01-11 | Vb Assets, Llc | System and method for a cooperative conversational voice user interface |
US9015049B2 (en) | 2006-10-16 | 2015-04-21 | Voicebox Technologies Corporation | System and method for a cooperative conversational voice user interface |
US10515628B2 (en) | 2006-10-16 | 2019-12-24 | Vb Assets, Llc | System and method for a cooperative conversational voice user interface |
US8073681B2 (en) * | 2006-10-16 | 2011-12-06 | Voicebox Technologies, Inc. | System and method for a cooperative conversational voice user interface |
US8515765B2 (en) | 2006-10-16 | 2013-08-20 | Voicebox Technologies, Inc. | System and method for a cooperative conversational voice user interface |
US10755699B2 (en) | 2006-10-16 | 2020-08-25 | Vb Assets, Llc | System and method for a cooperative conversational voice user interface |
US20130144597A1 (en) * | 2006-10-26 | 2013-06-06 | Mobile Technologies, Llc | Simultaneous translation of open domain lectures and speeches |
US11222185B2 (en) | 2006-10-26 | 2022-01-11 | Meta Platforms, Inc. | Lexicon development via shared translation database |
US9830318B2 (en) | 2006-10-26 | 2017-11-28 | Facebook, Inc. | Simultaneous translation of open domain lectures and speeches |
US9128926B2 (en) * | 2006-10-26 | 2015-09-08 | Facebook, Inc. | Simultaneous translation of open domain lectures and speeches |
US8886536B2 (en) | 2007-02-06 | 2014-11-11 | Voicebox Technologies Corporation | System and method for delivering targeted advertisements and tracking advertisement interactions in voice recognition contexts |
US10134060B2 (en) | 2007-02-06 | 2018-11-20 | Vb Assets, Llc | System and method for delivering targeted advertisements and/or providing natural language processing based on advertisements |
US9406078B2 (en) | 2007-02-06 | 2016-08-02 | Voicebox Technologies Corporation | System and method for delivering targeted advertisements and/or providing natural language processing based on advertisements |
US9269097B2 (en) | 2007-02-06 | 2016-02-23 | Voicebox Technologies Corporation | System and method for delivering targeted advertisements and/or providing natural language processing based on advertisements |
US8145489B2 (en) | 2007-02-06 | 2012-03-27 | Voicebox Technologies, Inc. | System and method for selecting and presenting advertisements based on natural language processing of voice-based input |
US11080758B2 (en) | 2007-02-06 | 2021-08-03 | Vb Assets, Llc | System and method for delivering targeted advertisements and/or providing natural language processing based on advertisements |
US8527274B2 (en) | 2007-02-06 | 2013-09-03 | Voicebox Technologies, Inc. | System and method for delivering targeted advertisements and tracking advertisement interactions in voice recognition contexts |
US8738353B2 (en) * | 2007-09-05 | 2014-05-27 | Modibo Soumare | Relational database method and systems for alphabet based language representation |
US20090063131A1 (en) * | 2007-09-05 | 2009-03-05 | Modibo Soumare | Methods and systems for language representation |
US8983839B2 (en) | 2007-12-11 | 2015-03-17 | Voicebox Technologies Corporation | System and method for dynamically generating a recognition grammar in an integrated voice navigation services environment |
US9620113B2 (en) | 2007-12-11 | 2017-04-11 | Voicebox Technologies Corporation | System and method for providing a natural language voice user interface |
US8370147B2 (en) | 2007-12-11 | 2013-02-05 | Voicebox Technologies, Inc. | System and method for providing a natural language voice user interface in an integrated voice navigation services environment |
US10347248B2 (en) | 2007-12-11 | 2019-07-09 | Voicebox Technologies Corporation | System and method for providing in-vehicle services via a natural language voice user interface |
US8452598B2 (en) | 2007-12-11 | 2013-05-28 | Voicebox Technologies, Inc. | System and method for providing advertisements in an integrated voice navigation services environment |
US8326627B2 (en) | 2007-12-11 | 2012-12-04 | Voicebox Technologies, Inc. | System and method for dynamically generating a recognition grammar in an integrated voice navigation services environment |
US8140335B2 (en) | 2007-12-11 | 2012-03-20 | Voicebox Technologies, Inc. | System and method for providing a natural language voice user interface in an integrated voice navigation services environment |
US8719026B2 (en) | 2007-12-11 | 2014-05-06 | Voicebox Technologies Corporation | System and method for providing a natural language voice user interface in an integrated voice navigation services environment |
US8204735B2 (en) * | 2008-03-07 | 2012-06-19 | Kabushiki Kaisha Toshiba | Machine translating apparatus, method, and computer program product |
US20090228263A1 (en) * | 2008-03-07 | 2009-09-10 | Kabushiki Kaisha Toshiba | Machine translating apparatus, method, and computer program product |
US9753918B2 (en) | 2008-04-15 | 2017-09-05 | Facebook, Inc. | Lexicon development via shared translation database |
US8589161B2 (en) | 2008-05-27 | 2013-11-19 | Voicebox Technologies, Inc. | System and method for an integrated, multi-modal, multi-device natural language voice services environment |
US10089984B2 (en) | 2008-05-27 | 2018-10-02 | Vb Assets, Llc | System and method for an integrated, multi-modal, multi-device natural language voice services environment |
US10553216B2 (en) | 2008-05-27 | 2020-02-04 | Oracle International Corporation | System and method for an integrated, multi-modal, multi-device natural language voice services environment |
US9305548B2 (en) | 2008-05-27 | 2016-04-05 | Voicebox Technologies Corporation | System and method for an integrated, multi-modal, multi-device natural language voice services environment |
US9711143B2 (en) | 2008-05-27 | 2017-07-18 | Voicebox Technologies Corporation | System and method for an integrated, multi-modal, multi-device natural language voice services environment |
US9105266B2 (en) | 2009-02-20 | 2015-08-11 | Voicebox Technologies Corporation | System and method for processing multi-modal device interactions in a natural language voice services environment |
US8738380B2 (en) | 2009-02-20 | 2014-05-27 | Voicebox Technologies Corporation | System and method for processing multi-modal device interactions in a natural language voice services environment |
US8326637B2 (en) | 2009-02-20 | 2012-12-04 | Voicebox Technologies, Inc. | System and method for processing multi-modal device interactions in a natural language voice services environment |
US10553213B2 (en) | 2009-02-20 | 2020-02-04 | Oracle International Corporation | System and method for processing multi-modal device interactions in a natural language voice services environment |
US8719009B2 (en) | 2009-02-20 | 2014-05-06 | Voicebox Technologies Corporation | System and method for processing multi-modal device interactions in a natural language voice services environment |
US9953649B2 (en) | 2009-02-20 | 2018-04-24 | Voicebox Technologies Corporation | System and method for processing multi-modal device interactions in a natural language voice services environment |
US9570070B2 (en) | 2009-02-20 | 2017-02-14 | Voicebox Technologies Corporation | System and method for processing multi-modal device interactions in a natural language voice services environment |
US9171541B2 (en) | 2009-11-10 | 2015-10-27 | Voicebox Technologies Corporation | System and method for hybrid processing in a natural language voice services environment |
US9502025B2 (en) | 2009-11-10 | 2016-11-22 | Voicebox Technologies Corporation | System and method for providing a natural language content dedication service |
US20120323951A1 (en) * | 2011-06-20 | 2012-12-20 | Alexandru Mihai Caruntu | Method and apparatus for providing contextual based searches |
US9721003B2 (en) * | 2011-06-20 | 2017-08-01 | Nokia Technologies Oy | Method and apparatus for providing contextual based searches |
US8909516B2 (en) * | 2011-10-27 | 2014-12-09 | Microsoft Corporation | Functionality for normalizing linguistic items |
US20130110497A1 (en) * | 2011-10-27 | 2013-05-02 | Microsoft Corporation | Functionality for Normalizing Linguistic Items |
US20150161109A1 (en) * | 2012-01-13 | 2015-06-11 | Google Inc. | Reordering words for machine translation |
US9436681B1 (en) * | 2013-07-16 | 2016-09-06 | Amazon Technologies, Inc. | Natural language translation techniques |
US9898459B2 (en) | 2014-09-16 | 2018-02-20 | Voicebox Technologies Corporation | Integration of domain information into state transitions of a finite state transducer for natural language processing |
US11087385B2 (en) | 2014-09-16 | 2021-08-10 | Vb Assets, Llc | Voice commerce |
US10430863B2 (en) | 2014-09-16 | 2019-10-01 | Vb Assets, Llc | Voice commerce |
US10216725B2 (en) | 2014-09-16 | 2019-02-26 | Voicebox Technologies Corporation | Integration of domain information into state transitions of a finite state transducer for natural language processing |
US9626703B2 (en) | 2014-09-16 | 2017-04-18 | Voicebox Technologies Corporation | Voice commerce |
US10229673B2 (en) | 2014-10-15 | 2019-03-12 | Voicebox Technologies Corporation | System and method for providing follow-up responses to prior natural language inputs of a user |
US9747896B2 (en) | 2014-10-15 | 2017-08-29 | Voicebox Technologies Corporation | System and method for providing follow-up responses to prior natural language inputs of a user |
US10810357B1 (en) * | 2014-10-15 | 2020-10-20 | Slickjump, Inc. | System and method for selection of meaningful page elements with imprecise coordinate selection for relevant information identification and browsing |
US10614799B2 (en) | 2014-11-26 | 2020-04-07 | Voicebox Technologies Corporation | System and method of providing intent predictions for an utterance prior to a system detection of an end of the utterance |
US10431214B2 (en) | 2014-11-26 | 2019-10-01 | Voicebox Technologies Corporation | System and method of determining a domain and/or an action related to a natural language input |
US10789316B2 (en) | 2016-04-08 | 2020-09-29 | Pearson Education, Inc. | Personalized automatic content aggregation generation |
US10642848B2 (en) | 2016-04-08 | 2020-05-05 | Pearson Education, Inc. | Personalized automatic content aggregation generation |
US10380126B1 (en) | 2016-04-08 | 2019-08-13 | Pearson Education, Inc. | System and method for automatic content aggregation evaluation |
US10459956B1 (en) | 2016-04-08 | 2019-10-29 | Pearson Education, Inc. | System and method for automatic content aggregation database evaluation |
US10325215B2 (en) * | 2016-04-08 | 2019-06-18 | Pearson Education, Inc. | System and method for automatic content aggregation generation |
US10419559B1 (en) | 2016-04-08 | 2019-09-17 | Pearson Education, Inc. | System and method for decay-based content provisioning |
WO2017176496A1 (en) * | 2016-04-08 | 2017-10-12 | Pearson Education, Inc. | System and method for automatic content aggregation generation |
US10331784B2 (en) | 2016-07-29 | 2019-06-25 | Voicebox Technologies Corporation | System and method of disambiguating natural language processing requests |
EP3522038A4 (en) * | 2016-09-28 | 2020-06-03 | Systran International Co. Ltd. | Method for translating characters and apparatus therefor |
US20210192138A1 (en) * | 2018-09-14 | 2021-06-24 | Beijing Bytedance Network Technology Co., Ltd. | Method and device for generating modified statement |
US11531814B2 (en) * | 2018-09-14 | 2022-12-20 | Beijing Bytedance Network Technology Co., Ltd. | Method and device for generating modified statement |
US20200097549A1 (en) * | 2018-09-26 | 2020-03-26 | Asustek Computer Inc. | Semantic processing method, electronic device, and non-transitory computer readable recording medium |
US11501077B2 (en) * | 2018-09-26 | 2022-11-15 | Asustek Computer Inc. | Semantic processing method, electronic device, and non-transitory computer readable recording medium |
Also Published As
Publication number | Publication date |
---|---|
US6721697B1 (en) | 2004-04-13 |
WO2001029697A9 (en) | 2002-08-08 |
WO2001029697A1 (en) | 2001-04-26 |
AU2466901A (en) | 2001-04-30 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US6721697B1 (en) | Method and system for reducing lexical ambiguity | |
US6330530B1 (en) | Method and system for transforming a source language linguistic structure into a target language linguistic structure based on example linguistic feature structures | |
US6928448B1 (en) | System and method to match linguistic structures using thesaurus information | |
US6778949B2 (en) | Method and system to analyze, transfer and generate language expressions using compiled instructions to manipulate linguistic structures | |
US6223150B1 (en) | Method and apparatus for parsing in a spoken language translation system | |
US6243669B1 (en) | Method and apparatus for providing syntactic analysis and data structure for translation knowledge in example-based language translation | |
US6278968B1 (en) | Method and apparatus for adaptive speech recognition hypothesis construction and selection in a spoken language translation system | |
US6442524B1 (en) | Analyzing inflectional morphology in a spoken language translation system | |
US6356865B1 (en) | Method and apparatus for performing spoken language translation | |
US6266642B1 (en) | Method and portable apparatus for performing spoken language translation | |
US6282507B1 (en) | Method and apparatus for interactive source language expression recognition and alternative hypothesis presentation and selection | |
US6529865B1 (en) | System and method to compile instructions to manipulate linguistic structures into separate functions | |
Dien et al. | Vietnamese Word Segmentation. | |
US11386269B2 (en) | Fault-tolerant information extraction | |
KR100481598B1 (en) | Apparatus and method for analyzing compounded morpheme | |
US6535886B1 (en) | Method to compress linguistic structures | |
US20010029443A1 (en) | Machine translation system, machine translation method, and storage medium storing program for executing machine translation method | |
Tukur et al. | Tagging part of speech in hausa sentences | |
Marcińczuk et al. | Statistical proper name recognition in Polish economic texts | |
Ramesh et al. | Interpretable natural language segmentation based on link grammar | |
Vasiu et al. | Enhancing tokenization by embedding romanian language specific morphology | |
Okhovvat et al. | An Accurate Persian Part-of-Speech Tagger. | |
Đorđević et al. | Different approaches in serbian language parsing using context-free grammars | |
KR19990015131A (en) | How to translate idioms in the English-Korean automatic translation system | |
JP3698454B2 (en) | Parallel phrase analysis device and learning data automatic creation device |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION |