Conversational Speech Translation - Challenges and Techniques, by Chris Wendt, Microsoft

Editor's Notes

1. As we all know the idea of being able to speak naturally with someone who doesn’t understand your language has been a long-held dream. Whether we’re talking about the biblical story of the tower of babel, or 20th century sci-fi such as the Star Trek universal communicator or Douglas Adam’s “babelfish”
2. About this time last year, we set ourselves a goal of trying to turn this age-old dream into reality. We realized that there was a confluence of factors that taken together gave us the opportunity to make this happen. We ourselves in the MT field have been making steady progress on MT quality both with better algorithms and by applying ever greater amounts of data, such that our best MT systems today are really quite good. At the same time, the ASR field has seen a technological leap over the last few years with the use of DNNs leading to dramatically lower errors rates. And finally, we at Microsoft Research, felt we had a golden opportunity not just to do a great technology demo, but to actually put this in the hands of 100s of Ms of users through Skype. In order to achieve this we faced a number of challenges, which is what I will be talking about for the next hour.
3. On one hand: Help Skype users On the other hand: have a generalized speech translation system and service for 3rd parties.
4. There are three main challenges here that we need to go through to have a realistic S2S system : The gulf between speech and text It’s not enough to just chain a really good ASR system with a really good MT system How people talk to each other is not how they write Building really good conversational ASR and MT systems Significant changes in the data we use to train the ASR and MT systems. The gap between technology demo and consumer product Plugging into Skype Interesting problems one encounters with real consumers
5. [READ SLIDE] So, if we take the raw ASR output and just throw it at MT, it doesn’t work so well. We need components to process the ASR, remove disfluencies, etc and make it more palatable to MT. Likewise we need to adapt MT to handle this kind of input
6. So let’s take a closer look at the different types of disfluencies – first you have uh, your, um, fillers, then, you know, I mean, your discourse markers and and and repetition, and finally correct--, I mean, speech repairs, where people go back and repeat, I mean, change what they said. In this example, here the speaker changed no to yes, and “I am” to “I have”.
7. So let’s take a closer look at the different types of disfluencies – first you have uh, your, um, fillers, then, you know, I mean, your discourse markers and and and repetition, and finally correct--, I mean, speech repairs, where people go back and repeat, I mean, change what they said. In this example, here the speaker changed no to yes, and “I am” to “I have”.
8. Another thing that is missing is punctuation -- If we don’t get punctuation right, we are risking a lot more than just word salad. You may know the example “Let’s eat grandma”, where a missing comma could lead to a tragic outcome. When translating, the problem gets worse.
9. In the case of Spanish (and possibly other languages), we have an additional problem with accented words [READ SLIDE]
10. In addition to *how* people speak (genre), there’s also a big difference in *what* the talk about (domain).
11. Now let’s take a closer look on Speech Correction component which helps in bridging the gap between spoken and written text.
12. [READ SLIDE] Adapting MT to ASR starts with building a good baseline conversationally-oriented MT system
13. Now let’s take a closer look on Speech Correction component which helps in bridging the gap between spoken and written text.
14. Speech Correction component helps in bridging the gap between spoken and written text.
15. Let’s listen to what the user said. Very good fluent English. But if we read the english transcripts, it is hard to understand it, not to mention the translation. So, if we take the raw ASR output and just throw it at MT, it doesn’t work so well. We need components to process the ASR, remove disfluencies etc and make it more palatable to MT. Likewise we need to adapt MT to handle this kind of input Missing sentence boundaries, punctuation, casing and disfluency removal
16. First we know who is talking to who, we can have user profiles for both of them that can enable us to do better job in both recognition and translation. Personalization and customization plays crucial role in open domain S2S since people are usually talking about broadly different things. For example here we can recognize a person name “gurdeep” rather than “go deep” Good customization and personalization is very crucial for open domain Skype Translator, people can talk about their palnned vacation to Columbia to Spanish speaker which needs different vocabulary than talking to a Chinese supplier next day next product plans.
17. Open domain conversational S2S can benefit from customizing and personalizing the models according to the users profiles. We can use users’ profile to create customized models that can fir their topics and vocabulary. Describe the diagram above. Currency we use this infra-structure for contact names recognition.
18. One of the issues with ASR is that the vocab cannot include every possible person name (or place name etc). Expanding the vocab drastically to include millions of names can compromise WER, because names may be misrecognized in place of regular words. However in the Skype Translator, we found that when the system didn’t recognize the caller or callee’s name at the start of a call, it often derailed the entire conversation. So we opted for a surgical fix for now while we investigate more broad-based options. What we’ve done is added a very small restrictive grammar comprised of common greetings etc, but with placeholders for names. At the start of a call we dynamically compile the contact names for the current caller and callee into this grammar, and our ASR engine can use this grammar in parallel with it’s broad based regular LM.
19. Instead of using one best output from ASR, we can use n-best in lattice format. Lattice rescoring can help us in getting many possible alternatives form SR and then use very large LM to score them
20. In the early days we were fixated on WER and it’s effect on BLEU and the error cascade, meaning that if we were to pipeline multiple error-prone components together, the errors would multiply. This is a well-studied problem that other researchers have studied for a number of years. One approach that’s been tried is to take the ASR lattice directly as input to the MT decoder. This has been studied by many groups and is conceptually elegant, but the implementation is quite complex. A lattice representation allows an MT system to arbitrate between multiple ambiguous hypotheses from upstream processing so that the best translation can be produced. A simplification is decoding over a confusion network, where the ASR confusables are compactly encoded as a “word sausage”. This is very easy to decode in MT because it affects mostly just the phrase-lookup portion, leaving the rest of the decoding untouched except for some extra features. We found that the MT portion of this worked well. However collapsing an ASR-lattice into a confusion network is an ill-defined operation which can result in some nasty artifacts in the confusion network such as a proliferation of epsilon arcs. After some experimentation with N-best rescoring and confusion networks, we decided to try a couple of different things. Confusion network: A Confusion Network (CN), also known as a sausage, is a weighted directed graph with the peculiarity that each path from the start node to the end node goes through all the other nodes. Speech Lattice: Maintains a set of candidates as a subset of models.
21. We decided that before we plunged into full-fledged MT decoding over lattices, we would first try simply rescoring the ASR lattice with a much bigger LM, adding some extra MT-friendly features and tuning model weights. We discovered we could get very good ASR and end-to-end BLEU gains, at which point we decided not to bother with decoding over lattices in the MT decoder itself
22. Now we have “almost” perfect transcription that match what is the user said, improved and customized as well. Are we ready to send this to MT. No yet. Disfleuncy handling should be done before we translate. But actually disfluence removal and punctuation need to be done concurrently.
23. Finally, we come into segmentation into sentence units, punctuation and casing which should be ready for a state-of-the art MT system to produce reliable translation.
24. So let’s take a closer look at the different types of disfluencies – first you have uh, your, um, fillers, then, you know, I mean, your discourse markers and and and repetition, and finally correct--, I mean, speech repairs, where people go back and repeat, I mean, change what they said. In this example, here the speaker changed no to yes, and “I am” to “I have”.
25. Traditionally this problem has been solved by first doing segmentation and then disfluency removal But there is an interaction between segmentation and disfluency handling. If segmentation is done first, disfluency will lose a chance to make a better correction, and we’ll be left with numerous disfluent fragments. On the other hand, complex disfluency removal (speech repairs) needs sentence boundaries, so you can’t do that first either. What we did is split the difference. We do some simple disfluency is first, then segmentation, then complex disfluency removal.
26. Conditional random fields (CRFs) are a class of statistical modelling method often applied in pattern recognition and machine learning, where they are used for structured prediction. Whereas an ordinary classifier predicts a label for a single sample without regard to "neighboring" samples, a CRF can take context into account; e.g., the linear chain CRF popular in natural language processing predicts sequences of labels for sequences of input samples. First remove simple disfluencies, then segment, then remove complex disfluencies First two stages are CRF sequence taggers Complex disfluency handling: Uses metadata annotated by previous stages Using iterative parsing (NLPwin parser) Needs sentence units
27. Brown clustering is a hard hierarchical agglomerative clustering problem based on distributional information. It is typically applied to text, grouping words into clusters that are assumed to be semantically related by virtue of their having been embedded in similar contexts.
28. In complex disfluency removal we take advantage of the NLPwin parser that was built for the MS Word grammar checker and so is robust to ungrammatical input. For example here we have a repeated subtree that is linguistically similar, and so the first subtree is removed. We also look for are constituents that  appear to be disconnected from other parts of the tree. When we spot a disfluency we remove it and reparse the resulting sentence because the removal of the disfluency could change the entire parse. We remove errors one by one and stop when we have no more edits or we hit a limit on the number of parses.
29. Lots of numbers here. Looking at the last column which is what we care about, here are the takeaways Sentence breaking based on speaker pauses is a bad idea (lose 0.5 BLEU points) CRF sentence breaking by itself adds about 0.5 BLEU (vs no breaking at all i.e. translate the full utterance) Disfluency removal by itself adds about 1.3 BLEU Doing both gives you about 2 BLEU points and doing the split before/after adds another 0.5
30. This is Vinny, who participated in the Mystery Skype session we had with the schools in Beijing. Vinny’s deaf, so it was wonderful for him to participate in these calls with his classmates. Even when he was unable to hear the response back from the students, he could read the translation of what they were saying.
31. Although the use here demonstrates the use of the technology with deaf or hard of hearing students, it’s not much of a stretch to adapt the technology, since the components already exist, to hearing students that speak other languages. In fact, it could be used in that manner now. We haven’t tested it in this scenario…yet.