Does it work at all?

It is very easy to build a voice and get it to say a few phrases and think that the job is done. As you build the voice it is worth testing each part as you built it to ensure it basically performs as expected. But once its all together more general tests are needed. Before you submit it to any formal tests that you will use for benchmarking and grading progrees in the voice, more basic tests should be carried out.

In fact it is stating such initial tests more concretely. Every we have ever built has always had a number mistakes in it that can be trivially fixed. Such as the mfccs were not generated after fixing the pitchmarks. Therefore you syould go through each stage of the build procedure and ensure it really did do what you though it should do, especially if you are totally convinced that section worked perfectly.

Try to find around 100-500 sentences to play through it. It is amazing home many general problems are thrown up when you extend your test set. The next stage is to play so real text. That may be news text from the web, output from your speech translation system, or some email. Initially it is worth just synthesizing the whole set without even listening to it. Problems in analysis and missing diphones etc may be shown up just in the processing of the text. Then you want to listen to the output and identify problems. This make take some amount of investigation. What you want to do is identify where the problem is, is it bad tex analysis, bad lexical entry, a prosody problem, or a waveform synthesis problem. You may need to synthesizes parts of the text in isolation (e.g. using the Festival function SayText and look at the structure of the utterance generated, e.g. using the function utt.features. For example to see what words have been identified from the text analysis

(utt.features utt1 'Word '(name))

Or to see the phones generated

(utt.features utt1 'Segment '(name))

Thus you can view selected parts of an utterance and find out if it is being created as you intended. For some things a graphical display of the utterance may help.

Once you identify where the problem is you need to decide how to fix it (or if it is worth fixing). The problem may be a number of different places:

• Phonetic error: the acoustics of a unit doesn't match the label. This may be because the speaker said the wrong word/phoneme or the labeller had the wrong. Or possible some other acoustic variant that has not been considered

• Lexical error: the word is pronounced with the wrong string of phonemes/stress/tone. Either the lexical entry is wrong or the letter to sound rules are not doing ht right thing. Or there are multiple valid pronunciations for that word (homographs) and the wrong one is selectec because the homograph disambiguation is wrong, or there is not a disambiguator.

• Text error: the text analysis doesn't deal properly with the word. It may be that a punctuation system is spoken (or not spoken) as expected, titles, symbols, compounds etc aren't dealt with properly

• Some other error: some error that is not one of the above. As you progress in correction and tuningm errors in the category will grow and you must find some way to avoid such errors.

Before rushing out and getting one hundred people to listen to your new synthetic voice, it is worth doing significant internal testing and evaluation, informally to find errors and test them. Remember the purpose of evaluation in this case is to find errors and fix them. We are not, at least not at this stage, evaluating the voices on an abstract scale, where unseen test data, and blind testing is important.