Go to the first, previous, next, last section, table of contents.


14 Utterances

The utterance structure lies at the heart of Festival. This chapter describes its basic form and the functions available to manipulate it.

14.1 Utterance structure

Festival's basic object for synthesis is the utterance. An represents some chunk of text that is to be rendered as speech. In general you may think of it as a sentence but in many cases it wont actually conform to the standard linguistic syntactic form of a sentence. In general the process of text to speech is to take an utterance which contaisn a simple string of characters and convert it step by step, filling out the utterance structure with more information until a waveform is built that says what the text contains.

The processes involved in convertion are, in general, as follows

Tokenization
Converting the string of characters into a list of tokens. Typically this means whitespace separated tokesn of the original text string.
Token identification
identification of general types for the tokens, usually this is trivial but requires some work to identify tokens of digits as years, dates, numbers etc.
Token to word
Convert each tokens to zero or more words, expanding numbers, abbreviations etc.
Part of speech
Identify the syntactic part of speech for the words.
Prosodic phrasing
Chunk utterance into prosodic phrases.
Lexical lookup
Find the pronucnation of each word from a lexicon/letter to sound rule system including phonetic and syllable structure.
Intonational accents
Assign intonation accents to approrpiate syllables.
Assign duration
Assign duration to each phone in the utterance.
Generate F0 contour (tune)
Generate tune based on accents etc.
Render waveform
Render waveform from phones, duration and F) target values, this itself may take several steps including unit selection (be they diphones or other sized units), imposition of dsesired prosody (duration and F0) and waveform reconstruction.

The number of steps and what actually happens may vary and is dependent on the particular voice selected and the utterance's type, see below.

Each of these steps in Festival is achived by a module which will typically add new information to the utterance structure.

An utterance structure consists of a set of items which may be part of one or more relations. Items represent things like words and phones, though may also be used to represent less concrete objects like noun phrases, and nodes in metrical trees. An item contains a set of features, (name and value). Relations are typically simple lists of items or trees of items. For example the the Word relation is a simple list of items each of which represent a word in the utternace. Those words will also be in other relations, such as the SylStructure relation where the word will be the top of a tree structure containing its syllables and segments.

Unlike previous versions of the system items (then called stream items) are not in any particular relations (or stream). And are merely part of the relations they are within. Importantly this allows much more general relations to be made over items that was allowed in the previous system. This new architecture is the continuation of our goal of providing a general efficient structure for representing complex interrelated utterance objects.

The architecture is fully general and new items and relations may be defined at run time, such that new modules may use any relations they wish. However within our standard English (and other voices) we have used a specific set of relations ass follows.

Token
a list of trees. This is first formed as a list of tokens found in a character text string. Each root's daughters are the Word's that the token is related to.
Word
a list of words. These items will also appear as daughters (leaf nodes) of the Token relation. They may also appear in the Syntax relation (as leafs) if the parser is used. They will also be leafs of the Phrase relation.
Phrase
a list of trees. This is a list of phrase roots whose daughters are the Word's within those phrases.
Syntax
a single tree. This, if the probabilistic parser is called, is a syntactic binary branching tree over the members of the Word relation.
SylStructure
a list of trees. This links the Word, Syllable and Segment relations. Each Word is the root of a tree whose immediate daughters are its syllables and their daughters in turn as its segments.
Syllable
a list of syllables. Each member will also be in a the SylStructure relation. In that relation its parent will be the word it is in and its daughters will be the segments that are in it. Syllables are also in the Intonation relation giving links to their related intonation events.
Segment
a list of segments (phones). Each member (except silences) will be leaf nodes in the SylStructure relation. These may also be in the Target relation linking them to F0 target points.
IntEvent
a list of intonation events (accents and bounaries). These are related to syllables through the Intonation relation as leafs on that relation. Thus their parent in the Intonation relation is the syllable these events are attached to.
Intonation
a list of trees relating syllables to intonation events. Roots of the trees in Intonation are Syllables and their daughters are IntEvents.
Wave
a single item with a feature called wave whose value is the generated waveform.

This is a non-exhaustive list some modules may add other relations and not all utterance may have all these relations, but the above is the general case.

14.2 Utterance types

The primary purpose of types is to define which modules are to be applied to an utterance. UttTypes are defined in `lib/synthesis.scm'. The function defUttType defines which modules are to be applied to an utterance of that type. The function utt.synth is called applies this list of module to an utterance before waveform synthesis is called.

For example when a Segment type Utterance is synthesized it needs only have its values loaded into a Segment relation and a Target relation, then the low level waveform synthesis module Wave_Synth is called. This is defined as follows

(defUttType Segments
  (Initialize utt)
  (Wave_Synth utt))

A more complex type is Text type utterance which requires many more modules to be called before a waveform can be synthesized

(defUttType Text
  (Initialize utt)
  (Text utt)
  (Token utt)
  (POS utt)
  (Phrasify utt)
  (Word utt)
  (Intonation utt)
  (Duration utt)
  (Int_Targets utt)
  (Wave_Synth utt)
)

The Initialize module should normally be called for all types. It loads the necessary relations from the input form and deletes all other relations (if any exist) ready for synthesis.

Modules may be directly defined as C/C++ functions and declared with a Lisp name or simple functions in Lisp that check some global parameter before calling a specific module (e.g. choosing between different intonation modules).

These types are used when calling the function utt.synth and individual modules may be called explicitly by hand if required.

Because we expect waveform synthesis methods to themselves become complex with a defined set of functions to select, join, and modify units we now support an addition notion of SynthTypes like UttTypes these define a set of functions to apply to an utterance. These may be defined using the defSynthType function. For example

(defSynthType Festival
  (print "synth method Festival")
  
  (print "select")
  (simple_diphone_select utt)

  (print "join")
  (cut_unit_join utt)

  (print "impose")
  (simple_impose utt)
  (simple_power utt)

  (print "synthesis")
  (frames_lpc_synthesis utt)
  )

A SynthType is selected by naming as the value of the parameter Synth_Method.

Duration the application of the function utt.synth there are three hooks applied. This allows addition control of the synthesis process. before_synth_hooks is applied before any modules are applied. after_analysis_hooks is applied at the start of Wave_Synth when all text, linguistic and prosodic processing have been done. after_synth_hooks is applied after all modules have been applied. These are useful for things such as, altering the volume of a voice that happens to be quieter than others, or for example outputing information for a talking head before waveform synthesis occurs so preparation of the facial frames and synthesizing the waveform may be done in parallel. (see `festival/examples/th-mode.scm' for an example use of these hooks for a talking head text mode.)

14.3 Example utterance types

A number of utterance types are currently supported. It is easy to add new ones but the standard distribution includes the following.

Text
Raw text as a string.
(Utterance Text "This is an example")
Words
A list of words
(Utterance Words (this is an example))
Words may be atomic or lists if further features need to be specified. For example to specify a word and its part of speech you can use
(Utterance Words (I (live (pos v)) in (Reading (pos n) (tone H-H%))))
Note: the use of the tone feature requires an intonation mode that supports it. Any feature and value named in the input will be added to the Word item.
Phrase
This allows explicit phrasing and features on Tokens to be specified. The input consists of a list of phrases each contains a list of tokens.
(Utterance
 Phrase
 ((Phrase ((name B))
   I saw the man
   (in ((EMPH 1)))
   the park)
  (Phrase ((name BB))
   with the telescope)))
ToBI tones and accents may also be specified on Tokens but these will only take effect if the selected intonation method uses them.
Segments
This allows specification of segments, durations and F0 target values.
(Utterance 
 Segments
 ((# 0.19 )
  (h 0.055 (0 115))
  (@ 0.037 (0.018 136))
  (l 0.064 )
  (ou 0.208 (0.0 134) (0.100 135) (0.208 123))
  (# 0.19)))
Note the times are in seconds NOT milliseconds. The format of each segment entry is segment name, duration in seconds, and list of target values. Each target value consists of a pair of point into the segment (in seconds) and F0 value in Hz.
Phones
This allows a simple specification of a list of phones. Synthesis specifies fixed durations (specified in FP_duration, default 100 ms) and monotone intonation (specified in FP_F0, default 120Hz). This may be used for simple checks for waveform synthesizers etc.
(Utterance Phones (# h @ l ou #))
Note the function SayPhones allows synthesis and playing of lists of phones through this utterance type.
Wave
A waveform file. Synthesis here simply involves loading the file.
(Utterance Wave fred.wav)

Others are supported, as defined in `lib/synthesis.scm' but are used internally by various parts of the system. These include Tokens used in TTS and SegF0 used by utt.resynth.

14.4 Utterance modules

The module is the basic unit that does the work of synthesis. Within Festival there are duration modules, intonation modules, wave synthesis modules etc. As stated above the utterance type defines the set of modules which are to be applied to the utterance. These modules in turn will create relations and items so that ultimately a waveform is generated, if required.

Many of the chapters in this manual are solely concerned with particular modules in the system. Note that many modules have internal choices, such as which duration method to use or which intonation method to use. Such general choices are often done through the Parameter system. Parameters may be set for different features like Duration_Method, Synth_Method etc. Formerly the values for these parameters were atomic values but now they may be the functions themselves. For example, to select the Klatt duration rules

(Parameter.set 'Duration_Method Duration_Klatt)

This allows new modules to be added without requiring changes to the central Lisp functions such as Duration, Intonation, and Wave_Synth.

14.5 Accessing an utterance

There are a number of standard functions that allow one to access parts of an utterance and traverse through it.

Functions exist in Lisp (and of course C++) for accessing an utterance. The Lisp access functions are

`(utt.relationnames UTT)'
returns a list of the names of the relations currently created in UTT.
`(utt.relation.items UTT RELATIONNAME)'
returns a list of all items in RELATIONNAME in UTT. This is nil if no relation of that name exists. Note for tree relation will give the items in pre-order.
`(utt.relation_tree UTT RELATIONNAME)'
A Lisp tree presentation of the items RELATIONNAME in UTT. The Lisp bracketing reflects the tree structure in the relation.
`(utt.relation.leafs UTT RELATIONNAME)'
A list of all the leafs of the items in RELATIONNAME in UTT. Leafs are defined as those items with no daughters within that relation. For simple list relations utt.relation.leafs and utt.relation.items will return the same thing.
`(utt.relation.first UTT RELATIONNAME)'
returns the first item in RELATIONNAME. Returns nil if this relation contains no items
`(utt.relation.last UTT RELATIONNAME)'
returns the last (the most next) item in RELATIONNAME. Returns nil if this relation contains no items
`(item.feat ITEM FEATNAME)'
returns the value of feature FEATNAME in ITEM. FEATNAME may be a feature name, feature function name, or pathname (see below). allowing reference to other parts of the utterance this item is in.
`(item.features ITEM)'
Returns an assoc list of feature-value pairs of all local features on this item.
`(item.name ITEM)'
Returns the name of this ITEM. This could also be accessed as (item.feat ITEM 'name).
`(item.set_name ITEM NEWNAME)'
Sets name on ITEM to be NEWNAME. This is equivalent to (item.set_feat ITEM 'name NEWNAME)
`(item.set_feat ITEM FEATNAME FEATVALUE)'
set the value of FEATNAME to FEATVALUE in ITEM. FEATNAME should be a simple name and not refer to next, previous or other relations via links.
`(item.relation ITEM RELATIONNAME)'
Return the item as viewed from RELATIONNAME, or nil if ITEM is not in that relation.
`(item.relationnames ITEM)'
Return a list of relation names that this item is in.
`(item.relationname ITEM)'
Return the relation name that this item is currently being viewed as.
`(item.next ITEM)'
Return the next item in ITEM's current relation, or nil if there is no next.
`(item.prev ITEM)'
Return the previous item in ITEM's current relation, or nil if there is no previous.
`(item.parent ITEM)'
Return the parent of ITEM in ITEM's current relation, or nil if there is no parent.
`(item.daughter1 ITEM)'
Return the first daughter of ITEM in ITEM's current relation, or nil if there are no daughters.
`(item.daughter2 ITEM)'
Return the second daughter of ITEM in ITEM's current relation, or nil if there is no second daughter.
`(item.daughtern ITEM)'
Return the last daughter of ITEM in ITEM's current relation, or nil if there are no daughters.
`(item.leafs ITEM)'
Return a list of all lefs items (those with no daughters) dominated by this item.
`(item.next_leaf ITEM)'
Find the next item in this relation that has no daughters. Note this may traverse up the tree from this point to search for such an item.

As from 1.2 the utterance structure may be fully manipulated from Scheme. Relations and items may be created and deleted, as easily as they can in C++;

`(utt.relation.present UTT RELATIONNAME)'
returns t if relation named RELATIONNAME is present, nil otherwise.
`(utt.relation.create UTT RELATIONNAME)'
Creates a new relation called RELATIONNAME. If this relation already exists it is deleted first and items in the relation are derefenced from it (deleting the items if they are no longer referenced by any relation). Thus create relation guarantees an empty relation.
`(utt.relation.delete UTT RELATIONNAME)'
Deletes the relation called RELATIONNAME in utt. All items in that relation are derefenced from the relation and if they are no longer in any relation the items themselves are deleted.
`(utt.relation.append UTT RELATIONNAME ITEM)'
Append ITEM to end of relation named RELATIONNAME in UTT. Returns nil if there is not relation named RELATIONNAME in UTT otherwise returns the item appended. This new item becomes the last in the top list. ITEM item may be an item itself (in this or another relation) or a LISP description of an item, which consist of a list containing a name and a set of feature vale pairs. It ITEM is nil or inspecified an new empty item is added. If ITEM is already in this relation it is dereferenced from its current position (and an emtpy item re-inserted).
`(item.insert ITEM1 ITEM2 DIRECTION)'
Insert ITEM2 into ITEM1's relation in the direction specified by DIRECTION. DIRECTION may take the value, before, after, above and below. If unspecified, after is assumed. Note it is not recommended to insert above and below and the functions item.insert_parent and item.append_daughter should normally be used for tree building. Inserting using before and after within daughters is perfectly safe.
`(item.append_daughter PARENT DAUGHTER)'
Append DAUGHTER, an item or a description of an item to the item PARENT in the PARENT's relation.
`(item.insert_parent DAUGHTER NEWPARENT)'
Insert a new parent above DAUGHTER. NEWPARENT may be a item or the description of an item.
`(item.delete ITEM)'
Delete this item from all relations it is in. All daughters of this item in each relations are also removed from the relation (which may in turn cause them to be deleted if they cease to be referenced by any other relation.
`(item.relation.remove ITEM)'
Remove this item from this relation, and any of its daughters. Other relations this item are in remain untouched.
`(item.move_tree FROM TO)'
Move the item FROM to the position of TO in TO's relation. FROM will often be in the same relation as TO but that isn't necessary. The contents of TO are dereferenced. its daughters are saved then descendants of FROM are recreated under the new TO, then TO's previous daughters are derefenced. The order of this is important as FROM may be part of TO's descendants. Note that if TO is part of FROM's descendants no moving occurs and nil is returned. For example to remove all punction terminal nodes in the Syntax relation the call would be something like
(define (syntax_relation_punc p)
  (if (string-equal "punc" (item.feat (item.daughter2 p) "pos"))
      (item.move_tree (item.daughter1 p) p)
  (mapcar syntax_remove_punc (item.daughters p))))
`(item.exchange_trees ITEM1 ITEM2)'
Exchange ITEM1 and ITEM2 and their descendants in ITEM2's relation. If ITEM1 is within ITEM2's descendents or vice versa nil is returns and no exchange takes place. If ITEM1 is not in ITEM2's relation, no exchange takes place.

Daughters of a node are actually represented as a list whose first daughter is double linked to the parent. Although being aware of this structure may be useful it is recommended that all access go through the tree specific functions *.parent and *.daughter* which properly deal with the structure, thus is the internal structure ever changes in the future only these tree access function need be updated.

With the above functions quite elaborate utterance manipulations can be performed. For example in post-lexical rules where modifications to the segments are required based on the words and their context. See section 13.8 Post-lexical rules, for an example of using various utterance access functions.

14.6 Features

In previous versions items had a number of predefined features. This is no longer the case and all features are optional. Particularly the start and end features are no longer fixed, though those names are still used in the relations where yjeu are appropriate. Specific functions are provided for the name feature but they are just short hand for normal feature access. Simple features directly access the features in the underlying EST_Feature class in an item.

In addition to simple features there is a mechanism for relating functions to names, thus accessing a feature may actually call a function. For example the features num_syls is defined as a feature function which will count the number of syllables in the given word, rather than simple access a pre-existing feature. Feature functions are usually dependent on the particular realtion the item is in, e.g. some feature functions are only appropriate for items in the Word relation, or only appropriate for those in the IntEvent relation.

The third aspect of feature names is a path component. These are parts of the name (preceding in .) that indicated some trversal of the utterance structure. For example the features name will access the name feature on the given item. The feature n.name will return the name feature on the next item (in that item's relation). A number of basic direction operators are defined.

n.
next
p.
previous
nn.
next next
pp.
previous
parent.
daughter1.
first daughter
daughter2.
second daughter
daughtern.
last daughter
first.
most previous item
last.
most next item

Also you may specific traversal to another relation relation, though the R:<relationame>. operator. For example given an Item in the syllable relation R:SylStructure.parent.name would give the name of word the syllable is in.

Some more complex examples are as follows, assuming we are starting form an item in the Syllable relation.

`stress'
This item's lexical stress
`n.stress'
The next syllable's lexical stress
`p.stress'
The previous syllable's lexical stress
`R:SylStructure.parent.name'
The word this syllable is in
`R:SylStructure.parent.R:Word.n.name'
The word next to the word this syllable is in
`n.R:SylStructure.parent.name'
The word the next syllable is in
`R:SylStructure.daughtern.ph_vc'
The phonetic feature vc of the final segment in this syllable.

A list of all feature functions is given in an appendix of this document. See section 32 Feature functions. New functions may also be added in Lisp.

In C++ feature values are of class EST_Val which may be a string, int, or a float (or any arbitrary object). In Scheme this distinction cannot not always be made and sometimes when you expect an int you actually get a string. Care should be take to ensure the right matching functions are use in Scheme. It is recommended you use string-append or string-match as they will always work.

If a pathname does not identify a valid path for the particular item (e.g. there is no next) "0" is returned.

When collecting data from speech databases it is often useful to collect a whole set of features from all utterances in a database. These features can then be used for building various models (both CART tree models and linear regression modules use these feature names),

A number of functions exist to help in this task. For example

(utt.features utt1 'Word '(name pos p.pos n.pos))

will return a list of word, and part of speech context for each word in the utterance.

See section 26.2 Extracting features, for an example of extracting sets of features from a database for use in building stochastic models.

14.7 Utterance I/O

A number of functions are available to allow an utterance's structure to be made available for other programs.

The whole structure, all relations, items and features may be saved in an ascii format using the function utt.save. This file may be reloaded using the utt.load function. Note the waveform is not saved using the form.

Individual aspects of an utterance may be selectively saved. The waveform itself may be saved using the function utt.save.wave. This will save the waveform in the named file in the format specified in the Parameter Wavefiletype. All formats supported by the Edinburgh Speech Tools are valid including nist, esps, sun, riff, aiff, raw and ulaw. Note the functions utt.wave.rescale and utt.wave.resample may be used to change the gain and sample frequency of the waveform before saving it. A waveform may be imported into an existing utterance with the function utt.import.wave. This is specifically designed to allow external methods of waveform synthesis. However if you just wish to play an external wave or make it into an utterance you should consider the utterance Wave type.

The segments of an utterance may be saved in a file using the function utt.save.segs which saves the segments of the named utterance in xlabel format. Any other stream may also be saved using the more general utt.save.relation which takes the additional argument of a relation name. The names of each item and the end feature of each item are saved in the named file, again in Xlabel format, other features are saved in extra fields. For more elaborated saving methods you can easily write a Scheme function to save data in an utterance in whatever format is required. See the file `lib/mbrola.scm' for an example.

A simple function to allow the displaying of an utterance in Entropic's Xwaves tool is provided by the function display. It simply saves the waveform and the segments and sends appropriate commands to (the already running) Xwaves and xlabel programs.

A function to synthesize an externally specified utterance is provided for by utt.resynth which takes two filename arguments, an xlabel segment file and an F0 file. This function loads, synthesizes and plays an utterance synthesized from these files. The loading is provided by the underlying function utt.load.segf0.


Go to the first, previous, next, last section, table of contents.