Search Images Maps Play YouTube News Gmail Drive More »
Sign in
Screen reader users: click this link for accessible mode. Accessible mode has the same essential features but works better with your reader.

Patents

  1. Advanced Patent Search
Publication numberUSRE42647 E1
Publication typeGrant
Application numberUS 10/193,594
Publication dateAug 23, 2011
Filing dateSep 30, 2002
Priority dateMay 8, 1997
Also published asDE19753454A1, DE19753454C2, US6088673
Publication number10193594, 193594, US RE42647 E1, US RE42647E1, US-E1-RE42647, USRE42647 E1, USRE42647E1
InventorsJung Chul Lee, Min Soo Hahn, Hang Seop Lee, Jae Woo Yang, YoungJik Lee
Original AssigneeElectronics And Telecommunications Research Institute
Export CitationBiBTeX, EndNote, RefMan
External Links: USPTO, USPTO Assignment, Espacenet
Text-to speech conversion system for synchronizing between synthesized speech and a moving picture in a multimedia environment and a method of the same
US RE42647 E1
Abstract
The present invention provides a text-to-speech conversion system (TTS) for interlockingsynchronizing with multimedia and a method for organizing input data of the TTS which can enhance the naturalnaturalness of synthesized speech and accomplish the synchronization of multimedia with TTS by defining additional prosody information, the information required to interlocksynchronize TTS with multimedia, and interface between thesethis information and TTS for use in the production of the synthesized speech.
Images(3)
Previous page
Next page
Claims(32)
1. A text-to-speech conversion system for interlocking with multimedia comprising;
a multimedia information input unit for organizing text, prosody information, information on synchronization with a moving picture, lip-shape information, picture information, and individual property information including a gender, age, accent, pronunciation and speech rate of synthesized speech;
a data distributor for distributing the information from said multimedia information input unit into information for each media;
a language processor for converting the text distributed by said data distributor into a phoneme stream, presuming prosody information and symbolizing the presumed prosody information;
a prosody processor for calculating a prosody control parameter value from the symbolized prosody information from the language processor;
a synchronization adjuster for adjusting a duration of each phoneme using the synchronization information distributed by said data distributor;
a synthesis unit database for receiving the individual property information from said data distributor, selecting synthesis units adaptable to gender and age and outputting data required for synthesis;
a signal processor for producing a synthesized speech using the prosody control parameter and the data output from said synthesis unit database; and
a picture output apparatus for outputting the picture information distributed by said data distributor onto a screen.
2. A method for organizing input data of a text-to-speech conversion system for interlocking with multimedia, said method comprising the steps of:
(a) classifying multimedia input information organized for enhancing natural synthesized speech and implementing synchronization of multimedia with text-to-speech into text, prosody information, information on synchronization with a moving picture, lip-shaped information, picture information, and individual property information using a multimedia information input unit;
(b) distributing using a data distributor the multimedia input information classified in the multimedia information input unit based on respective information;
(c) converting the text distributed by the data distributor into a phoneme stream, presuming prosody information and symbolizing the presumed prosody information using a language processor;
(d) calculating a prosody control parameter value which is not included in the multimedia input information using a prosody processor;
(e) adjusting a duration of each phoneme using a synchronization adjuster so as to synchronize a processing result of the prosody processor with a picture signal according to the synchronization information distributed by the data distributor;
(f) selecting synthesis units adaptable to gender and age based on the individual property information from the data distributor using a synthesis unit database and outputting data required for synthesis;
(g) producing synthesized speech using a signal processor based on the prosody information distributed by the data distributor, a processing result of the synchronization adjuster, and the data from the synthesis unit database; and
(h) outputting the picture information distributed by the data distributor onto a screen using a picture output unit.
3. The method in accordance with claim 2, wherein the organized multimedia information comprises text information, prosody information, information on synchronization with a moving picture, lip-shaped information, and individual property information.
4. The method in accordance with claim 3, wherein the prosody information comprises a number of phoneme, phoneme stream information, duration of each phoneme, pitch pattern of the phoneme, and energy pattern of the phoneme.
5. The method in accordance with claim 4, wherein the duration time of the phoneme is indicative of a value of pitch at a beginning point, a mid point, and an end point within the phoneme.
6. The method in accordance with claim 5, wherein the energy pattern of the phoneme is indicative of a value of energy in decibels at the beginning point, the mid point, and the end point within the phoneme.
7. The method in accordance with claim 3, wherein the synchronization information comprises text, lip-shape, location information with a moving picture, and duration information.
8. The method in accordance with claim 3, wherein the synchronization information comprises a beginning point, duration and delay time information of a starting point, and duration of each phoneme is controlled by the synchronization information.
9. The method in accordance with claim 3, wherein the synchronization information is composed of a duration of a beginning point of a sentence, a duration information of a starting point, and duration of each phoneme is controlled by forecast lip-shape considered an articulation manner of the phoneme and articulation control of lip-shape within the synchronization and duration information of the synchronization information.
10. The method in accordance with claim 3, wherein the synthesized speech is produced based on beginning point information, end point information, and phoneme information for each phoneme within an interval associated with a speech signal.
11. The method in accordance with claim 3, wherein the synthesized speech is produced based on a distance of an opening between an upper lip and a lower lip, a distance between end points of the lips, and an extent of projection of a lip, and a lip-shape quantized and normalized pattern is defined depending on articulation location and articulation manner of the phoneme on a basis of pattern with discriminative property.
12. The method in accordance with claim 3, wherein if the multimedia input information comprises prosody information, further comprising the steps of:
(i) converting the prosody information into a data structure recognizable by the signal processor; and
(j) transmitting the converted prosody information to the prosody processor and the synchronization adjustor.
13. The method in accordance with claim 3, wherein if the multimedia input information includes individual property information, further comprising the steps of:
(k) converting the individual property information into a data structure recognizable by the synthesis unit database and the prosody processor within the text-to-speech;
(l) transmitting the converted individual property information to the synthesis unit database and the prosody processor.
14. A text-to-speech conversion system (TTS) for synchronizing synthesized speech and a moving picture which is to be displayed on a picture output apparatus which is connected with the TTS, the TTS including a language processor for converting the text into phoneme stream and presuming prosody information from the phoneme stream; a prosody processor for calculating prosody control parameter values from the prosody information using a predefined rule; and a signal processor for producing synthesized speech using the prosody control parameter values and synthetic data stored in a synthesis unit database, characterized in that the TTS comprises:
a multimedia information input unit for inputting a set of multimedia information, the set of multimedia information including moving picture information, text information, and synchronization information;
a data distributor for classifying the set of multimedia information into a plurality of subsets of the multimedia information to distribute each subset of the multimedia information into a corresponding one of the language processor, prosody processor, signal processor and the picture output apparatus; and
a synchronization adjuster for adjusting the duration of each phoneme in the phoneme stream using the synchronization information subset distributed by said data distributor to synchronize between synthesized speech to be produced by the signal processor and the moving picture to be displayed on the picture output apparatus.
15. A method for synchronizing synthesized speech generated from a TTS and a moving picture which is to be displayed on a picture output apparatus which is connected with the TTS, the method comprising the steps of:
receiving a set of multimedia information which includes text information, moving picture information and synchronization information;
classifying the set of the received multimedia information into a plurality of subsets of the information including a synchronization information subset;
converting each classified text information subset into a phoneme stream;
presuming prosody information from each phoneme stream;
calculating prosody control parameter values based on the prosody information;
adjusting the duration of each phoneme of each phoneme stream using the respective classified synchronization information subset to synchronize between the synthesized speech and the moving picture; and
producing the synthesized speech using the prosody control parameter values and data in a synthesis unit database, in synchronism with the moving picture to be displayed.
16. The method according the claim 15, wherein said prosody control parameters are comprised of the number of phonemes, duration time of each phoneme, pitch pattern of each phoneme and energy pattern of each phoneme.
17. The method according to claim 16, wherein said pitch pattern of each phoneme is indicative of a value of pitch of the desired synthesized speech at a beginning point, a middle point, and an end point within each phoneme.
18. The method according to claim 16, wherein said energy pattern of each phoneme is indicative of a value of energy of the desired synthesized speech in decibels at a beginning point, a mid point and an end point within each phoneme.
19. The system according to claim 14, wherein said synchronization information subset includes lip-shape information, said lip-shape information including a number of lip-shape change points, a location of lip-shape change points in a sentence and a lip-shape representation at every lip-shape change point.
20. The system according to claim 14, wherein a set of said multimedia information further includes individual property information, said individual property information subset including gender and age information of the synthesized speech.
21. The system according to claim 14, wherein if a set of said multimedia information further includes prosody control parameters, said prosody control parameters are capable of being utilized in said synchronization adjuster without the processing of said language processor and prosody processor.
22. The system according the claim 21, wherein said prosody control parameters include the number of phonemes in the data stream, a duration time of each phoneme, a pitch pattern of each phoneme and an energy pattern of each phoneme.
23. The system according to claim 22, wherein said pitch pattern of each phoneme is indicative of a value of pitch of the desired synthesized speech at a beginning point, a middle point, and an end point within each phoneme.
24. The system according to claim 22, wherein said energy pattern of each phoneme is indicative of a value of energy of the desired synthesized speech in decibels at a beginning point, a mid point and an end point within each phoneme.
25. The method of claim 15, wherein the classified synchronization information subset includes lip-shape information, said lip-shape information including a number of lip-shape change points, a location of lip-shape change points in a sentence and a lip-shape representation at every lip-shape change point.
26. The method according to claim 15, wherein said set of the received multimedia information further includes an individual property information subset, said individual property information subset including gender and age information of the synthesized speech.
27. A method for synchronizing synthesized speech generated from a TTS and a moving picture which is to be displayed on a picture output apparatus which is connected with the TTS, the method comprising the steps of:
receiving a set of multimedia information which includes text information, moving picture information, synchronization information and prosody control parameters, said prosody control parameters including a duration of each phoneme;
classifying a set of the received multimedia information into a plurality of subsets of the information including a synchronization information subset;
adjusting the duration of each phoneme using the classified synchronization information subset to synchronize between the synthesized speech and the moving picture; and
producing the synthesized speech using the prosody control parameter values included in a set of the received multimedia information and data in a synthesis unit database, in synchronism with the moving picture to be displayed onto screen in the picture output apparatus.
28. A process for producing synthesized speech in synchronism with an associated moving picture characterized in that the process comprises receiving a set of multimedia information including text information, moving picture information and synchronization information; and synthesizing speech from the received text information in synchronization with the received moving picture information using the received synchronization information.
29. A speech synthesizer for use in synchronizing synthesized speech generated from a TTS and a moving picture which is to be displayed on a picture output apparatus which is connected with the TTS, the speech synthesizer comprising:
means for receiving prosody control parameters including a duration of each phoneme, synchronization information and moving picture data;
means for adjusting the duration of each phoneme of each phoneme stream using the synchronization information to synchronize between the synthesized speech and the moving picture; and
means for producing the synthesized speech using the prosody control parameter values and data in a synthesis unit database, in synchronism with the moving picture to be displayed.
30. A synthesizer for producing a synthesized speech using a text information, comprising:
receiving means for receiving the text information usable for synthesizing speech, and synchronization information including a number of lip-shape change points and a lip-shape representation at every lip-shape change point;
synthesizing means, for producing the synthesized speech from the text information using the synchronization information; and
outputting means for outputting the synthesized speech.
31. The synthesizer according to claim 30, wherein a picture output apparatus is connected to the synthesizer and operated in synchronization with the synthesizer.
32. The synthesizer according to claim 31, wherein the synchronization information is related to a moving picture outputted from the picture output apparatus.
Description
BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a text-to-speech conversion systems (hereinafter, referred to as TTS) for interlocking with multimediasynchronizing synthesized speech and a moving picture and a method for organizing input data of the same, and more particularly to a text-to-speech conversion system ( TTS ) for interlocking with multimediasynchronizing synthesized speech and a moving picture and a method for organizing input data of the same for enhancing the naturalnaturalness of the synthesized speech and accomplishing synchronization between multimedia and TTS by defining additional prosody information, the information required to interlock TTS with multimedia, and interface between these information and TTS for use in the production of the synthesized speech.

2. Description of the Related Art

Generally, the function of thea speech synthesizer is to provide different forms of information for a man using a computer user. To this end, the speech synthesizer should serve theprovide the user with synthesized speech with high quality from a given text. In addition, for the interlock with database produced in multimedia environment such as moving picture or animation, or a variety of media provided from a counterpart of conversionPreferably, the speech synthesizer should also produce the synthesized speech to be synchronized with theses mediavideo data such as a moving picture, animation and so on. ParticularlyIn particular, the synchronization function of TTS within the multimedia environment is essential to provide the user with high quality service with high quality.

As shown in FIG. 1, typically, a conventional TTS goes through the process consisting of 3 steps as follows until the synthesized speech is produced from on inputted text.

In a first step, a language processor 1 converts the text into a series of phoneme, presumes prosody information and symbolizes this information. Symbol of prosody information is presumed from a boundary of the phrase and paragraph, a location of accent in word, a sentence pattern, and so on using the analysis result of syntax.

In a second step, a prosody processor 2 calculates a value of prosody control parameter from the symbolized prosody information using a rule and a table. Prosody control parameter includes duration of phoneme, pitch contour, energy contour, and pause interval information.

In a third step, a signal processor 3 produces a synthesized speech using a synthesis unit database 4 and the prosody control parameter. In other words, this means that the conventional TTS should presume the information associated with the natural speech rate in the language processor 1 and the prosody processor 2 only by the inputted text.

Further, the conventional TTS has simple function to output data inputted by the unit of sentence as the synthesized speech. Accordingly, in order to output sentences stored in a file or sentences inputted through a communication network as the synthesized speech in succession, a main control program which reads sentences from the inputted data and transmits them to an input of TTS is required. Such a main control program includes a method to separate the text from the inputted data and then output the synthesized speech once from the beginning to the end, a method to produce the synthesized speech in interlock with a text editor, a method to look up the sentences by use of a graphic interface and produce the synthesized speech, and so on, but the object to which these methods are applicable is restricted to the text.

At present, studies on TTS have considerably advanced for the vernacular language in different countries and a commercial use has been accomplished in some countries. However, this is in situation of the only use for the syntheses of speech from the inputted text. In addition, by a prior organization, since it is impossible to presume from only the text the information required when moving picture is to be dubbed by use of TTS or when the natural interlock between the synthesized speech and multimedia such as animation is to be implemented, there is no method to realize these functions. Furthermore, there is also no result of the studies on use of additional data for enhancement of the natural in the synthesized speech and organization of these data.

SUMMARY OF THE INVENTION

Therefore, it is an object of the present invention to provide a text-to-speech conversion system ( TTS) for interlocking with multimediasynchronizing synthesized speech with a moving picture and a method for organizing input data of the sametherefore for enhancing the naturalnaturalness of synthesized speech and accomplishing synchronization ofsynchronizing multimedia with TTS by defining additional prosody information, the information required to interlocksynchronize TTS with multimedia, and an interface between thesesuch information and TTS for use in the production of theproducing synthesized speech.

In order to accomplish the above object, a TTS for interlocking with multimedia according to the present invention comprises a multimedia information input unit for organizing text, prosody, the information on synchronization with moving picture, lip-shape, and the information such as individual property; a data distributor by each media for distributing the information of the multimedia information input unit into the information by each media; a language processor for converting the text distributed by the data distributor by each media into phoneme stream, presuming prosody information and symbolizing the information; a prosody processor for calculating a value of prosody control parameter from the symbolized prosody information using a rule and a table; a synchronization adjuster for adjusting the duration of the phoneme using the synchronization information distributed by the data distributor by each media; a signal processor for producing a synthesized speech using the prosody control parameter and data in a synthesis unit database; and a picture output apparatus for outputting the picture information distributed by the data distributor by each media onto a screen.

In order to accomplish the above object, a method for organizing input data of a text-to-speech conversion system (TTS) for interlocking with multimedia comprises the steps of: classifying multimedia input information organized for enhancing the natural of synthesized speech and implementing the synchronization of multimedia with TIS into text, prosody, the information on synchronization with moving picture, lip-shape, and individual property information in a multimedia information input unit; distributing the information classified in the multimedia information input in a data distributor by each media, based on respective information; converting text distributed in the data distributor by each media into phoneme stream, presuming prosody information and symbolizing the information in a language processor; calculating a value of prosody control parameter other than prosody control parameter included in multimedia information in a prosody processor; adjusting the duration every each phoneme in a synchronization adjuster so that processing result in the prosody processor may be synchronized with a picture signal according to input of the synchronization information; producing the synchronized speech in a signal processor using the prosody information from the data distributor by each media, the processing result in the synchronization adjuster, and a synthesis unit database; and outputting the picture information distributed by the data distributor by each media onto a screen in a picture output apparatus.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing and other objects, features, aspects of the present invention will become more apparent from the following detailed description of the present invention when taken in conjunction with the accompanying drawings.

FIG. 1 is a constructional view of a conventional text-to-speech conversion system.

FIG. 2 is a constructional view of a hardware to which the present invention is applied.

FIG. 3 is a constructional view of a text-to-speech conversion system according to the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

Now, the present invention will be described in detail by way of the preferred embodiment.

Referring to FIG. 2, a constructional view of hardware to which the present invention is applied is shown. In FIG. 2, the hardware consists of a multimedia data input unit 5, a central processing unit 6, a synthesis database 7, a digital to analog (D/A) converter 8, and a picture output apparatus 9.

The multimedia data input unit 5 is inputted with data composed of multimedia such as picture and text and outputs this data to the central processing unit 6.

The central processing unit 6 distributes the multimedia data input of the present invention, adjusts synchronization, and performs algorithm based therein to produce synthesized speech.

The synthesis database 7 is a database used in the algorithm for producing the synthesized speech. This synthesis database 7 is stored in a storage device and transmits necessary data to the central processing unit 6.

The digital to analog (D/A) converter 8 converts the synthesized digital data into analog signal and outputs the analog signal.

The picture output apparatus 9 outputs inputted picture information onto a screen.

Table 1 and 2 are algorithms illustrating the state of organized multimedia input information, which consists of text, prosody, the information on synchronization with moving picture, lip-shape, and individual property information.

TABLE 1
Syntax
TTS_Sequence( ) {
TTS_Sequence_Start_Code
TTS_Sentence_ID
Language_Code
Prosody_Enable
Video_Enable
Lip_Shape_Enable
Trick_Mode_Enable
do{
     TTS_Sentence ( )
     }while (next_bits( )==TTS_Sentence_Start_Code
}

Here, the TTS_Sequence_Start_Code is a bit string represented with Hexadecimal ‘XXXXX’ and means a start of TTS sentence.

The TTS_Sentence_ID is a 10-bit ID andwhich represents a properunique identifying number offor each TTS data stream.

The language_Code represents an object language such as Korean language, English language, German language, Japanese language, French language etc,. to be synthesize.

The prosody_Enable is a 1-bit flag and has a value of ‘1’ when a prosody data of original sound is included in an organized data.

The Video_Enable is a 1-bit flag and has a value of ‘1’ when a TTS is interlocked with moving picture.

The Lip _Shape_Enable is a 1-bit flag and has a value of ‘1’ when a lip_shape data is included in an organized data.

The Trick_Mode_Enable is a 1-bit flag and has a value of ‘1’ when a data is organized to support a trick mode such as stop, restart, forward and backward.

TABLE 2
Syntax
TTS_Sentence ( ) {
TTS_Sentence_Start_Code
TTS_Sentence_ID
Silence
if (Silence) {
 Silence_Duration
 }
else {
 Gender
 Age
 if(!Video_Enable) {
  Speech_Rate
  }
 Length_of_Text
 TTS_Text( )
 if(Prosody_Enable) {
  Dur_Enable
  FO_Contour_Enable
  Energy_Contour_Enable
  Number_of_Phonemes
  for(j=0 ; j<Number_of_phonemes ; j++) {
   Symbol_each_phoneme
   if(Dur_Enable) {
   Dur_each_phoneme
   }
   if(FO_Contour_Enable {
   FO_contour_each_phoneme
   }
   if(Energy_Contour_Enable) {
   Energy_contour_each_phoneme
   }
  }
 }
 if(Video_Enable) {
  Sentence_Duration
  Position_in_Sentence
  offset
  }
 if(Lip_Shape_Enable) {
  Number_of_Lip_Event
  for(j=0 , j<Number_of_Lip_Event ; j++) {
   Lip_in_Sentence
   Lip_Shape
   }
  }
 }
}

Here, the TTS_Sentence_Start_Code is a bit string represented withby Hexadecimal ‘XXXXX’ and meansindicates a start of a TTS sentence. And the TTS_Sentence_Start_Code is a 10-bit ID and represents a proper number ofunique identifier for each TTS data stream.

The TTS_Sentence_ID is a 10-bit ID and represents a proper numbera unique identifier of each TTS sentence existedpresent in the TTS data stream.

The Silence become ais a one-bit flag which is set to ‘1’ when a present input frame of 1-bit flag is silencea silent speech section.

At stage of the Silence_Duration, arepresents the duration time of a present silencesilent speech section is represented byin milliseconds.

At stage of the Gender,indicates the desired gender is distinguished from aof the synthesized speech.

At stage of the Age, anindicates a desired apparent age of the synthesized speech distinguished into acategorized by baby, youth, middle age andor old age speech quality.

The Speech_Rate represents a desired output speech rate of the synthesized speech.

At stage of the Length_of_Text, arepresents the length of an input text sentence is represented byas a byte.

At stage of the TTS_Text,represents an optional length of a sentence text having optional length is represented.

The Dur_Enable is a 1-bit flag and become aset to ‘1’ when a duration time information is included in anthe organized data stream.

The FO_Contour_Enable is a 1-bit flag and become aset to ‘1’ when a pitch information of each phoneme is included in the organized data stream.

The Energy_Contour_Enable is a 1-bit flag and become aset to ‘1’ when an energy information of each phoneme is included in the organized data stream.

At stage of the Number_of_Phonemes,represents the number of phoneme neededphonemes required to synthesize a sentence are represented.

At stage of the Symbol_each_phoneme,represents a symbol, such as IPA, which is to represent each phoneme is represented.

The Dur_each_phoneme represents a duration time of each phoneme.

At stage of the FO_contour_each_phoneme,represents a pitch pattern of the phoneme represented by a pitch value of a beginning point, a mid point and an end point of the phoneme is represented.

At stage of the Energy_Contour_each_phoneme,represents an energy pattern of the phoneme is represented and an energy value of a beginning point, a mid point and an end point of the phoneme is represented by decibelin decibels (dB).

The Sentence_Duration represents a total duration time of the synthesized speech of the sentence.

The Position_in_Sentence represents a position of a present frame in the sentence.

At stage of the offset,Offset represents a delay time when the synthesized speech is interlockedsynchronized with a moving picture, and a beginning point of the sentence is in the GOP (Group Of Pictures), aThe delay time consumedidentifies the time from the beginning point of the GOP to the beginning point of the sentence is represented.

The Number_of_Lip_Event represents the number of changing point of lip-shape in the sentence.

LipinSentence represents the location of a lipshape changing part in a sentence.

The Lip_shape represents a lip—shape at lip—shape changing point of the sentence.

Text information includes a classification code for a used language and a sentence text. Prosody information includes the number of phoneme in the sentence, phoneme stream information, the duration of every each phoneme, pitch pattern of phoneme, energy pattern of phoneme, and is used for enhancing the natural of the synthesized speech. The synchronization information of the moving picture with the synthesized speech can be considered as the dubbing concept and the synchronization could be realized in three ways.

Firstly, there is a method to synchronize between the moving picture and the synthesized speech by the sentence unit by which method the duration of the synthesized speech is adjusted using the information about the beginning points of sentences, the durations of sentences, and the delay times of the beginning points of sentences. The beginning points of each sentence indicates the locations of scenes from which output of the synthesized speech for each sentence within the moving picture is started. The durations of sentences indicate the number of scenes in which the synthesized speech for each sentence lasts. In addition, the moving picture of MPEG-2 and MPEG-4 picture compression type in which Group of Picture (GOP) concept is used should start at not any scene but a beginning scene within Group of Picture for reproduction. Therefore, the delay time of the beginning point is the information required to synchronize between the Group of Picture and the TTS and indicates delay time between the beginning scene and a speech beginning point. This method is easy to be realized and can minimize additional effort but is difficult to accomplish natural synchronization by this method.

Secondly, there is a method by whichThe second method produces synthesized speech on a phoneme basis by use of beginning point information,and end point information, and phoneme information are marked everyfor each phoneme within an intervalperiod associated with a speech signal in the moving picture and these information is used to produce the synthesized speech. This method has an advantage in that the degree of accuracy is high since the synchronization between the moving picture and the synthesized speech by the phoneme unit can be attained butrealized. However, this method also has a disadvantage in that additional effort should be fairlymust be made to detect and record the duration information byof the phoneme unit within the speech interval of the moving picture.

Thirdly, there is a method to recordThe third method records the synchronization information based on the beginning point of speech, the end point of speech, lip-shape, information, and a change point of timeinformation of lip-shape change. Lip-shape is numeralizedinformation quantifies the to distance (extent of opening) between the upper lip and the lower lip, the distance (extent of width) between left and right and points of the lip, and the extent of projecting of the lip and is defined as a quantized and normalized pattern depended ondependent upon articulation location and articulation manner of the phoneme using a on the basis of pattern with high discriminative propertyhighly discriminating pattern. This method is a method to raiseimproves the efficiency of synchronization, while minimizing additional effort to produce the synchronization information for synchronization can be minimized.

The organizedOrganizing multimedia input information which is applied toin accordance with the present invention allows an information provider to select and implement optionally among 3any of the three synchronization methods as described above.

In addition, the organized multimedia input information is also used in the process to implement lip animation. Lip animation can be implemented by using phoneme stream prepared from the inputted text in the TTS and the duration every each phoneme, or phoneme stream distributed from the input information and the duration every each phoneme, or by using the information on lip-shape included in the inputted information.

The individual property information allows the user to change gender, age, and speech rate of the synthesized speech. Gender has male and female, and age is classified into 4, for example, 6-7 years, 18 years, 40 years, and 65 years. The change of speech rate may have 10 steps between 0.7 and 1.6 times of a standard rate. Quality of the synthesized speech can be diversified using these information.

FIG. 3 is a constructional view of the text-to-speech conversion system (TTS) according to the present invention. In FIG. 3, the TTS consists of a multimedia information input unit 10, a data distributor by each media 11, a standardized language processor 12, a prosody processor 13, a synchronization adjuster 14, a signal processor 15, a synthesis unit database 16, and a picture output apparatus 17.

The multimedia input unit 10 is configured as form of Table 1 and 2 and comprises text, prosody information, the information on synchronization with moving picture, the information on lip-shape. Among these, requisite information is text, other information can be optionally provided by an information provider as optional item for enhancing the individual property and the natural and accomplishing the synchronization with the multimedia, and if needed, can be amended by a TTS user by means of a character input device (keyboard) or a mouse. These information is transmitted to the data distributor by each media 11.

The data distributor by each media 11 receives the multimedia information of which the picture information is transmitted to the picture output apparatus 17, text is transmitted to the language processor 12, and the synchronization information is converted into data structure capable of utilizing in the synchronization adjuster 14 and transmitted to the synchronization adjustor 14. If prosody information is included in the inputted multimedia information, this multimedia information is converted into a data structure capable of utilizing in the signal processor 15 and then transmitted to the prosody processor 13 and the synchronization adjustor 14. If individual property information is included in the inputted multimedia information, this multimedia information is converted into data structure capable of utilizing in the synthesis unit database 16 and the prosody processor 13 within the TTS and then transmitted to the synthesis unit database 16 and the prosody processor 13.

The language processor 12 converts text into phoneme stream, presumes prosody information, symbolizes this information, and then transmits the symbolized information to the prosody processor 13. The symbol of prosody information is presumed from a boundary of the phrase and paragraph, a location of accent in word, a sentence pattern, and so on using the analysis result syntax.

The prosody processor 13 takes the processing result of the language processor 12 and calculates value of prosody control parameter other than prosody control parameter included in the multimedia information. Prosody control parameter includes duration pitch contour, energy contour, pause point, and pause length of phoneme. The calculated result is transmitted to thesynchronization adjustor 14.

The synchronization adjustor 14 takes the processing result of the prosody processor 13 and adjusts the duration every each phoneme in orderto synchronize the result with the picture signal. The adjustment of the duration every each phoneme utilizes the synchronization information transmitted from the data distributor by each media 11. First, lip-shape is assigned to each phoneme depended on articulation location and articulation manner of each phoneme and, on the basis of this, the assigned lip-shape information is compared to lip-shape information included in the synchronization information and then phoneme stream is divided into small groups by the number of lip-shape recorded in the synchronization information. Also, the duration of phoneme in the small groups is calculated again using the duration information of lip-shape included in the synchronization information. The adjusted duration information is transmitted to the signal processor, included in the processing result of the prosody processor.

The signal processor 15 receivesthe prosody information from the multimedia distributor 11 or the processing result of the synchronization adjustor 14 to produce and output the synthesized speech using the synthesis unit database 16.

The synthesis unit database 16 receives the individual property information from the multimedia distributor 11, selects synthesis units adaptable to gender and age, and then transmits data required for synthesis to the signal processor 15 in response to a request from the signal processor 15.

As can be seen from the description described above, according to the present invention, the individual property of the synthesized speech can be realized and the natural of the synthesized speech can be enhanced by organizing the individual property and prosody information presumed by the analysis of actual speech data, along with text information, as multistage information. Furthermore, a foreign movie can be dubbed in Korean by implementing the synchronization of the synthesized speech with the moving picture by way of the direct use of text information and lip-shape information which is presumed by the analysis of actual speech data and lip-shape in the moving picture for the production of the synthesized speech. Still furthermore, the present invention is applicable to a variety of field such as communication service, office automation, education and so on by making the synchronization between the picture information and the TTS in the multimedia environment possible.

Although the present invention and its advantages have been described in detail, it should be understood that various changes, substitutions and alterations can be made herein without departing from the spirit and scope of the invention as defined by the appended claims.

It is therefore intended by the appended claims to cover any and all such applications, modifications, and embodiments within the scope of the present invention.

Patent Citations
Cited PatentFiling datePublication dateApplicantTitle
US4260229 *Jan 23, 1978Apr 7, 1981Bloomstein Richard WCreating visual images of lip movements
US4305131 *Mar 31, 1980Dec 8, 1981Best Robert MDialog between TV movies and human viewers
US5111409 *Jul 21, 1989May 5, 1992Elon GasperAuthoring and use systems for sound synchronized animation
US5313522 *Apr 15, 1993May 17, 1994Slager Robert PApparatus for generating from an audio signal a moving visual lip image from which a speech content of the signal can be comprehended by a lipreader
US5386581 *Sep 3, 1993Jan 31, 1995Matsushita Electric Industrial Co., Ltd.Multimedia data editing apparatus including visual graphic display of time information
US5500919 *Nov 18, 1992Mar 19, 1996Canon Information Systems, Inc.Graphics user interface for controlling text-to-speech conversion
US5557661 *Oct 26, 1994Sep 17, 1996Nec CorporationSystem for coding and decoding moving pictures based on the result of speech analysis
US5608839Jun 21, 1994Mar 4, 1997Lucent Technologies Inc.Sound-synchronized video system
US5615300 *May 26, 1993Mar 25, 1997Toshiba CorporationText-to-speech synthesis with controllable processing time and speech quality
US5630017May 31, 1995May 13, 1997Bright Star Technology, Inc.Advanced tools for speech synchronized animation
US5636325 *Jan 5, 1994Jun 3, 1997International Business Machines CorporationSpeech synthesis and analysis of dialects
US5657426 *Jun 10, 1994Aug 12, 1997Digital Equipment CorporationMethod and apparatus for producing audio-visual synthetic speech
US5677739Mar 2, 1995Oct 14, 1997National Captioning InstituteSystem and method for providing described television services
US5677993 *Aug 31, 1993Oct 14, 1997Hitachi, Ltd.Information processing apparatus using pointing input and speech input
US5689618May 31, 1995Nov 18, 1997Bright Star Technology, Inc.For a programmed computer
US5729694 *Feb 6, 1996Mar 17, 1998The Regents Of The University Of CaliforniaSpeech coding, reconstruction and recognition using acoustics and electromagnetic waves
US5751906 *Jan 29, 1997May 12, 1998Nynex Science & TechnologyMethod for synthesizing speech from text and for spelling all or portions of the text by analogy
US5774854 *Nov 22, 1994Jun 30, 1998International Business Machines CorporationFor converting input text into an output acoustic signal
US5777612 *Nov 22, 1995Jul 7, 1998Fujitsu LimitedMultimedia dynamic synchronization system
US5850629 *Sep 9, 1996Dec 15, 1998Matsushita Electric Industrial Co., Ltd.User interface controller for text-to-speech synthesizer
US5860064 *Feb 24, 1997Jan 12, 1999Apple Computer, Inc.Method and apparatus for automatic generation of vocal emotion in a synthetic text-to-speech system
US5970459 *Nov 14, 1997Oct 19, 1999Electronics And Telecommunications Research InstituteSystem for synchronization between moving picture and a text-to-speech converter
AT72083B Title not available
DE4101022A1Jan 16, 1991Jul 23, 1992Medav Digitale SignalverarbeitVariable speed reproduction of audio signal without spectral change - dividing digitised audio signal into blocks, performing transformation, and adding or omitting blocks before reverse transformation
EP0225729B1Nov 10, 1986Jan 22, 1992BRITISH TELECOMMUNICATIONS public limited companyImage encoding and synthesis
EP0689362A2Jun 14, 1995Dec 27, 1995AT&amp;T Corp.Sound-synchronised video system
EP0706170A2May 24, 1995Apr 10, 1996CSELT Centro Studi e Laboratori Telecomunicazioni S.p.A.Method of speech synthesis by means of concatenation and partial overlapping of waveforms
GB2231246A Title not available
JPH0564171A Title not available
JPH0738857A Title not available
JPH0830287A Title not available
JPH02234285A Title not available
JPH03241399A Title not available
JPH04285769A Title not available
JPH04359299A Title not available
JPH05188985A Title not available
JPH05313686A Title not available
JPH06326967A Title not available
JPH06348811A Title not available
JPH07306692A Title not available
WO1985004747A1Dec 4, 1984Oct 24, 1985First ByteReal-time text-to-speech conversion system
Non-Patent Citations
Reference
1Nakumura et al. "Speech Recognition and Lip Movement Synthesis"; HMM based Audio-Visual Integration; pp. 93-98, 1997.
2Nakumura et al. "Speech Recognition and Lip Movement Synthesis"; HMM based Audio-Visual Integration; pp. 93-98.
3Yamamoto et al. pp. 245-246 Nara Institute of Science and Technology Sep. 1997.
4Yamamoto et al. pp. 245-246 Nara Institute of Science and Technology.
Classifications
U.S. Classification704/260, 704/278, 704/275, 715/727, 704/276
International ClassificationG10L13/08, G10L13/06, G10L21/06, G10L13/00, G06F3/16, G10L13/04
Cooperative ClassificationG10L13/043, G10L2021/105
European ClassificationG10L13/04U