|Publication number||US8204746 B2|
|Application number||US 12/058,501|
|Publication date||Jun 19, 2012|
|Filing date||Mar 28, 2008|
|Priority date||Mar 29, 2007|
|Also published as||EP1976255A2, EP1976255A3, EP1976255B1, US8521528, US9224389, US9484035, US20080243515, US20120253806, US20130346080, US20160111092|
|Publication number||058501, 12058501, US 8204746 B2, US 8204746B2, US-B2-8204746, US8204746 B2, US8204746B2|
|Original Assignee||Intellisist, Inc.|
|Export Citation||BiBTeX, EndNote, RefMan|
|Patent Citations (14), Non-Patent Citations (2), Referenced by (9), Classifications (17), Legal Events (7)|
|External Links: USPTO, USPTO Assignment, Espacenet|
This non-provisional patent application claims priority under 35 U.S.C. §119(e) to U.S. Provisional Patent Application Ser. No. 60/920,849, filed Mar. 29, 2007, the disclosure of which is incorporated by reference.
The invention relates in general to speech recognition and, specifically, to a system and method for providing an automated call center inline architecture.
Customer call centers, or simply, “call centers,” are often the first point of contact for customers seeking direct assistance from manufacturers and service vendors. Call centers are reachable by telephone, including data network-based telephone services, such as Voice-Over-Internet (VoIP), and provide customer support and problem resolution. Although World Wide Web- and email-based customer support are becoming increasingly available, call centers still offer a convenient and universally-accessible forum for remote customer assistance.
The timeliness and quality of service provided, by call centers is critical to ensuring customer satisfaction, particularly where caller responses are generated through automation. Generally, the expectation level of callers is lower when they are aware that an automated system, rather than a live human agent, is providing assistance. However, customers become less tolerant of delays, particularly when the delays occur before every automated system-generated response. Minimizing delays is crucial, even when, caller volume is high.
Automated call processing requires on-the-fly speech recognition. Parts of speech are matched against a stored grammar that represents the automated system's “vocabulary.” Spoken words and phrases are identified from which the caller's needs are determined, which can require obtaining further information from the caller, routing the call, or playing information to the caller in audio form.
Accurate speech recognition hinges on a rich grammar embodying a large vocabulary. However, a rich grammar, particularly when provided in multiple languages, creates a large search space and machine latency can increase exponentially as the size of a grammar grows. Consequently, the time required to generate an automated response will also increase. Conventional approaches to minimizing automated system response delays compromise quality over speed.
U.S. Patent Publication No. 2005/0002502 to Cloren, published Jan. 6, 2005, discloses an apparatus and method for processing service interactions. An interactive voice and data response system uses a combination of human agents, advanced speech recognition, and expert systems to intelligently respond to customer inputs. Customer utterances or text are interpreted through speech recognition and human intelligence. Human agents are involved only intermittently during the course of a customer call to free individual agents from being tied up for the entire call duration. Multiple agents could be used in tandem to check customer intent and input data and the number of agents assigned to each component of customer interaction can be dynamically adjusted to balance workload. However, to accommodate significant end-user traffic, the Cloren system trades off speech recognition accuracy against agent availability and system performance progressively decays under increased caller volume.
Therefore, there is a need for providing speech recognition for an automated call center that minimizes caller response delays and ensures consistent quality and accuracy independent of caller volume. Preferably, such an approach would use tiered control structures to provide distributed voice recognition and decreased latency times while minimizing the roles of interactive human agents.
A system and method includes a centralized message server, a main speech recognizer, and one or more secondary speech recognizers. Additional levels of speech recognition servers are possible. The message server initiates a session with the main speech recognizer, which initiates a session with each of the secondary speech recognizers for each call received through a telephony interface. The main speech recognizer stores and forwards streamed audio data to each of the secondary speech recognizers and a secondary grammar reference that identifies a non-overlapping grammar section that is assigned to each respective secondary speech recognizer by the message server. Each secondary speech recognizer performs speech recognition on the streamed audio data against the assigned secondary grammar to generate secondary search results, which are sent to the main speech recognizer for incorporation into a new grammar that is generated using a main grammar template provided by the message server. The main speech recognizer performs speech recognition on the stored streamed audio data to generate a set of search results, which are sent to the message server. The main speech recognizer employs a form of an n-best algorithm, which chooses the n most-likely search results from each of the secondary search results to build the new grammar.
One embodiment provides a system and method for providing an automated call center inline architecture. A plurality of grammar references and prompts are maintained on a script engine. A call is received through a telephony interface. Audio data is collected using the prompts from the script engine, which are transmitted to the telephony interface via a message server. Distributed speech recognition is performed on a speech server. The grammar references are received from the script engine via the message server. Speech results are determined by applying the grammar references to the audio data. A new grammar is formed from the speech results. Speech recognition results are identified by applying the new grammar to the audio data. The speech recognition results are received as a display on an agent console.
Still other embodiments will become readily apparent to those skilled in the art from the following detailed description, wherein are described embodiments of the invention by way of illustrating the best mode contemplated for carrying out the invention. As will be realized, the invention is capable of other and different embodiments and its several details are capable of modifications in various obvious respects, all without departing from the spirit and the scope of the present invention. Accordingly, the drawings and detailed description are to be regarded as illustrative in nature and not as restrictive.
Prior Art Call Processing System
Due to the inflexibility of conventional call systems, callers often experience long hold times, which can lead to frustration and possible loss of revenue.
The speech server 13 performs speech recognition by analyzing the speech utterances 17 using the grammar reference 16. The grammar reference 16 is applied to the speech utterances 17 to identify parts of speech that match the grammar, which can be recorded as speech recognition results 18 for use in determining the caller's needs. The speech recognition results 18 are transmitted to the application server 11 via the IVR platform 12. The application server 11 determines whether additional information is needed from the caller. If required, a further prompt 15 can be selected and sent to the IVR platform 12 for collecting additional audio responses 17 from the caller. A confidence level can be assigned to the speech recognition results, which provides a score indicating the accuracy of the results.
The application server and the speech server can also communicate directly, rather than through the IVR platform.
If the prompt 25 includes audio, the prompt is automatically played to the caller through the IVR platform 22. When the prompt 25 includes text, the text is first converted using a text-to-speech engine (not shown) for playback to the caller. The caller can provide audio responses 26, which include speech utterances, in response to the prompts 25. The speech utterances 26 are collected from the caller and transmitted to a speech server 23, which coordinates the grammar reference 27 and the speech utterances 26. The speech server 23 performs speech recognition by applying the grammar reference 27 to the speech utterances 26 to identify parts of speech that can be recorded as speech recognition results 28. The speech recognition results 28 are directly sent to the application server 21 for further processing. If additional information is needed from the caller, the process of obtaining and processing the audio responses is repeated.
System for Performing Distributed Speech Recognition
Prior art call systems, such as described above with reference to
To decrease the delays created by the use of a rich grammar while still maintaining accurate results, distributed call center processing can be performed by delegating individualized speech recognition tasks over a plurality of hierarchically-structured speech recognizers.
The calls are received through the telephony interface 32, which sends control messages 36 to the script engine 31 via the message server 34 to inform the script engine 31 of the received call. In return, the script engine 31 sends prompts 37 and a plurality of grammar references 38 to the message server 34. The message server 34 transmits the prompts 37 to the telephony interface 32 and the grammar references 38 to the speech server 33. The prompts 37 can include call information in the form of a file or script. The file can include text, sound, or image information. When the prompts 37 include speech, the speech is automatically played back to the caller via the telephony interface 32, but when the prompt includes text, the text must first be converted into speech prior to playback to the caller. Other types of prompts are possible.
The caller can provide audio responses 39, including speech utterances, which are sent to the speech server 33 via the message server 34. Using the grammar references, the speech server performs distributed speech recognition on the audio responses 39, which is further discussed below with reference to
If the distributed speech recognition results 40 are sent to the agent console 35, the results 40 are displayed for review by an agent. The agent can modify the distributed speech recognition results 40 or select a further prompt 37, including a file or script for playback to the caller in response to the audio data 39. The further prompt 37 can be selected based on the results of the speech recognition. When further communication with the caller is necessary, the process of collecting and processing the audio responses 39 can be repeated.
In one embodiment, the message server can assign a confidence level to the distributed speech recognition results 40, which provides an indication of accuracy, if the confidence level is below a predetermined threshold, a placeholder can be assigned to the results 40, prior to sending the results to the agent console 35 or script engine 31. The placeholder provides notification that further results 40 are being generated. The script engine can automatically query the message server 34 for the further results 40. The agent can also query the message server 34 for the further results 40, as well as modify the further results 40.
To reduce caller hold time during speech recognition processing, the speech server can include multiple tiers of hierarchically-organized recognizers to conduct distributed speech recognition.
The system 50 is architected into two or more tiers of speech recognizers. In one embodiment, a main recognizer 54 and one or more secondary recognizers 55 are organized into two tiers. The main recognizer 54 and secondary recognizers 55 are interconnected to the message server 51 over a network infrastructure 57, such as the Internet or a non-public enterprise data network. An agent console 56 is further connected to the message server 51 over the network structure 57. The network infrastructure 57 can be either wired or wireless and, in one embodiment, is implemented based on the Transmission Control Protocol/Internet Protocol (TCP/IP) network communications specification, although other types or combinations of networking implementations are possible. Similarly, other network topologies and arrangements are possible.
The main recognizer 54 interfaces directly to the message server 51 and to each of the secondary recognizers 55 as a top-level or root tier of a speech recognition hierarchy. Each of the secondary recognizers 55 are interfaced directly to the main recognizer 54 as a second level or tier of the speech recognition hierarchy. Further levels or tiers of tertiary recognizers, quaternary recognizers, and so forth, are possible.
The message server 51 sends streamed audio data for each call to the main recognizer 54 and secondary recognizers 55, which then perform distributed speech recognition, as further described below with reference to
Operationally, upon startup, the telephony gateway 52 opens a T-1 carrier device channel for each available T-1 time slot. The telephony gateway 52 initiates a new connection to the message server 51, one connection per T-1 device channel, and the message server 51, in turn, initiates a corresponding new connection to the main recognizer 54. Finally, for each open T-1 device channel, the main recognizer 54 initiates a new connection to each of the secondary recognizers 55. The number of secondary recognizers 55 is independent from the number T-1 device channels.
The separate telephony gateway-to-message server, message server-to-main recognizer, and main recognizer-to-secondary recognizer connections form one concurrent session apiece. When a customer call is answered or connected, the telephony gateway 52 sends a call message to the message server 51. The message server 51 then sends a new call message to the main recognizer 54, which starts a new speech recognition session. The main recognizer 54 sends a new call message to each of the secondary recognizers 55, which also start new speech recognition sessions. Thus, given n secondary recognizers 55, n+1 concurrent speech recognition sessions are used for each call.
Each component, including the message server 51, main recognizer 54, and secondary recognizers 55, is implemented as a computer program, procedure or module written as source code in a conventional, programming language, such as the C++ programming language, and presented for execution by a computer system as object or byte code. Alternatively, the components could be directly implemented in hardware, either as integrated circuitry or burned into read-only memory components. The various implementations of the source code and object and byte codes can be held on a computer-readable storage medium or embodied on a transmission medium In a carrier wave. The system 50 operates in accordance with a sequence of process steps, as further described below with reference to
Grammar and Search Result Distribution
Speech recognition is performed through message exchange and streamed audio data communicated via the network infrastructure 57.
For each speech utterance, the message server 51 sends a main grammar template 64 and a set of secondary grammar references 65 to the main recognizer 54. The main recognizer 54 stores the main grammar template 67, which specifies the structure for a new grammar 70 that will eventually be generated based on secondary search results provided by the secondary recognizers 55. The main recognizer 54 forwards the secondary grammar references 65 to each of the secondary recognizers 55, which use their respective secondary grammar reference 65 to identify a secondary grammar 68 a-c for use in secondary speech recognition. In one embodiment, each secondary grammar 68 a-c is a non-overlapping section of a main grammar, and the message server 51 assigns each section to the secondary recognizers 55 to balance work load and minimize grammar search latency times.
Speech recognition is performed on streamed audio data 66, which is received from the telephony interface 52 by way of the message server 51. The streamed audio data 66 is forwarded to and stored by the main recognizer 54 and by each of the secondary recognizers 55. The secondary recognizers 55 each perform speech recognition on the streamed audio data 66 against their respective secondary grammars 68 a-c to generate a set of raw secondary search results. Each secondary speech recognizer 55 then applies a form of the n-best algorithm by selecting the n most likely search results from each set of raw secondary search results, which are then sent to the main recognizer 54 as secondary search results 69 a-c. The main recognizer 54 uses the secondary search results 69 a-c to form the new grammar 70. Other forms of applicative search result selection algorithms are possible. Speech recognition can be performed by each secondary recognizer 55 using a speech recognition engine, such as the OpenSpeech Recognizer speech engine, licensed by Nuance Communications, Inc., Burlington, Mass. Other speech, recognition engines and approaches are possible.
The main recognizer 54 constructs a new grammar 70 based on the stored main grammar template 67 using the secondary search results 69 a-c as a new “vocabulary.” As the secondary search results 69 a-c generated by each secondary recognizer 55 differ based on the non-overlapping secondary grammars 68 a-c used, the main grammar 54 compensates for probabilistic ties or close search results by using the secondary search results 69 a-c, which each include the n most likely secondary search results identified by each secondary recognizer 55, to form the new grammar 70. The main recognizer 54 then performs speech recognition on the stored streamed audio data 66 against the new grammar 70 to generate a set of speech recognition results 71, which are sent to the message server 51. Speech recognition can be performed by the main recognizer 54 using a speech recognition engine, such as the OpenSpeech Recognizer speech engine, described above. Other speech recognition engines and approaches are possible.
Method for Performing Distributed Speech Recognition
Control over distributed speech recognition is mainly provided through the message server 51, which sends the main grammar template 64 and secondary grammar references 65 to initiate speech recognition for each speech utterance. The main recognizer 54 and secondary recognizers 55 then operate in concert to perform the distributed speech recognition.
In a further embodiment, additional levels or tiers of tertiary recognizers, quaternary recognizers, and so forth, can be implemented by expanding on the operations performed by the main recognizer 54 and secondary recognizers 55. For example, secondary grammar templates can be sent to the secondary recognizers 55 instead of secondary grammar references, and tertiary grammar references can be sent to tertiary recognizers, which perform tertiary speech recognition and send tertiary search results to the secondary recognizers 55. The secondary recognizers 55 would then construct new secondary grammars using the tertiary search results based on the secondary grammar templates, against, which speech recognition would be performed. Other arrangements and assignments of new grammars and non-overlapping grammars are possible.
Main and Secondary Recognizers
In one embodiment, the message server 51, main recognizer 54, and each of the secondary recognizers 55 are implemented on separate computing platforms to minimize competition for resources, such as memory access, hard disk data retrieval and communications, between the recognizers, which may lead to latency delays.
Referring next to
While the invention has been particularly shown and described as referenced to the embodiments thereof those skilled in the art will understand that the foregoing and other changes in form and detail may be made therein without departing from the spirit and scope of the invention.
|Cited Patent||Filing date||Publication date||Applicant||Title|
|US5799276 *||Nov 7, 1995||Aug 25, 1998||Accent Incorporated||Knowledge-based speech recognition system and methods having frame length computed based upon estimated pitch period of vocalic intervals|
|US5956683||Apr 4, 1996||Sep 21, 1999||Qualcomm Incorporated||Distributed voice recognition system|
|US5960399||Dec 24, 1997||Sep 28, 1999||Gte Internetworking Incorporated||Client/server speech processor/recognizer|
|US6006183 *||Dec 16, 1997||Dec 21, 1999||International Business Machines Corp.||Speech recognition confidence level display|
|US6633846||Nov 12, 1999||Oct 14, 2003||Phoenix Solutions, Inc.||Distributed realtime speech recognition system|
|US6704707||Mar 14, 2001||Mar 9, 2004||Intel Corporation||Method for automatically and dynamically switching between speech technologies|
|US6789065 *||Jan 24, 2001||Sep 7, 2004||Bevocal, Inc||System, method and computer program product for point-to-point voice-enabled driving directions|
|US7016849 *||Mar 25, 2002||Mar 21, 2006||Sri International||Method and apparatus for providing speech-driven routing between spoken language applications|
|US7092888||Oct 24, 2002||Aug 15, 2006||Verizon Corporate Services Group Inc.||Unsupervised training in natural language call routing|
|US20030125948 *||Jan 2, 2003||Jul 3, 2003||Yevgeniy Lyudovyk||System and method for speech recognition by multi-pass recognition using context specific grammars|
|US20040236580 *||Mar 2, 2004||Nov 25, 2004||Bennett Ian M.||Method for processing speech using dynamic grammars|
|US20050002502||May 5, 2004||Jan 6, 2005||Interactions, Llc||Apparatus and method for processing service interactions|
|US20060009980 *||Jul 12, 2004||Jan 12, 2006||Burke Paul M||Allocation of speech recognition tasks and combination of results thereof|
|US20070043566 *||Aug 19, 2005||Feb 22, 2007||Cisco Technology, Inc.||System and method for maintaining a speech-recognition grammar|
|1||Zhang et al., "The Study on Distributed Speech Recognition System," Acoustics, Speech, and Signal Processing, 2000. ICASSP '00. Proceedings. 2000 IEEE International Conference on Jun. 5-9, 2000, Piscataway, NJ, USA, IEEE, vol. 3, pp. 1431-1434 (Jun. 5, 2000).|
|2||Zhou et al., "Lucent Automatic Speech Recognition: A Speech Recognition Engine for Internet and Telephony Service Applications," ICSLP 2000, pp. 1-4 (Oct. 16, 2000).|
|Citing Patent||Filing date||Publication date||Applicant||Title|
|US9058805||May 13, 2013||Jun 16, 2015||Google Inc.||Multiple recognizer speech recognition|
|US9224389 *||Aug 26, 2013||Dec 29, 2015||Intellisist, Inc.||System and method for performing distributed speech recognition|
|US9293136||Jun 1, 2015||Mar 22, 2016||Google Inc.||Multiple recognizer speech recognition|
|US9460718 *||Mar 12, 2014||Oct 4, 2016||Kabushiki Kaisha Toshiba||Text generator, text generating method, and computer program product|
|US9484035 *||Dec 28, 2015||Nov 1, 2016||Intellisist, Inc||System and method for distributed speech recognition|
|US9542947||Jul 31, 2013||Jan 10, 2017||Google Technology Holdings LLC||Method and apparatus including parallell processes for voice recognition|
|US20130346080 *||Aug 26, 2013||Dec 26, 2013||Intellisist, Inc.||System And Method For Performing Distributed Speech Recognition|
|US20140122071 *||Feb 22, 2013||May 1, 2014||Motorola Mobility Llc||Method and System for Voice Recognition Employing Multiple Voice-Recognition Techniques|
|US20140303974 *||Mar 12, 2014||Oct 9, 2014||Kabushiki Kaisha Toshiba||Text generator, text generating method, and computer program product|
|U.S. Classification||704/244, 704/251, 704/235|
|International Classification||G10L15/26, G10L15/04, G10L15/06|
|Cooperative Classification||H04M3/5166, G10L15/30, G10L15/193, G10L15/18, H04M2201/40, H04M2201/38, H04M3/51, G10L15/34, G10L15/32|
|European Classification||H04M3/51, G10L15/30|
|Mar 31, 2008||AS||Assignment|
Owner name: INTELLISIST, INC., WASHINGTON
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:ODINAK, GILAD;REEL/FRAME:020727/0501
Effective date: 20080327
|Dec 9, 2009||AS||Assignment|
Owner name: SQUARE 1 BANK, NORTH CAROLINA
Free format text: SECURITY AGREEMENT;ASSIGNOR:INTELLISIST, INC. DBA SPOKEN COMMUNICATIONS;REEL/FRAME:023627/0412
Effective date: 20091207
|Dec 14, 2010||AS||Assignment|
Owner name: INTELLISIST, INC., WASHINGTON
Free format text: RELEASE OF SECURITY INTEREST;ASSIGNOR:SQUARE 1 BANK;REEL/FRAME:025585/0810
Effective date: 20101214
|Mar 28, 2014||AS||Assignment|
Owner name: SILICON VALLEY BANK, CALIFORNIA
Free format text: SECURITY INTEREST;ASSIGNOR:INTELLISIST, INC.;REEL/FRAME:032555/0516
Effective date: 20120814
|Oct 23, 2015||AS||Assignment|
Owner name: PACIFIC WESTERN BANK (AS SUCCESSOR IN INTEREST BY
Free format text: SECURITY INTEREST;ASSIGNOR:INTELLISIST, INC.;REEL/FRAME:036942/0087
Effective date: 20150330
|Dec 18, 2015||FPAY||Fee payment|
Year of fee payment: 4
|Jul 6, 2016||AS||Assignment|
Owner name: INTELLISIST, INC., WASHINGTON
Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:SILICON VALLEY BANK;REEL/FRAME:039266/0902
Effective date: 20160430