Abstract: To improve accuracy of an evaluation in an acoustic quality evaluation test performed by comparing an evaluation target sound and a reference sound. A data correction apparatus 3 compares, in a call performed between a near-end terminal 1 and a far-end terminal 2, an evaluation target sound in which a voice output from the near-end terminal 1 is recorded and a reference sound in which a voice spoken by a call partner using the far-end terminal 2 to correct test data used in a listening test for evaluating acoustic quality of the call. A correction target determination unit 31 determines, as a correction target section, a voiced section that does not include the voice of the call partner detected from an acoustic signal representing the reference sound. A correction execution unit 32 updates the correction target section of the acoustic signal representing the reference sound with a non-voice signal predetermined.

Abstract: Disclosed herein are systems and methods of automated teller machine (ATM) management. The method can include receiving, from an ATM, an operating log from an out-of-band management chip. The method can further include determining, from the operating log, that the ATM is in an out-of-service state. The method can further include transmitting instructions to the out-of-band management chip to capture a system image of the ATM. In response, the method can include receiving the system image of the ATM from the out-of-band management chip and diagnosing a fault in the ATM based on the system image. The fault in the ATM can be the fault which caused the ATM to be in the out-of-service state. Finally, the method can include transmitting instructions for one or more corrective actions to the out-of-band management chip. The corrective actions can remedy the fault in the ATM.

Abstract: A system for verifying data integrity in a database may include accessing a first file with a management object. The management object corresponds to a set of database tables located in a database. The system loads a data extract file including the management object, the data extract file containing fields related to content associated with the management object. The system accesses a second file with a condition, the condition being indicative of the content associated with the management object. The system generates code to evaluate the management object in the data extract file. The system evaluates the content associated with the management object in the data extract file using the generated code. The system identifies an entry in the data extract file that does not meet the condition, the entry representative of content associated with the management object, and then presents the entry to a user.

Abstract: Described herein is a method and system that provides symptom based and context sensitive knowledgebase management during a remote management session with a connected device. Knowledgebase profiles are generated that are linked to symptoms and device metrics and knowledgebase search rules are set that are linked to the symptoms, and specific device metrics. The knowledgebase management system provides a remote support technician with the support information instantly during a remote management session based on the symptom described by an end user and an assessment of the connected device state in order to aid in the troubleshooting of the issue and assist in the transfer of the knowledge.

Abstract: A sales data processing apparatus includes the following. An operation unit receives operation by touch by an operator. A setting unit sets contents of operation corresponding to how the operation unit is touched associated with each operator. A controller performs, for each operator, contents of operation set by the setting unit associated with how the operation unit is touched in response to the operator touching the operation unit.

Abstract: The described method and system provide a connectivity problem resolution mechanism that uses a data mode call in order to diagnose a connection problem while avoiding audible ringing and or inadvertent user pick-up during error diagnosis and correction. The system includes three primary functions and elements including cellular error message recognition, error response handling configured to update and/or maintain processes, and new error message capture. This system is usable both online and offline. Error recognition may be based on error message matching or keyword matching.

Abstract: A method for retrofitting a vending machine with a video communications device includes securing a video communications device with a mounting feature of the video communications device removably coupled with a mounting location on a vending machine. The video communications device includes a video communications module for real-time interactions with a ticket office. The video communications module includes a display, a camera, a speaker, and a microphone. The video communications device includes a video communication device communications interface. The method includes electronically coupling the video communications device with a power supply of the vending machine and communicatively coupling the video communications device with the vending machine such that functions of the vending machine are controllable using the video communications device. The method includes communicatively coupling the video communications device with the ticket office for real-time video communications with the ticket office.

Abstract: In accordance with embodiments disclosed herein, there are provided apparatus, systems and methods for detecting a boundary condition via common mode diagnostics; and to apparatus, systems and methods for detecting a fault condition via common-mode rejection ratio diagnostics. For example, in one embodiment such means include, means for injecting differential mode signal probes onto two conductors of a Digital Subscriber Line (DSL line); means for measuring a common-mode rejection ratio (CMRR) of the differential mode signal probes; means for measuring impedance of the differential mode signal probes; and means for identifying which of an asymmetric fault condition or a symmetric fault condition is present on the line based on both: (a) the measured CMRR of the differential mode signal probes and (b) the measured impedance of the differential mode signal probes.

Abstract: A device includes a Session Initiation Protocol (SIP) phone, an audio interface for receiving or transmitting audio information with the SIP phone to test a SIP-based network, and a controller for controlling the SIP phone and the audio interface.

Abstract: In one of many possible embodiments, an exemplary system includes a test management subsystem configured to provide a user portal to a user of a circuit provided by a service provider, the user portal including a tool enabling the user to select a signal loop for testing at least a section of the circuit, the signal loop being selected from a plurality of signal loop options. The system also includes a network management subsystem communicatively coupled to the test management subsystem, the network management subsystem being configured to receive data representative of the selection from the test management subsystem and instruct, based on the selection, a network device along the circuit to execute a loop-back mode. In certain embodiments, the selected signal loop defines a test pattern signal flow for testing a subsection of the circuit.

Abstract: A method for monitoring voice quality in a communication network. In a first embodiment, the method comprises the steps of: establishing a voice communication pathway (VCPW) between first and second telephone terminals; establishing speech cadence between first and second terminals, wherein the terminals alternate between opposing send and listen/receive modes such that when one of the terminals is sending a voice test message the other terminal is either listening for or receiving the voice test message and vice versa; and calculating a voice quality score for one or more received voice test messages. In a second embodiment, synchronous cadence is established between a plurality of telephone terminals and a voice recognition server (VRS).

Abstract: A data analyzer operative to be located at a customer premises, causes the transmission of a probing signal into at least one telephone line. The data analyzer computes a parameter that represents a characteristic of at least one telephone line based upon at least a measured reflected probing signal.

Abstract: Voice user interfaces can be tested using test scripts generated by traversing a graph using a recursive search technique, such as recursive depth first search. Test data to accompany the test scripts can be generated by treating conditions in states of the interface as business rules, and fully exercising each state during traversal of the graph. Actual testing can be performed in a faster than real time manner by providing simulated input from test scripts, as opposed to requiring actual input from human testers. The interfaces can also be tested using a gatekeeper intermediary which provides for easy transitions between testing and deployment.

Abstract: A device under test (DUT) may be tested using a radio-frequency test station. A test station may include a test host, a test unit coupled to the test host, and a shielded enclosure. The shielded enclosure may contain a test antenna that is coupled to the test unit via a radio-frequency cable. The DUT may be placed in the shielded enclosure during testing. The DUT may not be electrically wired to the test host. The test host may direct the test unit to transmit control signals in the form of a text message using Short Message Service (SMS) so that the DUT is placed into different desired configurations during testing. The DUT may also be configured to respond by sending SMS messages back to the test unit, where the SMS messages generated by the DUT includes radio-frequency performance metrics measured using the DUT.

Abstract: In a computer, a test script is received that includes a first set of commands to be provided to an Internet protocol (IP) telephone. The test script is parsed to obtain the first set of commands. Parameters that identify the telephone are obtained. The parameters and at least one of the first set of commands are used to obtain at least one corresponding second command from a conversion table. A command script is generated to be executed by the telephone, the command script including the at least one corresponding second command.

Abstract: Voice user interfaces can be tested using test scripts generated by traversing a graph using a recursive search technique, such as recursive depth first search. Test data to accompany the test scripts can be generated by treating conditions in states of the interface as business rules, and fully exercising each state during traversal of the graph. Actual testing can be performed in a faster than real time manner by providing simulated input from test scripts, as opposed to requiring actual input from human testers. The interfaces can also be tested using a gatekeeper intermediary which provides for easy transitions between testing and deployment.

Abstract: A method for providing real time indicator of voice transmission quality in a Voice over Internet Protocol (VoIP) network is disclosed. The method allows to receive, at regular time interval during a call, a digital signature from a source call device. The received signature is compared to a referential signature stored on the destination devices, and in case the received signature falls out of the referential signature, a quality indication message is transmitted back to the source call device.

Abstract: A data management tool is provided. The tool comprises a first processor, a server application that, when executed on the first processor, manages allocation of equipment serial numbers (ESNs) used in executing test cases related to mobile devices, aggregates test progress data in a defect board, maintains a test chat forum, and sends a notification of a defect when the defect is reported to the server application, a second processor, and a thick client that, when executed on the second processor, receives the notification of the defect, participates in the test chat forum and requests one of the equipment serial numbers from the server application.

Abstract: In one of many possible embodiments, an exemplary system includes a test management subsystem configured to provide a user portal to a user of a circuit provided by a service provider, the user portal including a tool enabling the user to select a signal loop for testing at least a section of the circuit, the signal loop being selected from a plurality of signal loop options. The system also includes a network management subsystem communicatively coupled to the test management subsystem, the network management subsystem being configured to receive data representative of the selection from the test management subsystem and instruct, based on the selection, a network device along the circuit to execute a loop-back mode. In certain embodiments, the selected signal loop defines a test pattern signal flow for testing a subsection of the circuit.

Abstract: The present invention relates to methods and systems for minimizing alien crosstalk between connectors. Specifically, the methods and systems relate to isolation and compensation techniques for minimizing alien crosstalk between connectors for use with high-speed data cabling. A frame can be configured to receive a number of connectors. Shield structures may be positioned to isolate at least a subset of the connectors from one another. The connectors can be positioned to move at least a subset of the connectors away from alignment with a common plane. A signal compensator may be configured to adjust a data signal to compensate for alien crosstalk. The connectors are configured to efficiently and accurately propagate high-speed data signals by, among other functions, minimizing alien crosstalk.

Abstract: The present invention provides a system and method for a computer controlled test system in which the computer can change and monitor the compound observable state of an electronic device or telephone during testing procedure. In the preferred embodiment of the invention, the compound observable state of the electronic device includes a display bit map file or the status of lights or buttons. The compound observable state can be evaluated by the computer in real-time to alert nearby operators of a failure (i.e. a telephone that needs to be rebooted) or malfunction (not rising to the level of a failure). Also, the data related to the compound observable state can be stored in the computer for later review to assist in debugging the telephone's software.

Abstract: A method for generating reliability tests for a telephone system is based upon sampling an orthogonal array which covers various combinations of test parameters. Field data is collected of actual telephone activity on a telephone system. The field data is evaluated so as to determine call-mix characteristics. Probabilistic weights for the different call-mix characteristics are obtained, and then the probabilistic weights are used to sample the test case scenarios generated in the orthogonal array which have the same call-mix characteristics. These test case scenarios are used to run tests on the telephone system. These tests are preferably performed using automated test scripts. After the test data is collected, reliability metrics are calculated from the test data.

Abstract: A system for causing application of sealing current to one or more local telephony loops has a current source; a switching interface for connecting the current source to the one or more telephony loop interfaces; and a machine-readable instruction accessible to the switching interface, the instruction providing intelligence to the switching interface for identifying the one or more loops to be serviced, the time period for current application to each loop serviced, and a sequence order when more than one loop is serviced.

Abstract: One preferred embodiment of the present invention provides a system and method for evaluating the performance of an inter-messaging network of voice mail systems. This preferred embodiment includes an evaluation test unit that connects to the inter-messaging network and requests a test data file to be retrieved from a particular voice mail system in the inter-messaging network. The requests for the test data file are generated according to a schedule of operation so that the requests are repeated within a defined time span. Accordingly, the performance of the inter-messaging network, as represented by the result of the request attempt, is evaluated according to a defined level of performance, such as a preferred time limit. Other systems and methods are also provided.

Abstract: A method (100) and system (30) to enable automatic generation and testing of voice applications includes generating (102) a test driver application (TDA) (32) and generating (104) a modified original voice application (34) to be tested by the TDA within a call flow builder (10). The modified application can include or generate (106) “test hooks” or more particularly DTMF tones and DTMF grammars that can be used to synchronize the modified original voice application with the TDA. The TDA can test (110) all possible paths of the modified original voice application. Note the TDA and the modified original voice application can be generated and/or tested (112) in a test environment within the call flow builder or a telephony environment. The TDA can be automatically generated (108) to exercise all possible flows where the DTMF tones define the current state and location of the modified application.

Abstract: A dynamic service application is stored and executed on a mobile or wireless communication device (e.g., a cellular telephone) to enable it to be programmed without specialized hardware, software, and other proprietary information. For example, dynamic service application programs or scripts may be entered directly by a user or may be received as a wireless or radiated digital message transmission. The dynamic service application executes the dynamic service application script, which is of a format to accommodate wireless or radiated transmission and storage on the device. The dynamic service application script may be written directly by a user on a computer or a mobile communication device or may be written with the aid of scripting “wizard” software that runs on a computer and guides the writing of script without the user having to work directly with the dynamic service application script.

Abstract: The present invention disclosed a kind of simulate user calling's test system and method which built-in digital SPC exchange, include background processing module, foreground calling control processing module and hardware subsystem, therein: background processing module operation on exchange servicing platform, for supply user setting parameter and display operate interface for test result, foreground calling control processing module is include in the exchange main control module, for control said hardware subsystem execute test process according to designed logical flow and user mount parameter, hardware subsystem composed of loop circuit relay single board, simulation user interface board, interface board control processing unit, multifunction resources process board. Adopt present invention may use few cost to reach the test result which equal to commercial calling device, and may reach more mobility, reach inline test function.

Abstract: The subject of this patent is a system to generate RF frequency hopped signals that can be used to test radio receivers over the range of their signal specifications. The hop sequence and symbols in the signal is decoded from signals transmitted by the same or different radio of the same specification. The system can set various signal parameters to any prescribed value. These parameters include carrier frequency error, bit rate error, rise and fall times, amplitude ripple and roll off, modulation depth, nonlinearities, burst duration, burst start time and burst interval. Signal interference can be added to the signal including tones, other signals, noise and other signals of the same as the test radio. Similarly other communications equipment can be tested by the combination of receiving and demodulating a transmitted and regenerating the signal with the same symbols with specified signal parameters.

Abstract: A device for testing a data carrying service operating over a telecommunications line includes a plurality of test circuits. Each test circuit is arranged to determine and test one or more characteristics of a data carrying service and a termination of the telecommunications line is emulated. The operation of each of the plurality of test circuits is controlled, and the telecommunications lines is connected to one or more of the plurality of test circuits. The data carrying service remains connected via a connection during all of the testing operations provided by one or more of the test circuits which enables the device to automatically determine the identity of the data carrying service. The test circuits may be arranged to enable emulation of a termination to be interconnected so as to enable the device to provide a passive link in the data carrying service and/or provide throughput testing of the telecommunications line.

Abstract: A method and apparatus are disclosed that provide a technique for testing a data-processing system that enables communications between two or more telecommunications endpoints. An example of such a data-processing system is a Session-Initiation Protocol (SIP) proxy server. In particular, a test-execution system, in accordance with the illustrative embodiment of the present invention, transmits test-related signals to and receives test-related signals from one or more endpoints, such as SIP-enabled telephones. Some of the transmitted test-related signals direct an endpoint to transmit a message, such as a session-initiation request, to the data-processing system under test. Some of the received test-related signals comprise information from an endpoint, such as whether or not a message has been received from the data-processing system under test.

Abstract: Methods, apparatus, and computer program products are provided for testing a Voice Over Internet Protocol (VoIP) communication system. A VoIP test phone call is generated through an Internet communication link to a VoIP service provider for a called telephone number. Success of the VoIP test phone call is determined. The success of the VoIP test phone call may be determined based on whether it is established with communication equipment at the called telephone number within a threshold time, based on whether it is established within a threshold number of attempts, and/or based on a measured quality of service in the VoIP test phone call when it is established. Further VoIP test phone calls to test the VoIP communication system can be scheduled.

Abstract: A remote test unit and method of operation are provided for utilizes the ability of an access matrix ability to route signals. The Remote Test Unit can emulate a central Digital Subscriber Line Modem (DSLM-C) for testing customer premises equipment containing a remote Digital Subscriber Line Modem (remote Digital Subscriber Line Modem). The Remote Test Unit can also emulate a remote Digital Subscriber Line Modem for testing central offices equipment including a digital subscriber line access multiplexer (DSLAM) containing a central Digital Subscriber Line Modem. The Remote Test Unit can also emulate a concentrator connected to the DSLAM, a router connected to the concentrator, an Internet service provider (ISP) connected to the router, and a web site connected to the Internet service provider over the Internet.

Abstract: A device for testing a data carrying service operating over a telecommunications line includes a plurality of test circuits. Each test circuit is arranged to determine and test one or more characteristics of a data carrying service and a termination of the telecommunications line is emulated. The operation of each of the plurality of test circuits is controlled, and the telecommunications lines is connected to one or more of the plurality of test circuits. The data carrying service remains connected via a connection during all of the testing operations provided by one or more of the test circuits which enables the device to automatically determine the identity of the data carrying service. The test circuits may be arranged to enable emulation of a termination to be interconnected so as to enable the device to provide a passive link in the data carrying service and/or provide throughput testing of the telecommunications line.

Abstract: Automated tests are scheduled for a telecommunication network. A time slot array is created and a subset of the time slots are marked as unavailable according to respective maintenance windows. A remainder of the time slots are marked as available. Trunk groups are sequentially selected for respectively scheduling the tests. A particular available time slot is identified in the array for both of the testing units corresponding to the respective trunk group. The slot is marked as unavailable and then each trunk circuit is sequentially selected within the respective trunk group. A respective test is designated for a selected trunk circuit during the particular time slot and a counter is incremented. A check is made whether the counter indicates that a number of the tests have been scheduled that fills a capacity of the testing units for running simultaneous tests. If the capacity is not filled, then the method repeats the designating step for a next selected trunk circuit using the particular time slot.

Abstract: A method for automatically tracing an interface for an exchange and a subscriber network, includes setting an operator, reference, a procedure for notifying an event are defined, so that a user could perform interface tracing for an exchange and a subscriber network using a network management system such as EMS. The method for automatically tracing a subscriber applied to a communication system, includes: receiving setting particulars from an operator in order to perform tracing of a subscriber with respect to an interface for LE (Local Exchange) and a subscriber network (AN: Access network); checking setting particulars of the operator by receiving trace request for the interface from the operator, performing tracing relevant to the setting particulars, providing tracing results to the operator; and providing information for an event so that the operator could check the information upon occurrence of a trace event at an interface relevant to the setting particulars of the operator.

Abstract: A method for generating reliability tests for a telephone system is based upon sampling an orthogonal array which covers various combinations of test parameters. Field data is collected of actual telephone activity on a telephone system. The field data is evaluated so as to determine call-mix characteristics. Probabilistic weights for the different call-mix characteristics are obtained, and then the probabilistic weights are used to sample the test case scenarios generated in the orthogonal array which have the same call-mix characteristics. These test case scenarios are used to run tests on the telephone system. These tests are preferably performed using automated test scripts. After the test data is collected, reliability metrics are calculated from the test data.

Abstract: One preferred embodiment of the present invention provides a system and method for evaluating the performance of a network interface sub-system of an inter-messaging network of voice mail systems. This preferred embodiment includes a network query device that connects to the inter-messaging network and requests a test data file to be retrieved from a particular voice mail network interface sub-system in the inter-messaging network. The requests for the test data file are generated according to user command. Accordingly, the performance of a network interface sub-system in the inter-messaging network, as represented by the result of the request attempt, is evaluated according to a defined level of performance, such as a preferred time limit. Other systems and methods are also provided.

Abstract: A method of testing path a voice response system can include establishing a voice link between a test system and a voice response system and playing voice prompts to the test system over the voice link. The method also can include sending execution information to the test system over the voice link.

Abstract: An invention is provided for testing a quality of communication data received from a SUT. The invention includes storing reference test data comprising a plurality of data segments, and receiving degraded test data also comprising a plurality of data segments, from the SUT. The data segments are located in the degraded test data, and data segments in the degraded test data are matched to related data segments in the reference test data. Further, the data segments in the degraded test data are compared to corresponding data segments in the reference test data using a fixed point operation. In one aspect, the degraded test data can be normalized prior to locating the data segments, for example, utilizing a fixed point Fourier transform. Also, a receive filter can be applied to the normalized test data utilizing a fixed point operation.

Abstract: A system and method for emulating a telephone driver of a mobile device to assist in the development and testing of mobile telephony applications is described. An emulated telephony driver maintains internal states of a wireless network and characteristics of the mobile device. A wireless network is simulated such that neither a cellular radio nor a mobile subscription are required for handset application development and testing.

Abstract: One preferred embodiment of the present invention provides a system and method for evaluating the performance of an inter-messaging network of voice mail systems. This preferred embodiment includes an evaluation test unit that connects to the inter-messaging network and requests a test data file to be retrieved from a particular voice mail system in the inter-messaging network. The requests for the test data file are generated according to a schedule of operation so that the requests are repeated within a defined time span. Accordingly, the performance of the inter-messaging network, as represented by the result of the request attempt, is evaluated according to a defined level of performance, such as a preferred time limit. Other systems and methods are also provided.

Abstract: A system and method for multi-modal monitoring of a network are disclosed. In one embodiment of a system incorporating teachings of the present disclosure, a network management engine may be capable of issuing a request to a monitored network device and to receive network information from the monitored network device. In some embodiments, a multi-modal administration engine may allow a calling party to interact with the network management engine. A caller response unit associated with the multi-modal administration engine, may be capable of receiving a user input from the calling party and converting the user input into a request for network information. In preferred embodiments, a format converter associated with the multi-modal administration engine may translate at least a portion of a first signal representing network information into a second signal representing an audible sound.

Abstract: A remote test unit (RTU) and method of operation are provided for utilizes the ability of an access matrix ability to route signals. The RTU can emulate a central Digital Subscriber Line Modem (DSLM-C) for testing customer premises equipment containing a remote Digital Subscriber Line Modem (DSLM-R). The RTU can also emulate a DSLM-R for testing central offices equipment including a digital subscriber line access multiplexer (DSLAM) containing a DSLM-C. The RTU can also emulate a concentrator connected to the DSLAM, a router connected to the concentrator, an Internet service provider (ISP) connected to the router, and a web site connected to the ISP over the Internet. The RTU can further test, using emulation, ISO/OSI layers defined in the ISO/OSI reference model which are connected to the DSLAM.

Abstract: A system for monitoring and testing network elements of a wireless communication network includes at least one air surveillance device (ASD) connected to a target network element through a wire or wireless channel, and an air surveillance and management server (ASMS) which communicates with the ASD through the wireless communication network. The ASD collects radio environment data in real time, tests qualities of services around the target network element, and transmits the collected radio environment and test result data to the ASMS. The ASMS remotely controls the ASD, analyzes the data received from the ASD, and display an analysis result on a graphic user interface (GUI).

Abstract: A multiplexing apparatus selectively performs cell discard processing in the case of congestion on the basis of a use state of the same connection formed by cells from the side of a switching unit and subscribers without installing UPC units, and the multiplexing apparatus, which is connected to the switching unit and each of the plural subscribers through communication lines.

Abstract: A multi-unit test system performs automated and dynamically alterable, double-ended testing of telephone lines, whose service is provided on copper cable pairs. A prescribed set of (e.g., twelve) individual tests are executed either from a hand-held measurement unit, or by a central office-resident direct access test unit, or by a collaboration of the two. The hand-held measurement unit and the direct access test unit exchange FSK-based test control messages with one another that are effective to cause a selected electrical condition to be applied to a first portion of the wireline and to cause a prescribed electrical measurement to be made at a second portion of the wireline.

Abstract: A method of establishing a transmission power level for signals transmitted in a conductor is provided in an environment where signal interference is present between the signals in the conductor and separate signals carried in a proximate conductor. The method includes the steps of determining a reference signal interference level at an end of the proximate conductor from a reference signal capable of being propagated from an opposing end of the proximate conductor and determining a transmission power level for the signals to provide a signal interference level based on the reference signal interference level. A system embodying the method is also provided.

Abstract: An apparatus for and method of remotely controlling electronic devices, such as telecommunication equipment attached to telephone lines, test equipment, and/or home electronics includes a master board (105) having its own IP address and arranged and constructed to receive (401) instructions via the internet (103) from a remote terminal (101). The instructions are interpreted and converted into one or more commands that are directed (405) to a slave board (113 or 117) that performs, in response to the command(s), one or more operations on a telephone line and its associated hardware and/or software (115), a piece of test equipment (125), lights (123), or other type of electronic device. Any results of the operations may be returned to the terminal (101).

Abstract: A method and apparatus for internally testing portions of a subscriber loop interface circuit. Testing may be conducted on a semiconductor circuit including the subscriber loop interface circuit. The portion of the subscriber loop interface circuit being tested may be the ring trip detector and/or the off-hook detect circuit. Testing may be conducted without disabling the capability of the subscriber loop interface circuit to monitor the hook status of a subscriber.

Abstract: A method and system for evaluating telephone services provided by speech recognition interfaces an evaluation engine with a voice recognition service over a telephone system to submit speech utterance samples to the voice recognition service, receive the response of the voice recognition service to the sample utterances, and determine error and recognition of the sample utterances by the voice recognition service by comparing actual voice recognition service responses to expected responses. The evaluation engine permits evaluation of a voice recognition service for plural glossaries in different contexts, such as through predetermined nodes of a voice recognition service menu having plural glossaries.