US20140229236A1 - User Survey Service for Unified Communications - Google Patents

User Survey Service for Unified Communications Download PDF

Info

Publication number
US20140229236A1
US20140229236A1 US14/178,238 US201414178238A US2014229236A1 US 20140229236 A1 US20140229236 A1 US 20140229236A1 US 201414178238 A US201414178238 A US 201414178238A US 2014229236 A1 US2014229236 A1 US 2014229236A1
Authority
US
United States
Prior art keywords
user
survey
calls
server
service
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Abandoned
Application number
US14/178,238
Inventor
Sudhanshu Aggarwal
Arun Raghavan
Robert Osborne
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Unify Square Inc
Original Assignee
Unify Square Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Unify Square Inc filed Critical Unify Square Inc
Priority to US14/178,238 priority Critical patent/US20140229236A1/en
Publication of US20140229236A1 publication Critical patent/US20140229236A1/en
Assigned to VENTURE LENDING & LEASING VII, INC. reassignment VENTURE LENDING & LEASING VII, INC. SECURITY INTEREST Assignors: Unify Square, Inc.
Assigned to Unify Square, Inc. reassignment Unify Square, Inc. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: AGGARWAL, SUDHANSHU, OSBORNE, ROBERT, RAGHAVAN, Arun
Assigned to WESTERN ALLIANCE BANK reassignment WESTERN ALLIANCE BANK SECURITY INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: Unify Square, Inc.
Assigned to Unify Square, Inc. reassignment Unify Square, Inc. RELEASE BY SECURED PARTY (SEE DOCUMENT FOR DETAILS). Assignors: VENTURE LENDING & LEASING VII, INC.
Assigned to Unify Square, Inc. reassignment Unify Square, Inc. RELEASE BY SECURED PARTY (SEE DOCUMENT FOR DETAILS). Assignors: WESTERN ALLIANCE BANK
Abandoned legal-status Critical Current

Links

Images

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04MTELEPHONIC COMMUNICATION
    • H04M3/00Automatic or semi-automatic exchanges
    • H04M3/22Arrangements for supervision, monitoring or testing
    • H04M3/2227Quality of service monitoring
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06QINFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00Administration; Management
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06QINFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00Commerce
    • G06Q30/02Marketing; Price estimation or determination; Fundraising
    • G06Q30/0201Market modelling; Market analysis; Collecting market data
    • G06Q30/0203Market surveys; Market polls
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/50Network service management, e.g. ensuring proper service fulfilment according to agreements
    • H04L41/5032Generating service level reports
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/50Network service management, e.g. ensuring proper service fulfilment according to agreements
    • H04L41/508Network service management, e.g. ensuring proper service fulfilment according to agreements based on type of value added network service under agreement
    • H04L41/5096Network service management, e.g. ensuring proper service fulfilment according to agreements based on type of value added network service under agreement wherein the managed service relates to distributed or central networked applications
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L43/00Arrangements for monitoring or testing data switching networks
    • H04L43/06Generation of reports
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L43/00Arrangements for monitoring or testing data switching networks
    • H04L43/08Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
    • H04L43/0805Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters by checking availability
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L43/00Arrangements for monitoring or testing data switching networks
    • H04L43/50Testing arrangements
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L43/00Arrangements for monitoring or testing data switching networks
    • H04L43/50Testing arrangements
    • H04L43/55Testing of service level quality, e.g. simulating service usage
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L51/00User-to-user messaging in packet-switching networks, transmitted according to store-and-forward or real-time protocols, e.g. e-mail
    • H04L51/56Unified messaging, e.g. interactions between e-mail, instant messaging or converged IP messaging [CPM]
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/10Architectures or entities
    • H04L65/1063Application servers providing network services
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/50Network services
    • H04L67/535Tracking the activity of the user
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04MTELEPHONIC COMMUNICATION
    • H04M15/00Arrangements for metering, time-control or time indication ; Metering, charging or billing arrangements for voice wireline or wireless communications, e.g. VoIP
    • H04M15/58Arrangements for metering, time-control or time indication ; Metering, charging or billing arrangements for voice wireline or wireless communications, e.g. VoIP based on statistics of usage or network monitoring
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04MTELEPHONIC COMMUNICATION
    • H04M3/00Automatic or semi-automatic exchanges
    • H04M3/22Arrangements for supervision, monitoring or testing
    • H04M3/2236Quality of speech transmission monitoring
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04WWIRELESS COMMUNICATION NETWORKS
    • H04W68/00User notification, e.g. alerting and paging, for incoming communication, change of service or the like
    • H04W68/005Transmission of information for alerting of incoming communication
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/01Protocols
    • H04L67/10Protocols in which an application is distributed across nodes in the network
    • H04L67/1001Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers

Definitions

  • UC services include communication services (e.g., e-mail services, instant messaging services, voice communication services, video conference services, and the like) and UC data management and analysis services.
  • communication services e.g., e-mail services, instant messaging services, voice communication services, video conference services, and the like
  • UC data management and analysis services e.g., UC data management and analysis services.
  • UC platforms allow users to communicate over internal networks (e.g., corporate networks) and external networks (e.g., the Internet). This opens communication capabilities not only to users available at their desks, but also to users who are on the road, and even to users from different organizations. With such solutions, end users are freed from limitations of previous forms of communication, which can result in quicker and more efficient business processes and decision making.
  • internal networks e.g., corporate networks
  • external networks e.g., the Internet
  • the quality of communications in such platforms can be affected by a variety of problems, including software failures, hardware failures, configuration problems (e.g., system-wide or within components (e.g., firewalls, load balancers)), and network performance problems.
  • the potential impacts of these and other problems include immediate impact upon end users (both internal and roaming) and inefficient use of functionality that increases overall costs.
  • UC platforms may enable, a 100,000 user enterprise may accumulate on the order of 1 billion call records and 1 terabyte of data per year. Formally maintaining this data as an accurate and persistent long-term repository for reference and analysis can help an enterprise to meet its technical, business, and compliance needs.
  • Embodiments disclosed herein include computer systems and methods for using presence information to survey users.
  • a described UC system includes a survey service server that communicates with devices associated with a customer site.
  • a user survey service after determining if a user is available (e.g., online and involved in a meeting or other activity) based on presence information, a user survey service initiates a survey at that time via a real-time communication mechanism (e.g., instant messaging).
  • a real-time communication mechanism e.g., instant messaging
  • FIG. 1 is a block diagram that illustrates a generalized UC management and analysis system according to various aspects of the present disclosure
  • FIG. 2 is a block diagram that illustrates another example of a unified communication management and analysis system
  • FIG. 3 is a line graph that shows a percentage of poor calls for a relevant trailing period
  • FIG. 4 shows an example of a detailed report titled “Poor Calls Network Breakdown”
  • FIG. 5 shows an example of a detailed report titled “Poor Calls Geography Breakdown”
  • FIG. 6 shows a table with information relating to a call
  • FIG. 7 shows a user interface for accessing responses to a survey relating to voice quality
  • FIG. 8 is a flow chart that shows how targeted notifications and user feedback can be used to improve overall voice quality
  • FIG. 9 is a screenshot of an e-mail generated by a message generation component
  • FIG. 10 is a screenshot of an instant message (IM) generated by a message generation component
  • FIG. 11 is a screen shot of a main landing page titled “Communication Health Report” for a user
  • FIG. 12 illustrates a distributed real-time communications (RTC) monitoring system comprising a monitoring service, a cloud database, and transaction executors (agents);
  • RTC real-time communications
  • FIG. 13 illustrates a version of a rule-load balancing algorithm
  • FIG. 14 is a graph that illustrates a customer's availability and an average availability for all customers
  • FIG. 15 shows an alert generated by a monitoring service
  • FIG. 16 is a diagram of a UC system that includes a user survey service
  • FIG. 17 is a flow chart that illustrates a survey technique that employs presence information.
  • FIG. 18 is a block diagram that illustrates aspects of an exemplary computing device appropriate for use in accordance with embodiments of the present disclosure.
  • UC systems such as UC management and analysis systems, tools, and techniques.
  • UC systems such as UC systems based on the Lync platform available from Microsoft Corporation
  • UC services include communication services (e.g., e-mail services, instant messaging services, voice communication services, video conference services, and the like) and UC data management and analysis services, or other services.
  • communication services e.g., e-mail services, instant messaging services, voice communication services, video conference services, and the like
  • UC data management and analysis services or other services. Representative UC management and analysis services are described in detail below.
  • FIG. 1 is a block diagram that illustrates a generalized UC management and analysis system 100 according to various aspects of the present disclosure.
  • the system 100 includes client computing devices 102 A-N, a server 106 , and an administrator computing device 108 .
  • the components of the system 100 may communicate with each other via a network 90 .
  • the network 90 may comprise a wide-area network such as the Internet.
  • the network 90 may comprise one or more sub-networks (not shown).
  • the network 90 may include one or more local area networks (e.g., wired or wireless local area networks) that may, in turn, provide access to a wide-area network such as the Internet.
  • the client devices 102 A-N may be computing devices operated by end users of a UC system.
  • a user operating the administrator device 108 may connect to the server 106 to, for example, manage and analyze use of the UC system.
  • FIG. 2 is a block diagram that illustrates another example of a unified communication management and analysis system.
  • the system 200 comprises a client computing device 202 , a server 206 , and an administrator computing device 208 .
  • the server 206 comprises a data store 220 and a UC management and analysis engine 222 .
  • the data store 220 stores data that relates to operation and use of UC system, as will be further described below.
  • the management and analysis engine 222 interacts with the data store 220 .
  • the data store 220 can store data and definitions that define elements to be displayed to an end user on a client device 202 or administrator device 208 .
  • the data store 220 can store data that describes the frequency, quality, and other characteristics of communications (e.g., voice communications) that occur across an enterprise via a UC system.
  • a definition defining a set of interface elements can be used to present a graphical user interface at administrator device 208 that can be used by a system administrator that is seeking to diagnose the cause of a reported problem in the UC system, as explained in detail below.
  • a definition defining a set of interface elements can be used to present a graphical user interface at client device 202 to guide an end user to respond to a survey relating to the end user's experience with the UC system, as explained in detail below.
  • Interface elements such as text boxes, soft buttons, checkboxes, drop-down boxes, multimedia interface elements (e.g., audio or video players), and/or the like, may receive input from or present output (e.g., to an end user or system administrator).
  • the client device 202 includes output device(s) 210 , input device(s) 212 , and a UC client engine 214 .
  • the UC client engine 214 is configured to process input and generate output related to UC services and content (e.g., services and content provided by the server 206 ).
  • the UC client engine 214 also is configured to cause output device(s) 210 to provide output and to process input from input device(s) 212 related to UC services.
  • input device(s) 212 can be used to provide input (e.g., text input, video input, audio input, or other input) that can be used to participate in UC services (e.g., instant messages (IMs), voice calls), and output device(s) 210 (e.g., speakers, a display) can be used to provide output (e.g., graphics, text, video, audio) corresponding to UC services.
  • input e.g., text input, video input, audio input, or other input
  • output device(s) 210 e.g., speakers, a display
  • output e.g., graphics, text, video, audio
  • the administrator device 208 includes output device(s) 230 , input device(s) 232 , and UC administrator engine 234 .
  • the UC administrator engine 234 is configured to receive, send, and process information relating to UC services.
  • the UC administrator engine 234 is configured to cause output device(s) 230 to provide output and to process input from input device(s) 232 related to UC services.
  • input device(s) 232 can be used to provide input for administering or participating in UC services
  • output device(s) 230 can be used to provide output corresponding to UC services.
  • the UC client engine 214 and/or the UC administrator engine 234 can be implemented as a custom desktop application or mobile application, such as an application that is specially configured for using or administering UC services.
  • the UC client engine 214 and/or the UC administrator engine 234 can be implemented in whole or in part by an appropriately configured browser, such as the Internet Explorer® browser by Microsoft Corporation, the Firefox® browser by the Mozilla Foundation, and/or the like. Configuration of a browser may include browser plug-ins or other modules that facilitate instant messaging, recording and viewing video, or other functionality that relates to UC services.
  • an “engine” may include computer program code configured to cause one or more computing device(s) to perform actions described herein as being associated with the engine.
  • a computing device can be specifically programmed to perform the actions by having installed therein a tangible computer-readable medium having computer-executable instructions stored thereon that, when executed by one or more processors of the computing device, cause the computing device to perform the actions.
  • An exemplary computing device is described further below with reference to FIG. 18 .
  • the particular engines described herein are included for ease of discussion, but many alternatives are possible. For example, actions described herein as associated with two or more engines on multiple devices may be performed by a single engine. As another example, actions described herein as associated with a single engine may be performed by two or more engines on the same device or on multiple devices.
  • a “data store” contains data as described herein and may be hosted, for example, by a database management system (DBMS) to allow a high level of data throughput between the data store and other components of a described system.
  • the DBMS may also allow the data store to be reliably backed up and to maintain a high level of availability.
  • a data store may be accessed by other system components via a network, such as a private network in the vicinity of the system, a secured transmission channel over the public Internet, a combination of private and public networks, and the like.
  • a data store may include structured data stored as files in a traditional file system. Data stores may reside on computing devices that are part of or separate from components of systems described herein. Separate data stores may be combined into a single data store, or a single data store may be split into two or more separate data stores.
  • Examples in this section describe features of an end-to-end solution for enterprise-level unified communication (UC) data capture, analysis, and reporting. As with other examples described herein, the examples in this section can be used with enterprise-level UC systems.
  • UC enterprise-level unified communication
  • a UC system with enhanced data capture, analysis, and reporting capabilities as described herein can include one or more of the features described with reference to Examples 1-10 below. More generally, a comprehensive UC system with enhanced data capture, analysis, and reporting capabilities can provide the following functionality:
  • a UC system with enhanced data capture, analysis, and reporting capabilities can facilitate cost savings through consolidation, such as by (1) consolidating/replacing hundreds or thousands of disparate PBXs into one centralized global infrastructure; (2) consolidating multiple communications infrastructure components such as audio conferencing, instant messaging, application sharing, video conferencing, etc., into a single infrastructure; and (3) consolidating both internal and remote/external communications by employees, customers, partners, and suppliers into a single infrastructure.
  • Productivity gains can be realized through an increase in collaboration and the speed of business, via an innovative and intuitive end-user experience.
  • a comprehensive UC system with enhanced data capture, analysis, and reporting capabilities can include:
  • An enterprise-wide data warehouse is described that consolidates communications activity in an enterprise into a single data store that provides insights into an enterprise's communication patterns.
  • the data warehouse includes the following features:
  • the data warehouse can pull data from the following sources:
  • Reporting on various business outcomes based on enterprise user communications activity is described. Reports are built on business models and algorithms that map user communication activity and other inputs (location, media) to financial metrics (cost, savings, etc.).
  • KPIs e.g., a selection of three or some other number of important KPIs
  • a set of reference base KPIs can be used to measure success of a UC platform.
  • KPIs can indicate overall effectiveness and efficiency of a UC platform deployment, and trends that inform the projected effectiveness and efficiency of the deployment.
  • KPIs can be used to identify “problem spots” in the deployment, track user adoption (which affects cost savings as well as user productivity), and identify opportunities to optimize return on investment in the deployment.
  • a KPI is used to help determine compliance with SLAs. Further details on SLA compliance are provided in the example below.
  • a technique for classification of calls using location/subnet information, call metrics and algorithms for determining SLA intervals, and time slices based on configurable thresholds.
  • the example technique may include following processing steps:
  • communications activity and reports are secured centrally and made selectively available to users based on various “personas” (e.g., business function or organizational/administrative functions). Access can be scaled from a group level to an individual level. Permissions settings can be used to define different levels of access. Data access also can be restricted based on personas. For example, a user may be restricted to only viewing data controlled by his department, and not other departments, within an organization.
  • personas e.g., business function or organizational/administrative functions.
  • Access can be scaled from a group level to an individual level. Permissions settings can be used to define different levels of access.
  • Data access also can be restricted based on personas. For example, a user may be restricted to only viewing data controlled by his department, and not other departments, within an organization.
  • a quality assessment and classification tool can include the following functionality:
  • Maintaining acceptable audio quality requires an understanding of UC system infrastructure and proper functioning of the network, communication devices, and other components.
  • An administrator will often need to be able to quantifiably track overall voice quality in order to confirm improvements and identify areas of potential difficulty (or “hot spots”) that require further effort to resolve.
  • One way to track audio quality is through reports.
  • an administrator can identify hot spots to address and also convey (e.g., to senior management) information that supports broader conclusions about the system (e.g., that a system deployment is being successful over time, or that more investment is required).
  • SLA reporting may focus on sites as defined by subnet. However, not all customers may define subnets, or have the information to configure sites. Additionally, it is a complex process to keep subnet mapping accurate and up to date. However, there is a different set of information which is available, which could provide a very close approximation to users location, and that is geography information. Therefore, to provide an easier deployment model which allows for quicker SLA reports, it can be useful to allow for customers to utilize this same information.
  • SLA reports also can be used to break down call quality into different aspects which may have impacted the quality of those calls.
  • factors that could impact audio quality are: (a) the split of wired vs. wireless calls (potentially, audio quality impacts can be due to wireless issues); (b) device characteristics (devices can impact audio quality as perceived by the end user, especially unsupported devices or those without the correct drivers); (c) the effects of gateways between devices; (d) remote users vs. users local to known sites (e.g., if most of the audio quality issues are driven by remote users, this information can be very useful). Identifying situations that may apply with respect to factor (b), above, may require not utilizing network QoE metrics, but other metrics such as Sending MOS (quality of audio stream being sent from user).
  • Sending MOS quality of audio stream being sent from user.
  • This section describes examples of information that can be used for enhanced voice quality analysis and reporting.
  • wireless calls which have poor voice quality are important to group together to identify common patterns (e.g., whether the calls involve the same user) and to take appropriate action (e.g., educate the user to not use wireless, upgrade the wireless infrastructure).
  • Some problems may have more impact on voice quality than others, even within the same call.
  • a user who is using a wireless connection and is roaming outside the user's usual network may be calling another user who is on the corporate network using a wired connection.
  • the overall experience may be impacted by the first user's wireless connection.
  • An analysis of the conditions at the two endpoints can be conducted to determine which endpoint is more likely to impact a call and highlight one or more items to consider addressing (e.g., by encouraging a user to switch from a wireless connection to a wired connection for the next call).
  • Table 1 below includes examples of expected classifications of calls within the UC system.
  • a call with two endpoints is classified based on the endpoint with the lowest quality classification. For example, if a first endpoint uses a wireless connection and a second endpoint has similar conditions except that the second endpoint uses a wired, corporate connection, the call will be classified based on the first endpoint.
  • the following table is ordered with worst case being listed first:
  • Classification Scenario Types of issues User VPN Incorrectly utilizing VPN to access the network and, (P-U) by consequence, using audio over TCP. May be wired or wireless; until VPN is addressed it is hard to assess other impacts. Admin needs to determine the number of calls and whether this is associated with a certain set of users, and allow for user education. User Wireless/External: Though potentially there are issues (e.g., with an edge (P-U) User is external to server) independent of wireless, admin first needs to the main network address users use of wireless. Admin needs to and is using determine the number of calls and whether this is wireless. associated with a certain set of users, and allow for user education.
  • Admin needs to determine the number of calls and (P-U) User is on internal whether this is associated with a certain set of users, network, but is and allow for user education. using wireless.
  • External Federated Federation allows users in one enterprise (or (P-E) organization) to communicate with users in another enterprise (or organization). Users in the “Federated” enterprise are called “Federated partners.” Calls to/from a specific federated partner may be poor because of internal challenges or a specific federated partner's network/infrastructure. Being able to identify if all federated partners are having issues (e.g., with a set of internal users) or a specific partner is important.
  • MCUs multipoint control units
  • P-C multipoint control units
  • grouping of MCUs are also shown, to allow for potentially a single MCU or a pool of MCUs to be highlighted as impacting audio quality.
  • the audio quality may be (P-C) (bypass): impacted by the users' locations on the network, the Calls between a GW data center network, the GW being overloaded, or user's PC and the a bad configuration. Can break down based on user gateway (GW) organization/geographic area to identify potential which are locations that are having bad audio quality due to local bypassing the network issues.
  • GW user gateway
  • the grouping of MS's are mediation server also shown, to allow for potentially a single MS or (MS). pool of MS's to be highlighted as impacting audio quality.
  • the audio quality may be (P-C) (non-bypass): impacted by the users' locations on the network, the Calls between a MS data center network, the MS being overloaded, or user's PC and the a bad configuration. Can break down based on user mediation server. organization/geographic area to identify potential locations that are having bad audio quality due to local network. In addition, the grouping of MS's are also shown, to allow for potentially a single MS or pool of MS's to be highlighted as impacting audio quality.
  • P-C non-bypass
  • the audio quality may be (P-C) (non-bypass): impacted by the MS locations on the network or data Calls between an center, or GW locations on the network or data center MS and the GW. network, or by either being overloaded, or a bad configuration. Can break down based on server site of the MS or GW.
  • P-C non-bypass
  • the breakdown could be based on any of several factors (e.g., time of day, specific site, etc.), in at least one embodiment the break down is based on geography. This has the advantage of being generally aligned with users' interactivity (e.g., users who are in the Singapore geography are likely using the Singapore network more often) and any future training requirements. If geography information is not available or reliable, the value of breaking down the classification in this way is reduced.
  • Additional infrastructure components may exist within the same geographies as users, and can be within the same offices or, potentially, in unique locations (e.g., data centers). It is possible to have additional locations added to the existing geography hierarchy, with the potential to break these down to the calls associated with particular users or infrastructure components.
  • federation For federation, it is expected that although federated partners may share some of the same locations as an organization's geography, it may not be possible to confirm the location, since additional geographical information is not available. Therefore, federation can be a new element in the top level hierarchy with the ability to break down to each individual partner.
  • a poor call In order to determine what potential problems exist, it is vital to have a clear definition of what a poor call is, and what is an acceptable amount of poor calls.
  • the definition of a poor call can be provided by a UC platform, by a customer, or in some other way.
  • Some example thresholds for acceptable amounts of poor calls are as follows:
  • These thresholds can be set by default, and can be overridden if desired.
  • a customer can:
  • a user also called a “viewer” in this context
  • a global trends dashboard can provide a top level summary of information and trends. This can be at the global level or with the ability to select (via filters and hierarchies) trends for a certain classification/geography pairing.
  • a global trends dashboard can provide the following filters and reports:
  • the viewer is interested in knowing the biggest problem areas that require more investigation across all possible areas.
  • the viewer does not want to browse all possible areas, but instead to be quickly directed to specific areas to focus on (e.g., when a lot of users are using VPN).
  • the viewer has a specific theory or potential problem that they wish to investigate. For example, a lot of users in a certain geography are complaining about poor voice quality, but no root cause is known. In this case, the viewer wishes to see all information about that specific geography, including all call classifications, and then carry out further investigations to identify what is common to the complaining users.
  • a top/specific problems dashboard can use call classification as a first level of the hierarchy that can be broken down by geography, or can use geography as a first level of the hierarchy that can be broken down by call classification.
  • a top/specific problems dashboard also can use site/subnet mapping, which can then be broken down by call classification.
  • this dashboard will allow an organization to select a date range that is appropriate and see information associated with the call classifications as the most important grouping. This allows the organization to theorize that, for example, users are using wireless too much, and then find out which geography or geographies of users are using wireless too much. In addition, to save the viewer from having to drill down into all possible combinations of call classification and geography, a report can show the top call classification/geographies that have the worst poor call percentage.
  • a call classification breakdown dashboard can provide the following filters and reports:
  • Reports can be formatted for viewing in a variety of ways. For example, Reports 1-4 above can be presented side by side, with each report in a table format similar to the example table for Report 1 provided in Table 3, above, or in some other layout, to give a user a convenient view of the reported information (e.g., top problems).
  • FIG. 4 shows an example of a detailed report titled “Poor Calls Network Breakdown.”
  • the report in FIG. 4 shows network and geography information along with percentages of poor audio calls, by month.
  • the shaded cells or data points in FIG. 4 are highlighted to indicate (e.g., to an administrator) that poor call thresholds have been exceeded.
  • this dashboard will allow an organization to select a date range that is appropriate and see information associated with the geography hierarchy as the most important grouping. This allows the organization to theorize that, for example, a geography of users is having a significant problem and drill down into the call classifications to see if this problem is consistent across all call types or for a specific type of call. In addition, to save the viewer from having to drill down into all possible combinations of geography and call classification, a report can show the top geographies/call classification that have the worst poor call percentage.
  • a Geography Breakdown dashboard can provide the following filters and reports:
  • FIG. 5 shows an example of a detailed report titled “Poor Calls Geography Breakdown.”
  • the report in FIG. 5 shows network and geography information along with percentages of poor audio calls, by month.
  • the shaded cells or data points in FIG. 5 are highlighted to indicate (e.g., to an administrator) that poor call thresholds have been exceeded.
  • a Call Breakdown Report can provide the following filters and reports:
  • This report includes a table which displays a summary of all the poor calls that occurred within certain period.
  • enterprise calls are analyzed based on simultaneous events or conditions within an environment (e.g., user's environment, user's network/site, enterprise environment) and heuristics are utilized to establish correlation or cause-effect information for various call conditions and scenarios. For example, poor quality calls may be correlated with a user adding video and application sharing while on a low bandwidth connection.
  • environment e.g., user's environment, user's network/site, enterprise environment
  • features are described that facilitate proactively notifying users of conditions impacting call quality and reliability via instant messaging or other messaging channels (such as e-mail).
  • Users are notified based on the configurable metrics/parameters (which can be tuned by system administrators) and provided with information mined from call detail and voice quality records. This information is used to provide feedback to the user (e.g., feedback relating to call conditions, as well as other remediation recommendations).
  • a channel for users to provide feedback to operations teams is provided. Operational teams can be alerted to issues relating to specific user groups (e.g., executive users).
  • a real-time user notification service can monitor QoE servers or a UC data manager database and run a query periodically. Based on the result of the query, the service notifies end users. Both the notification message and the channel (e.g., IM, e-mail) can be configured.
  • the notification message and the channel e.g., IM, e-mail
  • the screen shot in FIG. 7 shows a user interface for accessing responses to a survey relating to voice quality.
  • the surveys themselves can be conducted using techniques and tools described in detail below.
  • survey responses associated with specific users are shown in a table.
  • Each row in the table includes a user ID (e.g., e-mail address), user comments, the user's overall voice quality rating (e.g., Very Satisfied, Somewhat Satisfied, Somewhat Dissatisfied, Very Dissatisfied, etc.), and a numeric QoE rating.
  • the numeric QoE rating is calculated based on QoE data for the user during a survey time interval.
  • the user interface allows clicking on the user ID to view additional information, such as per-user metrics (e.g., poor calls, QoE aggregate scores) compared against enterprise and/or industry benchmarks.
  • the user interface also allows selection of results corresponding to different surveys, which can be identified, for example, by the date on which the survey was conducted.
  • the user interface also provides information on how many comments a particular survey has generated.
  • the user interface includes functionality for graphing, commenting on, searching, and exporting information.
  • the user interface can be used for multiple organizations or companies, as shown in the “Select Company” drop-down box.
  • the user interface can include features for securely viewing such information (see the “Log Out” and “Change Password” links).
  • the user interface can be presented in a Web browser or as part of a dedicated, custom application.
  • FIG. 8 is a flow chart that shows how targeted notifications and user feedback can be used to improve overall voice quality.
  • the decision whether to send targeted notifications also can be based on a user's situation or a usage scenario.
  • a rules engine can detect if the user making a call was on Wifi and/or using unsupported devices, which can affect call quality.
  • the notification can be tailored to be appropriate to the situation. This results in a more accurate communication outreach and actionable results.
  • An operations team can maintain a record of users who have been contacted (or “pinged”) to limit the possibility that users will become annoyed or overwhelmed with information, while also allowing the team to determine if a follow-up message might be helpful.
  • Call quality metrics are stored in databases (e.g., QoE and/or CDR databases).
  • a voice quality rule engine monitors a database (e.g., a QoE database) for poor calls.
  • the voice quality rule engine triggers an outbound notification to an end user via a predetermined channel (e.g., IM, e-mail), in accordance with rules that apply to the conditions of a communication in which the user participated (e.g., rules related to WiFi communications, rules related to communications using unsupported devices, etc.).
  • a predetermined channel e.g., IM, e-mail
  • the triggered outbound notification is handled by a message generation component of a service that uses presence information to determine whether use IM or e-mail for delivering the notification. For example, if a user is detected to be online and available, the component can send an IM. As another example, if the user is detected to be unavailable (e.g., offline, or online but busy), the component can send an e-mail.
  • the user receiving the notification is given the option (e.g., via a link in the notification) to provide feedback for analysis.
  • FIG. 9 is a screenshot of an e-mail generated by a message generation component.
  • the “To:” field of the e-mail includes a white indicator graphic next to the user's e-mail address that provides presence information. In this example, the white color of the graphic indicates that the user is offline.
  • the “Bcc:” field includes a green indicator graphic that indicates that “End User Services” is online and available. (The green color is not shown in FIG. 9 .)
  • the text of the e-mail confirms that a call in which the user participated had poor voice quality, and suggests possible causes (poor wireless connectivity, unsupported audio device) of the poor voice quality.
  • the e-mail provides underlined links (see items numbered “1.” and “2.” in the text of the email message in FIG. 9 ) to help the addressee avoid similar problems in the future.
  • the e-mail also requests feedback and provides an underlined link in the “Feedback” section of the e-mail for this purpose.
  • FIG. 10 is a screenshot of an instant message (IM) generated by a message generation component.
  • IM instant message
  • a green indicator graphic indicates that “Survey Bot” is online and available. (The green color is replaced with dark shading in FIG. 10 .)
  • the text of the IM confirms that a call in which the user participated had poor voice quality, and suggests possible causes (poor wireless connectivity, unsupported audio device) of the poor voice quality.
  • the IM also requests feedback and provides a link (underlined in the “Feedback” section of the IM) for this purpose.
  • features are described that provide a per-user “score” for enterprise communications using an algorithm to compute a single score that takes into account the user's communication activity (based on various parameters and metrics), and that allow for benchmarking against a “peer group.”
  • FIG. 11 is a screen shot of a main landing page (“Communication Health Report”) for a user (“User 1”).
  • the main landing page also can include UC system availability, a user's open feedback items (e.g., if the user is connected to the UC system), and other messages about UC, such as training opportunities for offered UC services.
  • the various elements shown in FIG. 11 are only examples. The elements shown in FIG. 11 can be arranged in different ways. Further, individual elements shown in FIG. 11 can be omitted, supplemented with additional elements, and/or replaced with different elements showing different information.
  • FIG. 11 shows a meeting health index score tile 1110 .
  • a user's meeting health index score is a composite score that includes the type of network being used when the user is communicating, the audio device being used (e.g., headset, handset, microphone, external speakers) as well as the network and device behaviors of those who participate in meetings or calls with the user.
  • the maximum score is five stars
  • the user's score for the time period e.g., a week
  • An average score (e.g., for other users within the user's organization) also can be provided to the user, but is omitted from FIG. 11 .
  • the user can compare the user's individual score with the average score to get a better idea of how the user's meeting health index compares to the user's peers.
  • FIG. 11 also shows an audio device usage tile 1120 .
  • This tile includes a pie chart in which the fraction of calls that use approved devices and unsupported devices are shown, along with the fraction of calls in which the device being used actually caused audio quality issues.
  • FIG. 11 also shows a network awareness tile 1130 , which can be used to rate the user's overall network usage (e.g., “poor,” “fair,” or “excellent”) based on, for example, how often the user is participating in calls over communication channels (e.g., wireless channels) that tend to have lower voice quality.
  • a network awareness tile 1130 can be used to rate the user's overall network usage (e.g., “poor,” “fair,” or “excellent”) based on, for example, how often the user is participating in calls over communication channels (e.g., wireless channels) that tend to have lower voice quality.
  • FIG. 11 also shows a weekly metrics tile 1140 that includes counts of “good” and “poor” quality calls of different types. Alternatively, this tile can display different metrics and/or metrics for different time periods (e.g., monthly).
  • conference travel and lost opportunity cost savings are determined based on a calculation of what the estimated cost would have been for each enterprise participant for on-site conference attendance.
  • the model assumes that the location of the conference is the Organizer's location.
  • the UC data management system uses user geography information (e.g., region, country, state, and/or city) combined with a configurable travel probability matrix and associated travel and lost opportunity costs to determine cost savings.
  • user geography information e.g., region, country, state, and/or city
  • a configurable travel probability matrix e.g., associated travel and lost opportunity costs to determine cost savings.
  • the probability of the user traveling to the physical meeting location is based on the conference attendee count and the duration of the conference, as shown in Table 9, below:
  • An associated hourly travel and opportunity cost can be calculated based on a geographical difference (e.g., inter-region, inter-country) between the physical meeting location (which may be assumed to be the organizer's location) and the participant's location. For example, if a user in the United Kingdom is invited to a meeting in North America, the geographical difference is “inter-region,” whereas if the meeting is in France, the geographical difference is “inter-country.” These classifications can be adjusted, such as when a user is located in an isolated area of a large country, and inter-city travel is more expensive than for a user near a population center of a small country. Example calculations are shown in Table 10, below. The actual costs reflected in Table 10 can be adjusted. For example, costs may be increased over time as average travel costs increase. As another example, the opportunity cost of attending a meeting for a high-level executive may be significantly greater than the opportunity cost for the executive's assistant.
  • a geographical difference e.g., inter-region, inter-country
  • PII personally identifiable information
  • data obfuscation applies to all calls associated with a gateway (assumed to be PSTN calls), and the piece of data obfuscated is phone numbers.
  • the UC data management system allows an enterprise to determine when to obfuscate data (e.g., when data is imported, or a given number of days after the call occurred).
  • the format of the obfuscation can be, for example, as follows: +14253334444->+1425*******, where numerals represent numbers in a phone number, and * represents an obfuscated digit.
  • the number of digits to obfuscate (e.g., by converting to *) is configurable.
  • the UC data management system also can allow an enterprise to exclude specific phone numbers or groups of phone numbers from getting obfuscated.
  • a monitoring service is described that can help an enterprise understand how UC infrastructure is performing from an end user perspective.
  • the enterprise can gain the benefit of improved communications experience within and outside the enterprise by using a wide range of modalities and capabilities that were not available previously using dedicated legacy systems (such as a PBX).
  • UC platforms allow users to communicate over internal networks (e.g., corporate networks) and external networks (e.g., the Internet). This opens communication capabilities not only to users available at their desks, but also to users who are on the road, and even to users from different organizations. With such solutions, end users are freed from limitations of previous forms of communication, which can result in quicker and more efficient business processes and decision making.
  • internal networks e.g., corporate networks
  • external networks e.g., the Internet
  • the quality of communications in such platforms can be affected by a variety of problems, including software failures, hardware failures, configuration problems (e.g., system-wide or within components (e.g., firewalls, load balancers)), and network performance problems.
  • the potential impacts of these and other problems include immediate impact upon end users (both internal and roaming) and inefficient use of functionality that increases overall costs.
  • variable costs there are some fixed costs associated with resolving an issue, there are some variable costs that can be reduced to help address the overall impact.
  • One example of a variable cost is the time it takes for an issue to be reported and the time it takes to diagnose the problem. For example, a user may not report an issue immediately for a variety of reasons (e.g., the user may not realize that the issue is something that should be reported, may not be able to report the issue immediately, or may not know who to report to).
  • variable cost Another example of a variable cost is the time it takes to diagnose and resolve the problem after an issue has been reported. In some cases, such as hardware failure, it is simple to identify the root cause. In other cases, it can be difficult to diagnose the root cause of an issue, for a variety of reasons. For example, the individual carrying out the diagnosis may only have information that they receive from an end user, and such information may not be accurate, reliable, or actionable.
  • variable cost Another example of a variable cost is the time it takes to verify that an issue has been resolved. Issues may only exhibit themselves to individuals who are in a specific environment (e.g., connecting via the Internet), and it may not be possible for the individual that is attempting to resolve the issue to immediately verify whether a particular action has successfully resolved the issue.
  • variable costs can result in significant cost savings, and improving upon the processes (e.g., problem diagnosis) that can lead to increases in variable costs also can improve overall quality and user satisfaction. Accordingly, a dynamic monitoring service can add significant value to an organization.
  • a monitoring service as described herein can include one or more of the features described with reference to Examples 11-15 below. More generally, a UC system with a comprehensive monitoring service can provide the following functionality:
  • Previous monitoring services have suffered from several drawbacks, including the need to deploy tools on a dedicated server, with associated deployment and maintenance costs; ability to detect only issues in the specific location the tools deployed, requiring the tools to be deployed in multiple locations; dependence on components such as Web reverse proxies and firewalls; and the inability of diagnosis and resolution tools to improve over time in a way that can be used by the customer directly.
  • the following scenario illustrates how a monitoring service can be used effectively.
  • Alice a consultant working for ABC Consultants
  • Alice is visiting a potential client.
  • Alice realizes she needs Bob to help answer some questions and close the deal.
  • Alice attempts to start a call with Bob using her laptop.
  • the call fails.
  • Alice is able to connect, but the audio quality prevents any meaningful discussion with Bob.
  • Alice is not able to close the deal in a timely manner.
  • Alice decides to report the issue, but she has to wait until she returns to the office, where she is able to look up the relevant administrator (Charlie) and report the issue.
  • Charlie relevant administrator
  • Charlie asks Alice for as many details as possible. However, Alice did not have logging enabled on her laptop and is not possible to provide logs. Also, Alice is now able to make calls to Bob without any issues, and is not able to reproduce the problem. Charlie spends significant time to attempt to determine the root cause. During this time, Charlie receives calls from other users reporting similar problems. After a significant amount of time, and repeated trial and error, Charlie believes the problem is caused by a firewall configuration issue. Charlie makes the required update to address this configuration change, but has no reliable mechanism to verify that the update will address the issue seen by Alice and others.
  • ABC Consultations decides to implement a monitoring service, as described herein. While Charlie is carrying out his normal tasks for the day, he receives an alert stating that the monitoring service has detected an issue which is causing calls to fail. Quickly reviewing the alert details, Charlie determines that this is a significant issue that requires immediate attention. He immediately returns to his desk where he checks his e-mail which shows he has received an e-mail alert containing the following information:
  • Charlie Using the information that is made available within this e-mail, Charlie is able to diagnose the root cause quickly. After making the required update to the firewall configuration, Charlie is able to utilize the appropriate link within the e-mail to retry the problematic scenario and verify the fix. Once verified, Charlie visits a service portal and enters details of the root cause to help identify solutions for future similar issues, thereby adding to the knowledge base of the enterprise around this specific issue. Charlie is able to tell users that the issue had been identified previously and has been resolved.
  • the monitoring service can be used monitor a variety communications, including one or more of the following:
  • a monitoring service can be deployed externally (outside an organization's network) or internally (on a server inside an organization's network). Although an external service that supports the end user scenarios described above is likely to discover many issues that are impacting internal end users, other cases may not be detected. To address these situations, an enterprise can deploy an internal monitoring service on a server inside the corporate network. This internal server could synch with an external monitoring service, which can reduce setup and maintenance costs, and have one location at which to configure settings and receive alerts and reports.
  • Having a monitoring service on an internal server can have additional advantages. For example, the ability to actually detect if gateways are up and running, even if load balanced, can only be carried out completely with an internal server. In addition, being able to completely inspect configuration information and/or access logs can only be carried out with internal servers.
  • the monitoring service can provide administrators with the ability to not only test specific modalities, but to utilize a mix of these modalities and stress test the environment.
  • Table 11 includes a list of features that can be included in a monitoring service. Depending on implementation, a monitoring service may include more features, fewer features, or features that differ from those that are listed in Table 11.
  • Example monitoring service features Feature Description Web This is a web site that an admin can visit. On this site, the admin can learn Experience about the service, see tutorials, etc., and sign up for the service. When signing up, the admin can create an account (including billing information), and specify details about the end user services required. Service Once an admin has signed up, and periodically throughout the lifetime of Configuration utilizing the service, the admin can specify details about their deployment, which is stored in a secure manner. This specification can occur in several ways, e.g.: Admin utilizes the web experience of the service to view and modify details about their deployment. Admin utilizes a tool that when used in their deployment automatically detects changes, and provides these to the service.
  • the admin can also configure details such as the following: Determine end user services and identities within the enterprise that are to be used (e.g., validate the IM user experience using two identities - service1 and service2, which are deployed on two different pools). Determine frequency of end user experience verification (e.g. every 5 minutes). Determine what thresholds are acceptable to the deployment (e.g., acceptable for a single IM to fail, but not when 5 fail in an hour). Provide prioritized list of who should be alerted, and how, when an issue has been discovered. Inform the service when it is expected that issues should occur. For example, when a server is being patched (maintenance mode), then it is likely that more issues will arise.
  • Alerting Make alerts available using current mechanisms that the admin may Mechanism utilize (such as SCOM, Tivoli and Openview). Provide a web portal view of the service alerts. Allow admin to be notified via other modalities, such as SMS, IM, e- mail, or interactive voice response (IVR) alerts. Confirm receipt of alerts, and if not confirmed, send alerts to a secondary set of recipients.
  • an incident log portal (identified by a unique ID in an alert), which provides more detailed view of the incident, including a summary of the issue, details of the logs, potential causes, and any previous solutions associated with that specific deployment.
  • Provide access to community resources e.g., FAQs, channels for real-time communication between multiple admins). Allow admin to re-try the problematic service and verify a fix.
  • Reporting Report metrics per end user service e.g., Report metrics for specific geographies. Report metrics for a specific SIP URI. Report on effects/costs of outages. Compare deployment-specific information to an anonymized report from other enterprises. Export relevant data reports and integrate into other enterprise-specific reports.
  • Cloud-hosted mechanisms are described for simulating end user real time communications to assess communication service availability or conditions. Resolution mechanisms for specific problems also are described.
  • a distributed real-time communications (RTC) monitoring system comprises a monitoring service, a cloud database (e.g., a SQL database), and transaction executors (agents).
  • the system illustrated in FIG. 12 is highly scalable; there is no limitation on the number of monitoring service instances or the number of agents to be deployed. Agents can be deployed in various geographical locations (e.g. EMEA (Europe/Middle East/Africa), North America, APAC (Asia-Pacific)) so that real-life communication scenarios of a global organization can be appropriately represented. More agents can be deployed in a private cloud (illustrated in FIG. 12 ) of the organization and leverage additional synergies with their-real time communications system.
  • the various components shown in FIG. 12 can communicate securely with one another (e.g., via HTTPS).
  • Each agent executes tasks (known as synthetic transactions) which mimic RTC end user behavior (e.g., conference dial-in). Synthetic transaction results are processed by the monitoring service and stored in the cloud database, and appropriate alerts are raised in case of failures. Alerts can include not only diagnostics related information, but also potential root causes and resolution steps, which are extracted from the knowledge base based on historical results.
  • a scheduling algorithm takes a rule schedule (e.g., rule every 15 minutes), puts it in a queue, and assigns it to an agent (also referred to as a transaction executor or TxExecutor) for execution, while considering associated load balancing and resource utilization patterns.
  • a rule schedule e.g., rule every 15 minutes
  • an agent also referred to as a transaction executor or TxExecutor
  • a scheduling mechanism is configured to:
  • task scheduling and distribution can be broken into three parts (task scheduling, task distribution, and load balancing), which are discussed below in more detail:
  • the monitoring service generates tasks based on a rule definition (task template).
  • Task template Each task defines an end user RTC scenario executed in a specified geographical location. Tasks are generated periodically for each rule, with a defined scheduling interval (e.g., every N minutes). Newly generated tasks are added to a task queue.
  • each agent is deployed in a particular geographical location in the cloud and is responsible for simulating end users in that region.
  • an agent executes a REGISTER operation and sends its configuration to the monitoring service.
  • the configuration includes agent characteristics (e.g., deployment location) and capabilities (e.g., ability to execute certain tasks, maximum number of tasks to run in parallel, etc.).
  • the monitoring service sends a unique agent ID.
  • the agent is then considered to be registered and can start executing tasks.
  • the registered agent regularly polls the monitoring service for new tasks.
  • the monitoring service based on the agent's unique ID, looks up its characteristics and capabilities and sends back an appropriate task to be executed.
  • a rule-load balancing algorithm is responsible for enforcing a “least maximum” of rules to be executed concurrently at the same time slot. Accordingly, in this example, when a new periodically executed rule is added to the system, the rule-load balancing algorithm does the following:
  • the 1-rule time slot (time slot index 3) is therefore indicated as being the “first choice” time slot.
  • Time slots in which 2 rules are executed in parallel are indicated as being “second choice,” “third choice,” and “fourth choice,” respectively.
  • FIG. 13 also illustrates where the new rule is added in the sequence. In this example, the new rule is added at time slot index 3.
  • endpoint MPOP multiple points of presence
  • An MPOP constraint makes sure that only one endpoint of a given account is running at the same time slot.
  • the check of accounts used in the rules at a given time slot could be performed before rule load balancing algorithm starts examining a current time slot set for a least maximum.
  • Benchmarking can be based on statistical availability, and can be based on “peer group” or industry verticals.
  • Benchmarking of availability information, audio quality, etc. can be carried out. Benchmarking can be based on statistical availability (e.g., based on “peer group,” industry verticals, etc.).
  • FIG. 14 illustrates a customer's availability (“Customer X”) with a solid line and illustrates average availability for all customers with a dashed line. Comparison of these two lines can provide valuable information about Customer X's availability. For example, during Day 2 a slight drop in average availability (to about 98%) coincides with a very large drop in availability (to about 50%) for Customer X. On the other hand, Customer X's drop in availability (to about 60%) at the end of Day 4 is shown to be about the same as the average for all customers.
  • voice quality metrics packet loss, jitter, latency, etc.
  • voice-related synthetic transactions e.g., conference dial-in. This data can be used for raising immediate alerts or discovering audio quality degradation patterns while mining historical data.
  • FIG. 15 shows an alert generated by a monitoring service (“PowerMon”) for a low MOS score.
  • the alert provides a date and time and a descriptive name for the event (“APAC-US Inbound: user07->user01”).
  • the descriptive name includes information such as region (“APAC-US”), caller (“user07”), and callee (“user01”).
  • the alert also provides details (including NMOS scores) for the caller and the callee.
  • the alert also includes a table for reporting packet loss, jitter, latency, and degradation for inbound and outbound streams.
  • a monitoring service maintains a global knowledge base with data related to RTC system availability disruption investigations. In this way, future RTC system availability issues can be solved faster because potential root causes and resolution steps are automatically provided.
  • an agent sends results to the monitoring service.
  • the result contains multiple parameters (e.g., execution step, diagnostics code, exception type, SIP code, etc.) describing the failure.
  • the monitoring service uses this set of parameters to classify given failures into buckets. Possible root causes and resolution steps can be entered into system and mapped to the set of parameters (e.g., a particular bucket) after issue investigation. This data immediately becomes available for the classification and investigation of future RTC system availability issues.
  • a user survey service can help an enterprise to obtain information directly from users.
  • the user survey service can be used to obtain information from users about the performance of UC services.
  • a user survey service as described herein can include one or more of the features described with reference to Example 16 below.
  • a UC system with a comprehensive user survey service can provide at least the following functionality.
  • a user survey service provides an increased number and higher quality of responses to end user surveys by utilizing real-time communication information. After determining if a user is available (e.g., online and involved in a meeting or other activity) based on presence information, the user survey service initiates a survey at that time via a real-time communication mechanism (e.g., instant messaging).
  • a real-time communication mechanism e.g., instant messaging
  • the real-time communication is typically more immediate than other communications (e.g., e-mail) and generally increases the chance that the user will respond to the survey.
  • FIG. 16 is a diagram of a UC system that includes a user survey service.
  • the UC system includes a survey service server 1606 that communicates with devices associated with a customer site 1602 (e.g., edge server 1608 and user devices associated with a set of end users 1604 ).
  • devices associated with a customer site 1602 e.g., edge server 1608 and user devices associated with a set of end users 1604 .
  • the users/devices that are shown as being within the box in FIG. 16 that represents the customer site 1602 need not be physically present at the customer site 1602 , but may be connected to a network associated with the customer site 1602 .
  • the edge server 1608 (e.g., a Lync edge server) publishes presence information to the survey service server 1606 , which initiates a survey (e.g., via instant messages (IMs)) via the edge server 1608 based on user availability as indicated by the presence information.
  • IMs instant messages
  • the edge server 1608 is only one example of a possible deployment choice in a UC system; other servers with different functionality and configurations also may be used.
  • the survey service server can obtain presence information in different ways. For example, a user may be signed in to an application that is trusted by the UC system, or the user's presence can be detected and authenticated when the user signs in to a user account, even if the user is not currently using a trusted application.
  • users that are determined to be available are contacted by IM and asked to complete a survey.
  • Users that are determined to be unavailable e.g., offline, online but busy, etc.
  • Users that are not available can be excluded from the survey, or their participation can be requested in some other way (e.g., via e-mail).
  • FIG. 17 is a flow chart that illustrates a survey technique that employs presence information.
  • the technique shown in FIG. 17 can be implemented by the system shown in FIG. 16 , or by some other system.
  • a survey service starts up.
  • the service retrieves information (e.g., how to connect to a customer site, and which users associated with the customer site are configured to take surveys) from a database.
  • the service establishes a connection with the remote UC system (e.g., a Lync system) at step 1706 , and subscribes to presence information of targeted users in step 1708 .
  • the service receives the presence information in step 1710 and begins processing the survey in step 1712 .
  • the remote UC system e.g., a Lync system
  • step 1714 the service continues processing the survey until all survey processing is completed (see step 1724 ).
  • step 1716 the service processes an assigned user.
  • step 1718 the service determines if the user is available, and returns to step 1714 if not. If the user is available, the service determines if the user has been available for a configurable number of minutes in step 1720 , as a “cooling off” period to avoid disturbing a user that has just finished a call, for example.
  • the service can wait for the configurable interval to elapse in step 1728 , if needed, and continue processing for other users, as appropriate.) If the user has been available for the required period of time, the service determines, in step 1722 , if a threshold number of attempts to contact the user has been reached. If the threshold has been reached, the user is marked as “processed, no answer” in step 1726 . If the threshold has not been reached, the service determines, in step 1730 , whether the user has already been contacted within a configurable number of hours, to avoid pestering a user that has been recently contacted.
  • the service can wait for the configurable interval for this user to elapse in step 1728 , if needed, and continue processing for other users, as appropriate.) If the user has not been contacted within the configurable number of hours, the service contacts the user (e.g., via IM) to ask the user to take a survey (or to continue a survey if the user has previously started a survey) in step 1732 .
  • the service contacts the user (e.g., via IM) to ask the user to take a survey (or to continue a survey if the user has previously started a survey) in step 1732 .
  • the user can reply positively (e.g., “yes” or some other positive response, such as “start survey”) or negatively (e.g., “no” or some other negative response; a failure to reply within a given amount of time may be interpreted as a negative response). If the user's reply is negative, the service can wait for a configurable interval in step 1728 before attempting to contact the user again. Alternatively, the user may be given more options, such as “contact me later” or “do not contact me again.” If the user chooses not to be contacted again, the service can skip step 1728 and process another user, as appropriate.
  • the service starts or continues the survey, as appropriate, in step 1734 , and the user provides survey responses in step 1736 .
  • the service determines whether the survey has been completed. If so, the user is marked as “done” in step 1740 , and processing of the survey continues with additional users, as appropriate. If the survey has not been completed, the service marks the user as “contact again later” in step 1742 . (The service can wait for the configurable interval for this user to elapse in step 1728 , if needed, and continue processing for other users in the meantime, as appropriate.)
  • a survey service can omit the “cooling off” period after a user becomes available.
  • the service can omit the option of not contacting the user again after a threshold number of attempts have occurred, and instead continue to contact the user until the survey is completed.
  • client devices and administrator devices may be any suitable computing devices, including, but not limited to, laptop computers, desktop computers, smart phones, tablet computers, and/or the like.
  • Servers may include suitable computing devices configured to provide services described in further detail below.
  • the term “server” refers generally to a computing device that provides information (e.g., video and audio data) and/or services to other devices over a communication link (e.g., a network connection), and is not limited to any particular device configuration.
  • Servers may include one or more suitable devices, such as dedicated server computing devices, or virtualized computing instances or application objects executing on a computing device.
  • client can be used to refer to a computing device (e.g., a client device, an administrator device) that obtains information and/or accesses services provided by a server over a communication link, and is not limited to any particular device configuration.
  • a computing device e.g., a client device, an administrator device
  • the designation of a particular device as a client device does not necessarily imply or require the presence of a server.
  • a single device may act as a server, a client, a server and a client, or neither, depending on context and configuration.
  • Actual physical locations of clients and servers are not necessarily important, but the locations can be described as “local” for a client and “remote” for a server to illustrate a common usage scenario in which a client is receiving information provided by a server at a remote location.
  • FIG. 18 is a block diagram that illustrates aspects of an exemplary computing device 1800 appropriate for use in accordance with embodiments of the present disclosure.
  • the description below is applicable to servers, personal computers, mobile phones, smart phones, tablet computers, embedded computing devices, and other currently available or yet-to-be-developed devices that may be used in accordance with embodiments of the present disclosure.
  • the computing device 1800 includes at least one processor 1802 and a system memory 1804 connected by a communication bus 1806 .
  • the system memory 1804 may be volatile or nonvolatile memory, such as read only memory (“ROM”), random access memory (“RAM”), EEPROM, flash memory, or other memory technology.
  • ROM read only memory
  • RAM random access memory
  • EEPROM electrically erasable programmable read-only memory
  • flash memory or other memory technology.
  • system memory 1804 typically stores data and/or program modules that are immediately accessible to and/or currently being operated on by the processor 1802 .
  • the processor 1802 may serve as a computational center of the computing device 1800 by supporting the execution of instructions.
  • the computing device 1800 may include a network interface 1810 comprising one or more components for communicating with other devices over a network.
  • Embodiments of the present disclosure may access basic services that utilize the network interface 1810 to perform communications using common network protocols.
  • the network interface 1810 may also include a wireless network interface configured to communicate via one or more wireless communication protocols, such as WiFi, 2G, 3G, 4G, LTE, WiMAX, Bluetooth, and/or the like.
  • the computing device 1800 also includes a storage medium 1808 .
  • services may be accessed using a computing device that does not include means for persisting data to a local storage medium. Therefore, the storage medium 1808 depicted in FIG. 18 is optional.
  • the storage medium 1808 may be volatile or nonvolatile, removable or nonremovable, implemented using any technology capable of storing information such as, but not limited to, a hard drive, solid state drive, CD-ROM, DVD, or other disk storage, magnetic tape, magnetic disk storage, and/or the like.
  • computer-readable medium includes volatile and non-volatile and removable and non-removable media implemented in any method or technology capable of storing information, such as computer readable instructions, data structures, program modules, or other data.
  • system memory 1804 and storage medium 1808 depicted in FIG. 18 are examples of computer-readable media.
  • FIG. 18 does not show some of the typical components of many computing devices.
  • the computing device 1800 may include input devices, such as a keyboard, keypad, mouse, trackball, microphone, video camera, touchpad, touchscreen, stylus, and/or the like.
  • Such input devices may be coupled to the computing device 1800 by wired or wireless connections including RF, infrared, serial, parallel, Bluetooth, USB, or other suitable connections protocols using wireless or physical connections.
  • data can be captured by input devices and transmitted or stored for future processing.
  • the processing may include encoding data streams, which can be subsequently decoded for presentation by output devices.
  • Media data can be captured by multimedia input devices and stored by saving media data streams as files on a computer-readable storage medium (e.g., in memory or persistent storage on a client device, server, administrator device, or some other device).
  • Multimedia input devices may include a video camera. A video camera, when active, may provide a stream of video data.
  • multimedia input devices may include a microphone. A microphone, when active, may provide a stream of audio data.
  • Input devices can be separate from and communicatively coupled to computing device 1800 (e.g., a client device), or can be integral components of the computing device 1800 . In some embodiments, multiple input devices may be combined into a single, multifunction input device (e.g., a video camera with an integrated microphone). Any suitable input device either currently known or developed in the future may be used with described systems described herein.
  • the computing device 1800 may also include output devices such as a display, speakers, printer, etc.
  • the output devices may include video output devices such as a display or touchscreen.
  • the output devices also may include audio output devices such as external speakers or earphones.
  • the output devices can be separate from and communicatively coupled to the computing device 1800 , or can be integral components of the computing device 1800 . In some embodiments, multiple output devices may be combined into a single device (e.g., a display with built-in speakers). Any suitable output device either currently known or developed in the future may be used with described systems.
  • digital signal processors (which can be implemented in hardware, software, or some combination of hardware and software) can be used for processing media data such as audio data and video data.
  • a digital signal processing module can include encoders to encode and/or decoders to decode encoded data in formats such as MP3, Vorbis, AAC, HE-AAC, or Windows Media Audio (WMA) for audio, or MPEG-2/H.262, H.263, VC-1, or H.264 for video.
  • functionality of computing devices described herein may be implemented in computing logic embodied in hardware or software instructions, which can be written in a programming language, such as C, C++, COBOL, JAVATM, PHP, Perl, HTML, CSS, JavaScript, VBScript, ASPX, Microsoft .NETTM languages such as C#, and/or the like.
  • Computing logic may be compiled into executable programs or written in interpreted programming languages.
  • functionality described herein can be implemented as logic modules that can be duplicated to provide greater processing capability, merged with other modules, or divided into sub-modules.
  • the computing logic can be stored in any type of computer-readable medium (e.g., a non-transitory medium such as a storage medium) or computer storage device and be stored on and executed by one or more general-purpose or special-purpose processors, thus creating a special-purpose computing device configured to provide functionality described herein.
  • a computer-readable medium e.g., a non-transitory medium such as a storage medium
  • computer storage device e.g., a non-transitory medium such as a storage medium
  • general-purpose or special-purpose processors e.g., a general-purpose or special-purpose processors
  • the described systems can comprise multiple client devices and administrator devices, which can interact with the system one at a time or simultaneously.
  • processing stages in techniques described herein can be separated into additional stages or combined into fewer stages.
  • processing stages in techniques described herein can be omitted or supplemented with other techniques or processing stages.
  • processing stages illustrated as occurring in a particular order can instead occur in a different order.
  • processing stages that are described as being performed in a series of steps may instead be handled in a parallel fashion, with multiple modules or software processes concurrently handling one or more of the illustrated processing stages.
  • processing stages that are indicated as being performed by a particular device or module may instead be performed by one or more other devices or modules.
  • the present disclosure includes descriptions of various aspects of unified communication (UC) systems, including UC management and analysis systems and related tools and techniques. Described systems, tools, and techniques are adapted for enhanced UC data capture, analysis, and reporting; enhanced UC monitoring services; and a user survey service that can be used for conducting user surveys related to UC services.
  • UC unified communication

Abstract

The present disclosure includes descriptions of various aspects of unified communication (UC) systems, including UC management and analysis systems and related tools and techniques. Described systems, tools, and techniques are adapted for enhanced UC data capture, analysis, and reporting; enhanced UC monitoring services; and a user survey service that can be used for conducting user surveys related to UC services. Embodiments disclosed herein include computer systems and methods for using presence information to survey users. A described UC system includes a survey service server that communicates with devices associated with a customer site. In a described embodiment, after determining if a user is available (e.g., online and involved in a meeting or other activity) based on presence information, a user survey service initiates a survey at that time via a real-time communication mechanism (e.g., instant messaging).

Description

    CROSS-REFERENCE TO RELATED APPLICATION
  • This application claims the benefit of U.S. Provisional Patent Application No. 61/763,919, filed Feb. 12, 2013, the disclosure of which is hereby incorporated by reference in its entirety.
  • BACKGROUND
  • In general, unified communication (UC) systems provide UC services. UC services include communication services (e.g., e-mail services, instant messaging services, voice communication services, video conference services, and the like) and UC data management and analysis services.
  • UC platforms allow users to communicate over internal networks (e.g., corporate networks) and external networks (e.g., the Internet). This opens communication capabilities not only to users available at their desks, but also to users who are on the road, and even to users from different organizations. With such solutions, end users are freed from limitations of previous forms of communication, which can result in quicker and more efficient business processes and decision making.
  • However, the quality of communications in such platforms can be affected by a variety of problems, including software failures, hardware failures, configuration problems (e.g., system-wide or within components (e.g., firewalls, load balancers)), and network performance problems. The potential impacts of these and other problems include immediate impact upon end users (both internal and roaming) and inefficient use of functionality that increases overall costs.
  • Further, given the unprecedented level of consolidation/centralization that UC platforms may enable, a 100,000 user enterprise may accumulate on the order of 1 billion call records and 1 terabyte of data per year. Formally maintaining this data as an accurate and persistent long-term repository for reference and analysis can help an enterprise to meet its technical, business, and compliance needs.
  • SUMMARY
  • This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This summary is not intended to identify key features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.
  • Embodiments disclosed herein include computer systems and methods for using presence information to survey users. A described UC system includes a survey service server that communicates with devices associated with a customer site. In a described embodiment, after determining if a user is available (e.g., online and involved in a meeting or other activity) based on presence information, a user survey service initiates a survey at that time via a real-time communication mechanism (e.g., instant messaging).
  • DESCRIPTION OF THE DRAWINGS
  • The foregoing aspects and many of the attendant advantages of this invention will become more readily appreciated as the same become better understood by reference to the following detailed description, when taken in conjunction with the accompanying drawings, wherein:
  • FIG. 1 is a block diagram that illustrates a generalized UC management and analysis system according to various aspects of the present disclosure;
  • FIG. 2 is a block diagram that illustrates another example of a unified communication management and analysis system;
  • FIG. 3 is a line graph that shows a percentage of poor calls for a relevant trailing period;
  • FIG. 4 shows an example of a detailed report titled “Poor Calls Network Breakdown”;
  • FIG. 5 shows an example of a detailed report titled “Poor Calls Geography Breakdown”;
  • FIG. 6 shows a table with information relating to a call;
  • FIG. 7 shows a user interface for accessing responses to a survey relating to voice quality;
  • FIG. 8 is a flow chart that shows how targeted notifications and user feedback can be used to improve overall voice quality;
  • FIG. 9 is a screenshot of an e-mail generated by a message generation component;
  • FIG. 10 is a screenshot of an instant message (IM) generated by a message generation component;
  • FIG. 11 is a screen shot of a main landing page titled “Communication Health Report” for a user;
  • FIG. 12 illustrates a distributed real-time communications (RTC) monitoring system comprising a monitoring service, a cloud database, and transaction executors (agents);
  • FIG. 13 illustrates a version of a rule-load balancing algorithm;
  • FIG. 14 is a graph that illustrates a customer's availability and an average availability for all customers;
  • FIG. 15 shows an alert generated by a monitoring service;
  • FIG. 16 is a diagram of a UC system that includes a user survey service;
  • FIG. 17 is a flow chart that illustrates a survey technique that employs presence information; and
  • FIG. 18 is a block diagram that illustrates aspects of an exemplary computing device appropriate for use in accordance with embodiments of the present disclosure.
  • DETAILED DESCRIPTION
  • The present disclosure includes descriptions of various aspects of unified communication (UC) systems, such as UC management and analysis systems, tools, and techniques. In general, UC systems (such as UC systems based on the Lync platform available from Microsoft Corporation) provide UC services. As described herein with respect to various embodiments of the present disclosure, UC services include communication services (e.g., e-mail services, instant messaging services, voice communication services, video conference services, and the like) and UC data management and analysis services, or other services. Representative UC management and analysis services are described in detail below.
  • FIG. 1 is a block diagram that illustrates a generalized UC management and analysis system 100 according to various aspects of the present disclosure. In this generalized example, the system 100 includes client computing devices 102A-N, a server 106, and an administrator computing device 108. The components of the system 100 may communicate with each other via a network 90. For example, the network 90 may comprise a wide-area network such as the Internet. The network 90 may comprise one or more sub-networks (not shown). For example, the network 90 may include one or more local area networks (e.g., wired or wireless local area networks) that may, in turn, provide access to a wide-area network such as the Internet. The client devices 102A-N may be computing devices operated by end users of a UC system. A user operating the administrator device 108 may connect to the server 106 to, for example, manage and analyze use of the UC system.
  • FIG. 2 is a block diagram that illustrates another example of a unified communication management and analysis system. As shown in FIG. 2, the system 200 comprises a client computing device 202, a server 206, and an administrator computing device 208. In the example shown in FIG. 2, the server 206 comprises a data store 220 and a UC management and analysis engine 222. The data store 220 stores data that relates to operation and use of UC system, as will be further described below. The management and analysis engine 222 interacts with the data store 220.
  • In the example shown in FIG. 2, the data store 220 can store data and definitions that define elements to be displayed to an end user on a client device 202 or administrator device 208. For example, the data store 220 can store data that describes the frequency, quality, and other characteristics of communications (e.g., voice communications) that occur across an enterprise via a UC system. As another example, a definition defining a set of interface elements can be used to present a graphical user interface at administrator device 208 that can be used by a system administrator that is seeking to diagnose the cause of a reported problem in the UC system, as explained in detail below. As another example, a definition defining a set of interface elements can be used to present a graphical user interface at client device 202 to guide an end user to respond to a survey relating to the end user's experience with the UC system, as explained in detail below. Interface elements, such as text boxes, soft buttons, checkboxes, drop-down boxes, multimedia interface elements (e.g., audio or video players), and/or the like, may receive input from or present output (e.g., to an end user or system administrator).
  • In the example shown in FIG. 2, the client device 202 includes output device(s) 210, input device(s) 212, and a UC client engine 214. The UC client engine 214 is configured to process input and generate output related to UC services and content (e.g., services and content provided by the server 206). The UC client engine 214 also is configured to cause output device(s) 210 to provide output and to process input from input device(s) 212 related to UC services. For example, input device(s) 212 can be used to provide input (e.g., text input, video input, audio input, or other input) that can be used to participate in UC services (e.g., instant messages (IMs), voice calls), and output device(s) 210 (e.g., speakers, a display) can be used to provide output (e.g., graphics, text, video, audio) corresponding to UC services.
  • In the example shown in FIG. 2, the administrator device 208 includes output device(s) 230, input device(s) 232, and UC administrator engine 234. The UC administrator engine 234 is configured to receive, send, and process information relating to UC services. The UC administrator engine 234 is configured to cause output device(s) 230 to provide output and to process input from input device(s) 232 related to UC services. For example, input device(s) 232 can be used to provide input for administering or participating in UC services, and output device(s) 230 can be used to provide output corresponding to UC services.
  • The UC client engine 214 and/or the UC administrator engine 234 can be implemented as a custom desktop application or mobile application, such as an application that is specially configured for using or administering UC services. Alternatively, the UC client engine 214 and/or the UC administrator engine 234 can be implemented in whole or in part by an appropriately configured browser, such as the Internet Explorer® browser by Microsoft Corporation, the Firefox® browser by the Mozilla Foundation, and/or the like. Configuration of a browser may include browser plug-ins or other modules that facilitate instant messaging, recording and viewing video, or other functionality that relates to UC services.
  • In any of the described examples, an “engine” may include computer program code configured to cause one or more computing device(s) to perform actions described herein as being associated with the engine. For example, a computing device can be specifically programmed to perform the actions by having installed therein a tangible computer-readable medium having computer-executable instructions stored thereon that, when executed by one or more processors of the computing device, cause the computing device to perform the actions. An exemplary computing device is described further below with reference to FIG. 18. The particular engines described herein are included for ease of discussion, but many alternatives are possible. For example, actions described herein as associated with two or more engines on multiple devices may be performed by a single engine. As another example, actions described herein as associated with a single engine may be performed by two or more engines on the same device or on multiple devices.
  • In any of the described examples, a “data store” contains data as described herein and may be hosted, for example, by a database management system (DBMS) to allow a high level of data throughput between the data store and other components of a described system. The DBMS may also allow the data store to be reliably backed up and to maintain a high level of availability. For example, a data store may be accessed by other system components via a network, such as a private network in the vicinity of the system, a secured transmission channel over the public Internet, a combination of private and public networks, and the like. Instead of or in addition to a DBMS, a data store may include structured data stored as files in a traditional file system. Data stores may reside on computing devices that are part of or separate from components of systems described herein. Separate data stores may be combined into a single data store, or a single data store may be split into two or more separate data stores.
  • A. Techniques and Tools for Enhanced UC Data Capture, Analysis, and Reporting
  • Examples in this section describe features of an end-to-end solution for enterprise-level unified communication (UC) data capture, analysis, and reporting. As with other examples described herein, the examples in this section can be used with enterprise-level UC systems.
  • Overview of UC System with Enhanced Data Capture, Analysis, and Reporting
  • A UC system with enhanced data capture, analysis, and reporting capabilities as described herein can include one or more of the features described with reference to Examples 1-10 below. More generally, a comprehensive UC system with enhanced data capture, analysis, and reporting capabilities can provide the following functionality:
  • When used together with an enterprise-level UC platform, a UC system with enhanced data capture, analysis, and reporting capabilities can facilitate cost savings through consolidation, such as by (1) consolidating/replacing hundreds or thousands of disparate PBXs into one centralized global infrastructure; (2) consolidating multiple communications infrastructure components such as audio conferencing, instant messaging, application sharing, video conferencing, etc., into a single infrastructure; and (3) consolidating both internal and remote/external communications by employees, customers, partners, and suppliers into a single infrastructure. Productivity gains can be realized through an increase in collaboration and the speed of business, via an innovative and intuitive end-user experience.
  • Given the unprecedented level of consolidation/centralization that UC platforms may enable, a 100,000 user enterprise may accumulate on the order of 1 billion call records and 1 terabyte of data per year. Formally maintaining this data as an accurate and persistent long-term repository for reference and analysis can help an enterprise to meet its technical, business, and compliance needs.
  • A comprehensive UC system with enhanced data capture, analysis, and reporting capabilities can include:
      • A highly scalable data warehouse built on SQL Server Analysis Services/OLAP “cubes” to permanently store massive volumes of accurate CDR and quality of experience (QoE) data. The warehouse is scalable up to a million active users or more, for up to ten years or more. This provides a robust and highly scalable business intelligence foundation for the enterprise for its business, technical, and compliance needs.
      • A collaboration portal with features to provide access to all of the above reports, with the ability to support detailed queries that may, for example, enable interactive drill-down to analyze UC system performance in areas such as global voice quality. In some embodiments, the collaboration portal is a SharePoint service with a PerformancePoint component, available from Microsoft Corporation.
      • Defining, measuring, tracking, and trending KPIs, and aggregating such KPIs into scorecards best suited for the enterprise. Examples of KPIs include active user trends, call counts, and average mean opinion score (MOS) relating to audio quality. A scorecard can be assembled comprising relevant KPIs for the organization. User adoption statistics can be reviewed, and data can be tracked against success metrics, allowing for “course-correction” in a platform deployment, if needed. The return on investment (ROI) of a UC system deployment tends to be directly correlated with the extent of user adoption—the more users use the system, the more the cost savings and the incremental productivity generated by the enterprise. Tracking and precisely measuring actual user adoption is therefore key to estimating cost savings and productivity gains; key relevant statistics include the number of active users, call counts, conference minutes, on-net minutes, etc. Other statistics such as remote usage, communications with federated partners, modality-specific statistics (e.g. video and application sharing), and response groups also can help measure performance. User adoption statistics also play a key role in identifying “problem areas” in an actionable manner. Poor adoption—either across the enterprise or across a specific site or functional unit—could point to a variety of remediation or proactive steps such as training, device selection, voice quality, or other site- or region-specific considerations. Similarly, IT needs to know if adoption is much better than expected, which would often point to success of the deployment.
      • Functionality for enterprise-grade billing and CDR for unified communications. Traditional time-division multiplexing (TDM) telephony deployments have tended to have stringent needs around billing. However, significant reduction in domestic and international long-distance charges over the last twenty years, as well as the service consolidation and ability to shift phone calls to “on-net” calls provided by UC platforms, have dramatically changed enterprise approaches to billing needs. Instead of carrier-grade billing for call accounting, enterprises now tend to do a rough reconciliation (e.g. to within 5-10%) of their public switched telephone network (PSTN) phone charges, comparing the carrier bills they receive with internally-generated data based on CDRs and carrier rates. Some enterprises need to charge back their toll costs to appropriate internal cost centers and/or to specific clients. Finally, many enterprises need the ability to track individual CDRs for legal compliance purposes, as well as to detect fraud (e.g., unauthorized external use of enterprise resources or PSTN channels), etc. These needs require a robust enterprise-grade infrastructure for capturing authoritative CDR and billing-related data on a per-call basis, for retrieval and analysis at any point in the future.
      • Voice quality metrics and trends. Voice quality is often perceived as “mission-critical” when evaluating communications infrastructure. Enterprises often define, measure, track, and attempt to identify trends in performance in voice quality, and compare them against service level agreements (SLAs). SLAs often apply both at a global level as well as a regional and site level, across the entire enterprise.
      • Reports that help to illustrate trends over time in various areas, such as remote usage/“work from home”; collaboration among customers, suppliers, and partners; business activity over the work week, etc.
      • Powerful and easily accessible class customization capabilities for creating user interface dashboards and reports, significantly reducing the need to have custom reports externally developed to meet enterprise specific business needs.
      • Cost savings through optimization. Cost reduction can be realized, for example, by tracking actual device usage patterns (e.g., headsets vs. desk phones), site-level remote usage utilization for office space consolidation purposes, and consolidating trunks over large collections of sites into centralized SIP trunks, analyzing calling patterns across business units and geographies, to name just a few examples. Each of these examples can provide significant cost savings, especially for large enterprises.
  • The following examples illustrate some of the key features relating to the techniques and tools described herein for enhanced UC data capture, analysis, and reporting:
  • Example 1 Data Warehouse for Analyzing Global Communications Activity
  • An enterprise-wide data warehouse is described that consolidates communications activity in an enterprise into a single data store that provides insights into an enterprise's communication patterns.
  • In this example, the data warehouse includes the following features:
      • ability to store multiple sets of CDR/QoE data from different databases by keeping track of which database server instance the data is imported from;
      • removing duplicates (“de-duping”) of call records based on the session initiation protocol (SIP) dialog ID (in the event that the same calls are logged to different databases);
      • ability to control data import size to minimize load on the source databases and the UC data management system;
      • configurable scheduling of data import processes;
      • data cleansing (e.g., when analyzing conference participants, distinguishing true user participants from services); and
      • calculated charges on person-to-person (P2P) calls and conference usage.
  • In this example, the data warehouse can pull data from the following sources:
      • call details record (CDR) data;
      • quality of experience (QoE) data;
      • enterprise user data (active directory);
      • carrier rates data;
      • custom configuration files to enhance UC platform data (e.g., gateway detail information such as gateway groups and number of channels); and
      • a data model for business intelligence analytics.
    Example 2 KPIs, Metrics, and Financial Reporting Based on Communications Activity
  • Reporting on various business outcomes based on enterprise user communications activity is described. Reports are built on business models and algorithms that map user communication activity and other inputs (location, media) to financial metrics (cost, savings, etc.).
  • In this example, features relating to instant and real-time updates (e.g., via mobile device applications) to enterprise communications KPIs (e.g., a selection of three or some other number of important KPIs) are described. For example, a set of reference base KPIs can be used to measure success of a UC platform. KPIs can indicate overall effectiveness and efficiency of a UC platform deployment, and trends that inform the projected effectiveness and efficiency of the deployment. KPIs can be used to identify “problem spots” in the deployment, track user adoption (which affects cost savings as well as user productivity), and identify opportunities to optimize return on investment in the deployment.
  • In one embodiment, a KPI is used to help determine compliance with SLAs. Further details on SLA compliance are provided in the example below.
  • Example 3 Analyzing Service Level Objectives for Call Quality
  • In this example, a technique is described for classification of calls using location/subnet information, call metrics and algorithms for determining SLA intervals, and time slices based on configurable thresholds. The example technique may include following processing steps:
      • the UC data management system rates audio call quality by evaluating quality metrics against a defined set of acceptable threshold values (e.g., the quality metrics of each call of a set of many calls is evaluated against the acceptable threshold values);
      • the UC data management system discretizes calls into time intervals (e.g., by hour) and aggregates this data by site (e.g., grouping based on call endpoint subnet);
      • an SLA compliance algorithm evaluates the number of calls rated as “poor” within each site/time interval combination against SLA compliance requirements; and
      • SLA compliance is reported by the site and rolled up by time intervals. For example, if a time interval (e.g. from 2 p.m. to 3 p.m.) on a certain day is considered to not be compliant, then this will be rolled up into the relevant longer period, such as a month period. Therefore, if 2 p.m. to 3 p.m. on January 17th is non-compliant, the month of January would be considered non-compliant. In other words, a non-compliant time interval within a time period will result in non-compliance for the entire period.
    Example 4 Controlled Access to Communications Activity Based on User Personas
  • In this example, communications activity and reports are secured centrally and made selectively available to users based on various “personas” (e.g., business function or organizational/administrative functions). Access can be scaled from a group level to an individual level. Permissions settings can be used to define different levels of access. Data access also can be restricted based on personas. For example, a user may be restricted to only viewing data controlled by his department, and not other departments, within an organization.
  • Example 5 Classification of Communication Quality
  • In this example, techniques are described for classifying calls (video, audio, and multimedia) into distinct categories. These categories are then analyzed using heuristics and probabilistic methods to further map/transform data sets into actionable and prioritized recommendations. The prioritization is computed based on algorithms that consider various factors such as user location, user devices, network conditions, etc. User site information can be used in a heuristic for analyzing call patterns based on organization and/or geography. Example reports also are described for tracking overall voice quality with associated metrics within an organization's environment.
  • A quality assessment and classification tool can include the following functionality:
      • A user can identify factors that impact voice quality.
      • Cube mechanisms can be used to allow for identifying via a custom report less common scenarios of voice quality issues.
      • A user can filter the potential information to organizational geography.
      • A user can drill down into specific sets of all calls, filtered “poor calls,” etc., to see the actual individual calls and associated key metrics.
      • A user can see trends of metrics over a period of time, to allow the viewer to identify potential improvements or reduction in voice quality.
      • A user can determine whether existing investments have been valuable and made a return on investment or not.
      • A user can identify that potential additional investments will have a return on the investment by addressing a set of identifiable issues.
  • a. Voice Quality Overview
  • Maintaining acceptable audio quality requires an understanding of UC system infrastructure and proper functioning of the network, communication devices, and other components. An administrator will often need to be able to quantifiably track overall voice quality in order to confirm improvements and identify areas of potential difficulty (or “hot spots”) that require further effort to resolve. There may be a hierarchy of issues, ranging from network issues (typically being both common and important to fix), to issues that are specific to local users, to issues that are specific to remote users, over which an administrator may have little control.
  • b. Voice Quality Reporting Overview
  • One way to track audio quality is through reports. By utilizing reports, an administrator can identify hot spots to address and also convey (e.g., to senior management) information that supports broader conclusions about the system (e.g., that a system deployment is being successful over time, or that more investment is required).
  • Different systems and components may have different ways of classifying whether a call is classified as “poor.” In addition, organizations may have differing requirements for call quality, and may wish to have some control over the standards by which a call will be classified as “poor” or not. SLA reporting may focus on sites as defined by subnet. However, not all customers may define subnets, or have the information to configure sites. Additionally, it is a complex process to keep subnet mapping accurate and up to date. However, there is a different set of information which is available, which could provide a very close approximation to users location, and that is geography information. Therefore, to provide an easier deployment model which allows for quicker SLA reports, it can be useful to allow for customers to utilize this same information.
  • SLA reports also can be used to break down call quality into different aspects which may have impacted the quality of those calls. Examples of factors that could impact audio quality are: (a) the split of wired vs. wireless calls (potentially, audio quality impacts can be due to wireless issues); (b) device characteristics (devices can impact audio quality as perceived by the end user, especially unsupported devices or those without the correct drivers); (c) the effects of gateways between devices; (d) remote users vs. users local to known sites (e.g., if most of the audio quality issues are driven by remote users, this information can be very useful). Identifying situations that may apply with respect to factor (b), above, may require not utilizing network QoE metrics, but other metrics such as Sending MOS (quality of audio stream being sent from user).
  • c. Example Information for Enhanced Voice Quality Analysis and Reporting
  • This section describes examples of information that can be used for enhanced voice quality analysis and reporting.
  • Classification of Poor Calls:
  • In order to isolate a grouping of calls with poor voice quality, it is important to have consistent and meaningful classification of calls. For example, wireless calls which have poor voice quality are important to group together to identify common patterns (e.g., whether the calls involve the same user) and to take appropriate action (e.g., educate the user to not use wireless, upgrade the wireless infrastructure).
  • Additionally, some problems may have more impact on voice quality than others, even within the same call. For example, a user who is using a wireless connection and is roaming outside the user's usual network may be calling another user who is on the corporate network using a wired connection. In this case, the overall experience may be impacted by the first user's wireless connection. An analysis of the conditions at the two endpoints can be conducted to determine which endpoint is more likely to impact a call and highlight one or more items to consider addressing (e.g., by encouraging a user to switch from a wireless connection to a wired connection for the next call).
  • Table 1 below includes examples of expected classifications of calls within the UC system. In this example, a call with two endpoints is classified based on the endpoint with the lowest quality classification. For example, if a first endpoint uses a wireless connection and a second endpoint has similar conditions except that the second endpoint uses a wired, corporate connection, the call will be classified based on the first endpoint. The following table is ordered with worst case being listed first:
  • TABLE 1
    Classification of poor calls.
    Classification Scenario Types of issues
    User VPN Incorrectly utilizing VPN to access the network and,
    (P-U) by consequence, using audio over TCP. May be wired
    or wireless; until VPN is addressed it is hard to assess
    other impacts. Admin needs to determine the number
    of calls and whether this is associated with a certain set
    of users, and allow for user education.
    User Wireless/External: Though potentially there are issues (e.g., with an edge
    (P-U) User is external to server) independent of wireless, admin first needs to
    the main network address users use of wireless. Admin needs to
    and is using determine the number of calls and whether this is
    wireless. associated with a certain set of users, and allow for
    user education.
    User Wireless/Internal: Admin needs to determine the number of calls and
    (P-U) User is on internal whether this is associated with a certain set of users,
    network, but is and allow for user education.
    using wireless.
    External Federated Federation allows users in one enterprise (or
    (P-E) organization) to communicate with users in another
    enterprise (or organization). Users in the “Federated”
    enterprise are called “Federated partners.” Calls
    to/from a specific federated partner may be poor
    because of internal challenges or a specific federated
    partner's network/infrastructure. Being able to identify
    if all federated partners are having issues (e.g., with a
    set of internal users) or a specific partner is important.
    Can break down by user/organization to identify
    potential groups of users with federated users, as well
    break down based on individual federated partners.
    External Edge Potentially, a single edge server or an array of edge
    (P-E) servers has bandwidth problems, is being overloaded,
    or is incorrectly configured. Being able to identify
    which edge servers are having issues, and which sets
    of users maybe having issues (because they are using
    TCP as an example) is important. Can break down
    based on user/organization to identify potential group
    of users with issues and/or based on individual
    groupings of edge servers.
    Corp-Net PC-PC: Depending on network links, the audio may be
    (P-C) User is making a impacted. Identifying which network links are
    call to another impacted is important. By being able to break down
    user, in same or which organization/geographic area is seeing degraded
    different location. audio, the admin is able to see which locations are
    impacted.
    Corp-Net Conference Calls from users to multipoint control units (MCUs)
    (P-C) are impacted, potentially because of the users'
    locations on a network, or because of the data center
    network, the actual MCU being overloaded, or a bad
    configuration. Can break down based on user
    organization/geographic area to identify potential
    locations that are having bad audio quality due to local
    network issues. In addition, the grouping of MCUs are
    also shown, to allow for potentially a single MCU or a
    pool of MCUs to be highlighted as impacting audio
    quality.
    Corp-Net UC<−>GW Similar to conference calls, the audio quality may be
    (P-C) (bypass): impacted by the users' locations on the network, the
    Calls between a GW data center network, the GW being overloaded, or
    user's PC and the a bad configuration. Can break down based on user
    gateway (GW) organization/geographic area to identify potential
    which are locations that are having bad audio quality due to local
    bypassing the network issues. In addition, the grouping of MS's are
    mediation server also shown, to allow for potentially a single MS or
    (MS). pool of MS's to be highlighted as impacting audio
    quality.
    Corp-Net UC<−>MS Similar to conference calls, the audio quality may be
    (P-C) (non-bypass): impacted by the users' locations on the network, the
    Calls between a MS data center network, the MS being overloaded, or
    user's PC and the a bad configuration. Can break down based on user
    mediation server. organization/geographic area to identify potential
    locations that are having bad audio quality due to local
    network. In addition, the grouping of MS's are also
    shown, to allow for potentially a single MS or pool of
    MS's to be highlighted as impacting audio quality.
    Corp-Net MS<−>GW Similar to conference calls, the audio quality may be
    (P-C) (non-bypass): impacted by the MS locations on the network or data
    Calls between an center, or GW locations on the network or data center
    MS and the GW. network, or by either being overloaded, or a bad
    configuration. Can break down based on server site of
    the MS or GW.
  • Grouping of Related Calls:
  • The table above highlights classification of calls with certain general common characteristics, but this could result in a large number of calls across the organization that are not actionable. For example, if a certain amount of low voice quality is expected but there is real hot spot of issues within a certain set of users (e.g., a particular office), those poor calls could be hidden by the wider organization's good calls. Therefore, breaking down the classification to focus on a specific area can be useful.
  • Although the breakdown could be based on any of several factors (e.g., time of day, specific site, etc.), in at least one embodiment the break down is based on geography. This has the advantage of being generally aligned with users' interactivity (e.g., users who are in the Singapore geography are likely using the Singapore network more often) and any future training requirements. If geography information is not available or reliable, the value of breaking down the classification in this way is reduced.
  • In addition to current user geographies, there is a need for several classifications to have additional groupings that do not exist in current user geographies. These are for infrastructure components (MCUs, MS, GWs) which are potentially not in the same locations as users, data centers, etc., as well as for federated partners whose actual geography is not accurately known. Therefore, on top of user geographies, additional elements can be added into the geography hierarchy for voice quality purposes, as follows:
      • Existing Geography Hierarchy
        • Region->Country->Site/Province
          • Office
            • 1. Users
            • 2. <Infrastructure Components>
          • <Infrastructure Site>
            • 1. <Infrastructure Components>
      • Federation
        • <Domain 1>
        • <Domain 2>
  • Additional infrastructure components may exist within the same geographies as users, and can be within the same offices or, potentially, in unique locations (e.g., data centers). It is possible to have additional locations added to the existing geography hierarchy, with the potential to break these down to the calls associated with particular users or infrastructure components.
  • For federation, it is expected that although federated partners may share some of the same locations as an organization's geography, it may not be possible to confirm the location, since additional geographical information is not available. Therefore, federation can be a new element in the top level hierarchy with the ability to break down to each individual partner.
  • Classification of Poor Calls and Thresholds:
  • In order to determine what potential problems exist, it is vital to have a clear definition of what a poor call is, and what is an acceptable amount of poor calls. The definition of a poor call can be provided by a UC platform, by a customer, or in some other way. Some example thresholds for acceptable amounts of poor calls are as follows:
      • P-U calls: 0.5%;
      • P-E calls: 1.5%;
      • P-C calls: 0.1%; and
      • All calls: a sum of the percentages above.
  • These thresholds can be set by default, and can be overridden if desired.
  • Call Counts:
  • Not all classifications/geographies with poor audio quality will require the same level of attention. For example, a geography which is having 1 poor call out of 10, is likely worth investing more time in than a geography with 1 poor call out of 100. Therefore, it is important that wherever information is being displayed, the size of the voice quality problem can be compared. To this end, the following metrics can be shown:
      • Percentage of Poor Audio Calls from Total: the percentage of poor calls of a specific classification compared to the total of all calls in all classifications (e.g., APAC (Asia-Pacific) user calls with a poor wireless classification is 10 out of 1000 total calls in the enterprise—hence 1%);
      • Percentage of Poor Audio Calls: the percentage of poor calls of a specific classification compared to the total of classification calls (e.g., APAC user calls with a poor wireless classification is 10 out of 50 total calls—hence 20%);
      • Percentage of Poor Audio Calls Distribution: the percentage of poor calls of a specific classification compared to the total of all poor calls in all classifications (e.g., APAC user calls with a poor wireless classification is 10 out of 100 total poor calls in the enterprise—hence 10%);
      • Poor Audio Calls: the actual number of poor calls for that classification (e.g., the number of APAC user calls with a poor wireless classification is 10); and
      • Audio Calls: the actual total number of calls for that break down (e.g., the number of APAC user calls is 50).
  • Classification Call Summary:
  • For each grouping of poor calls, it can be useful to see details of the associated poor calls, e.g.:
      • “To” (target of call);
      • “From” (source of call);
      • Time of day & length of call;
      • Source location (geography, organization, site);
      • Target location (geography, organization, site);
      • Type of call (P2P, Conference, PSTN, etc.);
      • Infrastructure used (MCU, GW, etc.); and/or
      • QoE information (e.g., network MOS (NMOS), jitter, latency, packet loss).
        Depending on factors such as the selected grouping of poor calls, other information may be more relevant than the examples provided above.
  • Using information described herein, a customer can:
      • view trends of the overall system and specific call classifications/geographies to identify any improvements or degradation in overall voice quality;
      • where necessary, break down an overall superset of information (e.g., global voice quality) and drill down into specific areas to see if there is a hotspot of problems (e.g., a specific office using wireless connections);
      • find the top X classifications/geographies that likely require the immediate attention without the need to manually drill down;
      • if certain networks are consistently providing low quality audio, focus attention on determining a solution;
      • if a certain range of users is utilizing unsupported devices, determine a path for these users to utilize correct devices;
      • if certain users are experiencing bad quality audio while working remotely, determine if any technical solution is available (such as better networks) or provide better education to end users;
      • if certain gateways are providing bad audio quality, confirm whether those gateways are appropriate or need to be upgraded; and/or
      • use trends to be able to identify that relevant prior actions have been successful or if additional actions are required.
  • d. Example Dashboards
  • In this example, a user (also called a “viewer” in this context) has access to dashboards that provide information.
  • Global Trends Dashboard:
  • The viewer is interested in understanding global or call categorization/geography trends to determine if there has been an improvement in overall voice quality. A global trends dashboard can provide a top level summary of information and trends. This can be at the global level or with the ability to select (via filters and hierarchies) trends for a certain classification/geography pairing. A global trends dashboard can provide the following filters and reports:
      • Filter 1—Trailing Period (e.g., 7 days, 6 weeks, 12 weeks, 6 months, 12 months)—by default, this can be 12 weeks;
      • Filter 2—Geography hierarchy;
      • Filter 3—Call Classification hierarchy;
      • Report 1—Poor Call Summary Table: shows the percentage of Poor Audio Calls from Total, Poor Call Threshold, % Poor Audio Calls Distribution, Poor Audio calls, and Audio calls. An example of this table is shown below in Table 2:
  • TABLE 2
    Poor Call Summary Table (last 12 months)
    % Poor % Poor Poor
    Network Audio Calls Audio Calls Audio Audio
    Type from Total Threshold Distribution Calls Calls
    All 2.05%  2.00% 100%  184 8972
    Corporate 1.2% 0.10% 62.5% 115 7182
    External 0.4% 1.40% 21% 38 1428
    User 0.3% 0.50% 17% 31 362
      • Report 2—Line graph: for the relevant trailing period, shows the percentage of Poor Calls for P-U, P-E, P-C. For example, if looking at training for the last 12 months, then this will show the scores in that period. An example of this line graph is shown in FIG. 3.
  • Top/Specific Problems Dashboard:
  • The viewer is interested in knowing the biggest problem areas that require more investigation across all possible areas. In this case, the viewer does not want to browse all possible areas, but instead to be quickly directed to specific areas to focus on (e.g., when a lot of users are using VPN). Or, the viewer has a specific theory or potential problem that they wish to investigate. For example, a lot of users in a certain geography are complaining about poor voice quality, but no root cause is known. In this case, the viewer wishes to see all information about that specific geography, including all call classifications, and then carry out further investigations to identify what is common to the complaining users.
  • A top/specific problems dashboard can use call classification as a first level of the hierarchy that can be broken down by geography, or can use geography as a first level of the hierarchy that can be broken down by call classification. A top/specific problems dashboard also can use site/subnet mapping, which can then be broken down by call classification.
  • Call Classification Breakdown Dashboard (Top/Specific Problems):
  • Using this dashboard will allow an organization to select a date range that is appropriate and see information associated with the call classifications as the most important grouping. This allows the organization to theorize that, for example, users are using wireless too much, and then find out which geography or geographies of users are using wireless too much. In addition, to save the viewer from having to drill down into all possible combinations of call classification and geography, a report can show the top call classification/geographies that have the worst poor call percentage.
  • A call classification breakdown dashboard can provide the following filters and reports:
      • Filter 1—Trailing Period (e.g., 7 days, 6 weeks, 12 weeks, 6 months, 12 months)—by default, this can be 12 weeks;
      • Filter 2—Geography hierarchy;
      • Report 1—Top Problems (worst percentage of total poor calls) broken down by geography, examples of which are shown in Tables 3 and 4, below:
  • TABLE 3
    Top Problems (worst percentage of total poor calls) by geography
    % Poor Poor Call Poor Total
    Geography Audio Calls Threshold Quality Calls Audio Calls
    Singapore 1.38% 2.00% 184 8972
    Adelaide 0.76% 0.10% 115 7182
    New York 0.39% 1.40% 38 1428
    Barcelona 0.23% 0.50% 31 362
  • TABLE 4
    Top Problems by geography, including
    same-site calls and inter-site calls
    % Poor Poor Call Poor Total
    Geography Audio Calls Threshold Quality Calls Audio Calls
    Singapore 1.38% 2.00% 184 8972
    Adelaide <−> 0.76% 2.00% 115 7182
    New Zealand
    Singapore <−> 0.39% 2.00% 38 1428
    Beijing
    Barcelona 0.23% 2.00% 31 362
      • Report 2—Top Problems (worst percentage of poor calls for all calls within the User call classification) broken down by geography;
      • Report 3—Top Problems (worst percentage of poor calls for all calls within the Corporate call classification) broken down by geography;
      • Report 4—Top Problems (worst percentage of poor calls for all calls within the External call classification) broken down by geography; and
      • Report 5—Specific Problems Table, which has the following columns for the trailing period:
        • Column 1: Expandable call classification hierarchy (e.g. All, External/User/Corporate, User-Internal/Wireless, etc.);
        • Column 2: Expandable geography hierarchy (e.g., Global, Region, Country, Office); and
        • Column N+: Breakdown of both % Poor Audio Calls from total, and % Poor Audio Calls distribution.
          A user can drill down into a specific period (e.g., if a user looks at a specific week and wants to see per day or per hour for that week).
  • Reports can be formatted for viewing in a variety of ways. For example, Reports 1-4 above can be presented side by side, with each report in a table format similar to the example table for Report 1 provided in Table 3, above, or in some other layout, to give a user a convenient view of the reported information (e.g., top problems).
  • Reports can include a significant amount of detail; the detail that is actually presented can depend on factors such as an administrator's preferences. FIG. 4 shows an example of a detailed report titled “Poor Calls Network Breakdown.” The report in FIG. 4 shows network and geography information along with percentages of poor audio calls, by month. The shaded cells or data points in FIG. 4 are highlighted to indicate (e.g., to an administrator) that poor call thresholds have been exceeded.
  • From these reports, it will be possible to select a specific cell in a table and navigate to a Call Summary Report.
  • Geography Breakdown Dashboard (Top/Specific Problems):
  • Using this dashboard will allow an organization to select a date range that is appropriate and see information associated with the geography hierarchy as the most important grouping. This allows the organization to theorize that, for example, a geography of users is having a significant problem and drill down into the call classifications to see if this problem is consistent across all call types or for a specific type of call. In addition, to save the viewer from having to drill down into all possible combinations of geography and call classification, a report can show the top geographies/call classification that have the worst poor call percentage.
  • A Geography Breakdown dashboard can provide the following filters and reports:
      • Filter 1—Trailing Period (e.g., 7 days, 6 weeks, 12 weeks, 6 months, 12 months)—by default, this can be 12 weeks;
      • Filter 2—Call Classification hierarchy;
      • Report 1—Top Problems (worst percentage of total poor calls) broken down by call classification with geography information, an example of which is shown in Table 5, below:
  • TABLE 5
    Top Problems by call classification, with geography information
    % Poor % Poor
    Audio Calls Audio Calls
    from Total Distribution Poor
    Classi- (of Geog- (all Geog- Audio Audio
    fication raphy) Threshold raphies) Calls Calls
    Wireless/ 1.38% 2.00% 15% 184 8972
    Internal -
    Singapore
    Wireless/ 0.76% 0.10% 10% 115 7182
    Internal -
    Munich
    VPN - 0.39% 1.40%  5% 38 1428
    Singapore
      • Report 2—Specific Problems Table, which has the following columns for the trailing period:
        • Column 1: Expandable geography hierarchy (e.g., Global, Region, Country, Office);
        • Column 2: Expandable call classification hierarchy (e.g. All, External/User/Corporate, User-Internal/Wireless, etc.); and
        • Column N+: Breakdown of both % Poor Audio Calls from total, and % Poor Audio Calls distribution.
          A customer can drill down into a specific period (e.g., if a user looks at a specific week and wants to see per day or per hour for that week).
  • Reports can include a significant amount of detail; the detail that is actually presented can depend on factors such as an administrator's preferences. FIG. 5 shows an example of a detailed report titled “Poor Calls Geography Breakdown.” The report in FIG. 5 shows network and geography information along with percentages of poor audio calls, by month. The shaded cells or data points in FIG. 5 are highlighted to indicate (e.g., to an administrator) that poor call thresholds have been exceeded.
  • From these reports, it will be possible to select a specific cell in a table and navigate to a Call Summary Report or a Call Breakdown Report, as explained in further detail below.
  • Call Breakdown Report (Top/Specific Problems):
  • When a number of calls within a call classification/geography pairing is large, it can be difficult for the viewer to scan a list of calls and identify what may be a common problem. For example, there could be an extremely large number of users with occasional PC-to-PC issues, or a concentrated set of users who are having a large number of failures. Therefore, a call breakdown report can be useful for highlighting some of the likely common issues to investigate.
  • A Call Breakdown Report can provide the following filters and reports:
      • Filter 1—Start Date & End Date;
      • Filter 2—Geography Hierarchy;
      • Filter 3—Call Classification Hierarchy;
      • Report 1—Top X users: top X users who have the most poor calls, including the % Poor Audio Calls from Total, % Poor Audio Calls Distribution, and the relevant thresholds and call counts (see Table 6, below):
  • TABLE 6
    Top Users (worst percentage of total poor calls)
    % Poor % Poor Poor
    Audio Calls Audio Calls Audio Audio
    User from Total Distribution Threshold Calls Calls
    User1 1.38% 30% 2.00% 184 8972
    User2 0.76% 20% 0.10% 115 7182
    User3 0.39% 10% 1.40% 38 1428
    User4 0.23% 10% 0.50% 31 362
      • Report 2—Top X infrastructure components: top X infrastructure components that are used within the calls and have the most poor calls, including % Poor Calls, % Total Poor Calls, and the relevant thresholds and call counts (similar to the user table shown in Table 6, above, with Infrastructure components in the first column);
      • Report 3—Counts: see example in Table 7, below:
        • Total number of users that are impacted and having a poor call;
        • % of users that are impacted;
        • Total number of infrastructure components that are impacted and having a poor call;
        • % of infrastructure components impacted;
  • TABLE 7
    Overall Impact report
    Overall # impacted % of total impacted
    Users 100 25%
    Infrastructure
    8 75%
      • Report 4—Voice Quality Metrics: see example in Table 8, below:
        • Average QoE Information across all calls (NMOS, Jitter, Latency, Packet Loss); and
        • QoE Information across Top 10% of calls (NMOS, Jitter, Latency, Packet Loss).
  • TABLE 8
    Voice Quality Metrics report
    NMOS Packet
    Metric Degradation Jitter Loss Latency
    Average 0.5 0.3% 184 8972
    Top 10% 1.6 0.4% 115 7182
  • From each of these reports it will be possible to select a specific cell and navigate to the “Call Summary Report” which will show the calls associated with that metric.
  • Call Summary Report (Top/Specific Problems):
  • This report includes a table which displays a summary of all the poor calls that occurred within certain period.
  • A Call Summary Report can provide the following filters and reports:
      • Filter 1—Trailing Period (e.g., 7 days, 6 weeks, 12 weeks, 6 months, 12 months)—by default, this can be 12 weeks;
      • Filter 2—Organization hierarchy (e.g., Global, Region, Country, Office);
      • Filter 3—Network hierarchy (e.g., All, P-E/P-U/P-C, P-U-Internal/Wireless, etc.);
      • Filter 4—Type of call (e.g., All, MCU, GW, P2P); and
      • Report 1—shows all poor calls in sortable columns, such as: To (Callee), From (Caller), Organization info (Region/Country/Office), Caller/Callee Call Classification Hierarchy (e.g., geography, network), Type of call, Start Date/Time, Duration (mins/secs), NMOS Score, NMOS Degradation, Packet Loss, Jitter, Latency, Mediation Server (if applicable), Gateway (if applicable), MCU (if applicable), Edge Server (if applicable); an example of such a table is shown in FIG. 6.
  • From this report it is possible to select a call (e.g., by selecting a row in the table), and go to a call detail report.
  • Example 6 Analysis of Calls Using Correlations/Patterns
  • In this example, enterprise calls are analyzed based on simultaneous events or conditions within an environment (e.g., user's environment, user's network/site, enterprise environment) and heuristics are utilized to establish correlation or cause-effect information for various call conditions and scenarios. For example, poor quality calls may be correlated with a user adding video and application sharing while on a low bandwidth connection.
  • Example 7 Real-Time User Notification of Call Quality and Reliability Issues
  • In this example, features are described that facilitate proactively notifying users of conditions impacting call quality and reliability via instant messaging or other messaging channels (such as e-mail). Users are notified based on the configurable metrics/parameters (which can be tuned by system administrators) and provided with information mined from call detail and voice quality records. This information is used to provide feedback to the user (e.g., feedback relating to call conditions, as well as other remediation recommendations). A channel for users to provide feedback to operations teams is provided. Operational teams can be alerted to issues relating to specific user groups (e.g., executive users).
  • For example, a real-time user notification service can monitor QoE servers or a UC data manager database and run a query periodically. Based on the result of the query, the service notifies end users. Both the notification message and the channel (e.g., IM, e-mail) can be configured.
  • The screen shot in FIG. 7 shows a user interface for accessing responses to a survey relating to voice quality. The surveys themselves can be conducted using techniques and tools described in detail below.
  • In the example shown in FIG. 7, survey responses associated with specific users are shown in a table. Each row in the table includes a user ID (e.g., e-mail address), user comments, the user's overall voice quality rating (e.g., Very Satisfied, Somewhat Satisfied, Somewhat Dissatisfied, Very Dissatisfied, etc.), and a numeric QoE rating. The numeric QoE rating is calculated based on QoE data for the user during a survey time interval. The user interface allows clicking on the user ID to view additional information, such as per-user metrics (e.g., poor calls, QoE aggregate scores) compared against enterprise and/or industry benchmarks. The user interface also allows selection of results corresponding to different surveys, which can be identified, for example, by the date on which the survey was conducted. The user interface also provides information on how many comments a particular survey has generated. The user interface includes functionality for graphing, commenting on, searching, and exporting information. The user interface can be used for multiple organizations or companies, as shown in the “Select Company” drop-down box. The user interface can include features for securely viewing such information (see the “Log Out” and “Change Password” links). The user interface can be presented in a Web browser or as part of a dedicated, custom application.
  • FIG. 8 is a flow chart that shows how targeted notifications and user feedback can be used to improve overall voice quality. The decision whether to send targeted notifications also can be based on a user's situation or a usage scenario. For example, a rules engine can detect if the user making a call was on Wifi and/or using unsupported devices, which can affect call quality. The notification can be tailored to be appropriate to the situation. This results in a more accurate communication outreach and actionable results. An operations team can maintain a record of users who have been contacted (or “pinged”) to limit the possibility that users will become annoyed or overwhelmed with information, while also allowing the team to determine if a follow-up message might be helpful.
  • Call quality metrics are stored in databases (e.g., QoE and/or CDR databases). In the example shown in FIG. 8, in step 810 a voice quality rule engine monitors a database (e.g., a QoE database) for poor calls. In step 820, the voice quality rule engine triggers an outbound notification to an end user via a predetermined channel (e.g., IM, e-mail), in accordance with rules that apply to the conditions of a communication in which the user participated (e.g., rules related to WiFi communications, rules related to communications using unsupported devices, etc.). In step 830, the triggered outbound notification is handled by a message generation component of a service that uses presence information to determine whether use IM or e-mail for delivering the notification. For example, if a user is detected to be online and available, the component can send an IM. As another example, if the user is detected to be unavailable (e.g., offline, or online but busy), the component can send an e-mail. In step 840, the user receiving the notification is given the option (e.g., via a link in the notification) to provide feedback for analysis.
  • FIG. 9 is a screenshot of an e-mail generated by a message generation component. The “To:” field of the e-mail includes a white indicator graphic next to the user's e-mail address that provides presence information. In this example, the white color of the graphic indicates that the user is offline. The “Bcc:” field includes a green indicator graphic that indicates that “End User Services” is online and available. (The green color is not shown in FIG. 9.) The text of the e-mail confirms that a call in which the user participated had poor voice quality, and suggests possible causes (poor wireless connectivity, unsupported audio device) of the poor voice quality. The e-mail provides underlined links (see items numbered “1.” and “2.” in the text of the email message in FIG. 9) to help the addressee avoid similar problems in the future. The e-mail also requests feedback and provides an underlined link in the “Feedback” section of the e-mail for this purpose.
  • FIG. 10 is a screenshot of an instant message (IM) generated by a message generation component. A green indicator graphic indicates that “Survey Bot” is online and available. (The green color is replaced with dark shading in FIG. 10.) The text of the IM confirms that a call in which the user participated had poor voice quality, and suggests possible causes (poor wireless connectivity, unsupported audio device) of the poor voice quality. The IM also requests feedback and provides a link (underlined in the “Feedback” section of the IM) for this purpose.
  • Example 8 Per-User Metrics, Voice Quality Metrics, and Scorecard with Benchmarking
  • In this example, features are described that provide a per-user “score” for enterprise communications using an algorithm to compute a single score that takes into account the user's communication activity (based on various parameters and metrics), and that allow for benchmarking against a “peer group.”
  • FIG. 11 is a screen shot of a main landing page (“Communication Health Report”) for a user (“User 1”). The main landing page also can include UC system availability, a user's open feedback items (e.g., if the user is connected to the UC system), and other messages about UC, such as training opportunities for offered UC services. The various elements shown in FIG. 11 are only examples. The elements shown in FIG. 11 can be arranged in different ways. Further, individual elements shown in FIG. 11 can be omitted, supplemented with additional elements, and/or replaced with different elements showing different information.
  • FIG. 11 shows a meeting health index score tile 1110. A user's meeting health index score is a composite score that includes the type of network being used when the user is communicating, the audio device being used (e.g., headset, handset, microphone, external speakers) as well as the network and device behaviors of those who participate in meetings or calls with the user. As shown, the maximum score is five stars, and the user's score for the time period (e.g., a week) is three stars. An average score (e.g., for other users within the user's organization) also can be provided to the user, but is omitted from FIG. 11. In one example scenario, the user can compare the user's individual score with the average score to get a better idea of how the user's meeting health index compares to the user's peers.
  • FIG. 11 also shows an audio device usage tile 1120. This tile includes a pie chart in which the fraction of calls that use approved devices and unsupported devices are shown, along with the fraction of calls in which the device being used actually caused audio quality issues.
  • FIG. 11 also shows a network awareness tile 1130, which can be used to rate the user's overall network usage (e.g., “poor,” “fair,” or “excellent”) based on, for example, how often the user is participating in calls over communication channels (e.g., wireless channels) that tend to have lower voice quality.
  • FIG. 11 also shows a weekly metrics tile 1140 that includes counts of “good” and “poor” quality calls of different types. Alternatively, this tile can display different metrics and/or metrics for different time periods (e.g., monthly).
  • Example 9 Cost Saving Reports
  • In this example, features are described that provide a travel cost model for estimating travel cost savings based on an increase in web conferencing/online meetings. Conference travel and lost opportunity cost savings are determined based on a calculation of what the estimated cost would have been for each enterprise participant for on-site conference attendance. The model assumes that the location of the conference is the Organizer's location.
  • In particular, the UC data management system uses user geography information (e.g., region, country, state, and/or city) combined with a configurable travel probability matrix and associated travel and lost opportunity costs to determine cost savings. The probability of the user traveling to the physical meeting location is based on the conference attendee count and the duration of the conference, as shown in Table 9, below:
  • TABLE 9
    probability of user traveling to a physical meeting location.
    Attendee % Probability based on Conference Duration
    Attendee Count <=1 <=2 <=4 >4
    Count Min Max <=30 Min Hour Hours Hours Hours
    1 3 2.0 2.0 5.0 20.0 30.0
    4 5 2.0 3.0 6.0 25.0 40.0
    6 9 3.0 4.0 8.0 30.0 50.0
    10 24 3.0 6.0 12.0 37.0 60.0
    25 999 3.0 9.0 14.0 40.0 65.0
  • An associated hourly travel and opportunity cost can be calculated based on a geographical difference (e.g., inter-region, inter-country) between the physical meeting location (which may be assumed to be the organizer's location) and the participant's location. For example, if a user in the United Kingdom is invited to a meeting in North America, the geographical difference is “inter-region,” whereas if the meeting is in France, the geographical difference is “inter-country.” These classifications can be adjusted, such as when a user is located in an isolated area of a large country, and inter-city travel is more expensive than for a user near a population center of a small country. Example calculations are shown in Table 10, below. The actual costs reflected in Table 10 can be adjusted. For example, costs may be increased over time as average travel costs increase. As another example, the opportunity cost of attending a meeting for a high-level executive may be significantly greater than the opportunity cost for the executive's assistant.
  • TABLE 10
    Hourly travel and opportunity cost.
    Lost Opportunity Travel
    Geographical Cost Per Hour Cost Per Hour
    Difference ($US) ($US)
    Inter-Region 78.00 143.00
    Inter-Country 58.00 96.00
    Inter-City 45.00 50.00
    Inter-Building 30.00 4.00
    Same Building 0.00 0.00
    Dial-In/Enterprise 0.00 0.00
    No Cost 0.00 0.00
    Enterprise/Unknown 45.00 50.00
  • Example 10 Intelligent Data Obfuscation for Protecting Privacy
  • In this example, a method is described for obfuscation and removal of PII (personally identifiable information) on call detail records in a configurable approach that protects privacy information but still allows for data analysis and insights.
  • In at least one embodiment, data obfuscation applies to all calls associated with a gateway (assumed to be PSTN calls), and the piece of data obfuscated is phone numbers. The UC data management system allows an enterprise to determine when to obfuscate data (e.g., when data is imported, or a given number of days after the call occurred). The format of the obfuscation can be, for example, as follows: +14253334444->+1425*******, where numerals represent numbers in a phone number, and * represents an obfuscated digit. The number of digits to obfuscate (e.g., by converting to *) is configurable. In the example above, a few leading digits are retained, allowing an enterprise to be able to report and group calls (e.g., by area code or zone). The UC data management system also can allow an enterprise to exclude specific phone numbers or groups of phone numbers from getting obfuscated.
  • B. Enhanced Monitoring for UC Services
  • In this section, a monitoring service is described that can help an enterprise understand how UC infrastructure is performing from an end user perspective. When described techniques and tools are used with a UC platform, the enterprise can gain the benefit of improved communications experience within and outside the enterprise by using a wide range of modalities and capabilities that were not available previously using dedicated legacy systems (such as a PBX).
  • UC platforms allow users to communicate over internal networks (e.g., corporate networks) and external networks (e.g., the Internet). This opens communication capabilities not only to users available at their desks, but also to users who are on the road, and even to users from different organizations. With such solutions, end users are freed from limitations of previous forms of communication, which can result in quicker and more efficient business processes and decision making.
  • However, the quality of communications in such platforms can be affected by a variety of problems, including software failures, hardware failures, configuration problems (e.g., system-wide or within components (e.g., firewalls, load balancers)), and network performance problems. The potential impacts of these and other problems include immediate impact upon end users (both internal and roaming) and inefficient use of functionality that increases overall costs.
  • Although there are some fixed costs associated with resolving an issue, there are some variable costs that can be reduced to help address the overall impact. One example of a variable cost is the time it takes for an issue to be reported and the time it takes to diagnose the problem. For example, a user may not report an issue immediately for a variety of reasons (e.g., the user may not realize that the issue is something that should be reported, may not be able to report the issue immediately, or may not know who to report to).
  • Another example of a variable cost is the time it takes to diagnose and resolve the problem after an issue has been reported. In some cases, such as hardware failure, it is simple to identify the root cause. In other cases, it can be difficult to diagnose the root cause of an issue, for a variety of reasons. For example, the individual carrying out the diagnosis may only have information that they receive from an end user, and such information may not be accurate, reliable, or actionable.
  • Another example of a variable cost is the time it takes to verify that an issue has been resolved. Issues may only exhibit themselves to individuals who are in a specific environment (e.g., connecting via the Internet), and it may not be possible for the individual that is attempting to resolve the issue to immediately verify whether a particular action has successfully resolved the issue.
  • Reducing variable costs can result in significant cost savings, and improving upon the processes (e.g., problem diagnosis) that can lead to increases in variable costs also can improve overall quality and user satisfaction. Accordingly, a dynamic monitoring service can add significant value to an organization.
  • Detailed Overview of an Example Monitoring Service
  • A monitoring service as described herein can include one or more of the features described with reference to Examples 11-15 below. More generally, a UC system with a comprehensive monitoring service can provide the following functionality:
      • automatic, regular verification of system functionality;
      • alerting individuals designated for resolving identified issues based on factors such as issue type, time of day, etc.;
      • providing detailed information on the issue, including logs, traces and details of the experience an end user would see;
      • providing a mechanism to automatically retry the problematic scenario and verify that the issue is resolved; and
      • continued monitoring of the rest of the UC environment while the fault is being fixed, to reduce the chance of further problems.
  • Previous monitoring services have suffered from several drawbacks, including the need to deploy tools on a dedicated server, with associated deployment and maintenance costs; ability to detect only issues in the specific location the tools deployed, requiring the tools to be deployed in multiple locations; dependence on components such as Web reverse proxies and firewalls; and the inability of diagnosis and resolution tools to improve over time in a way that can be used by the customer directly.
  • The following scenario illustrates how a monitoring service can be used effectively.
  • Alice, a consultant working for ABC Consultants, is visiting a potential client. During negotiations, Alice realizes she needs Bob to help answer some questions and close the deal. Alice attempts to start a call with Bob using her laptop. Unfortunately, the call fails. After repeated attempts, Alice is able to connect, but the audio quality prevents any meaningful discussion with Bob. Alice is not able to close the deal in a timely manner. Later, Alice decides to report the issue, but she has to wait until she returns to the office, where she is able to look up the relevant administrator (Charlie) and report the issue.
  • Charlie asks Alice for as many details as possible. However, Alice did not have logging enabled on her laptop and is not possible to provide logs. Also, Alice is now able to make calls to Bob without any issues, and is not able to reproduce the problem. Charlie spends significant time to attempt to determine the root cause. During this time, Charlie receives calls from other users reporting similar problems. After a significant amount of time, and repeated trial and error, Charlie believes the problem is caused by a firewall configuration issue. Charlie makes the required update to address this configuration change, but has no reliable mechanism to verify that the update will address the issue seen by Alice and others.
  • Later, ABC Consultations decides to implement a monitoring service, as described herein. While Charlie is carrying out his normal tasks for the day, he receives an alert stating that the monitoring service has detected an issue which is causing calls to fail. Quickly reviewing the alert details, Charlie determines that this is a significant issue that requires immediate attention. He immediately returns to his desk where he checks his e-mail which shows he has received an e-mail alert containing the following information:
      • the extent of end user impact across the enterprise;
      • an indication that the fault is limited to audio connections and that IM conversations are working normally;
      • logs that allow him to see both detailed and high-level information and compare this information to “last known good” information such as transaction time;
      • a list of potential causes of the issue based on historical data, including number of times this issue has been discovered; and
      • a link which allows him to utilize the service to replicate the failure that was previously detected.
  • Using the information that is made available within this e-mail, Charlie is able to diagnose the root cause quickly. After making the required update to the firewall configuration, Charlie is able to utilize the appropriate link within the e-mail to retry the problematic scenario and verify the fix. Once verified, Charlie visits a service portal and enters details of the root cause to help identify solutions for future similar issues, thereby adding to the knowledge base of the enterprise around this specific issue. Charlie is able to tell users that the issue had been identified previously and has been resolved.
  • Charlie works with the firewall administrator to ensure that the monitoring service is used to verify that firewall changes have not accidentally caused any new issues. By using this “run now” mechanism, unintended impacts can be identified immediately and not cascade into a lengthy outage for end users.
  • Depending on implementation, the monitoring service can be used monitor a variety communications, including one or more of the following:
      • login to the UC infrastructure;
      • set the users presence, and obtain presence for the users contacts;
      • send and receive internal IMs;
      • start and receive incoming P2P audio calls, with audio of acceptable quality;
      • schedule/invite users to/join a conference, with audio of acceptable quality;
      • send and receive conference data, such as shared whiteboards and application data;
      • manage conference participants;
      • dial and connect to standard telephones at various locations, with acceptable quality;
      • receive calls from standard telephones various locations, with acceptable quality;
      • forward and receive calls by unified messaging service, with acceptable quality;
      • receive voicemail from roaming users, federated users, or standard telephone users;
      • set local presence and receive presence information from an external IM service;
      • send and receive IM via an external IM service;
      • set local presence and receive presence information from a federated contact;
      • send IMs to and receive IMs from a federated contact;
      • join and participate in conference as an anonymous user, with acceptable quality;
      • receive calls (including from RGS numbers) at standard telephones, with acceptable quality (RGS (Response Group Service) is an Automatic Call Distributor (ACD) feature of Microsoft Lync Server, and is similar to a small helpdesk application or reception desk capability, where customers, as an example, call the company number, are prompted for which department they wish to talk to, and are routed to a company employee in that department or receive a basic message if the call is occurring when no one is available (e.g., after business hours);
      • call from standard telephones (including calling unassigned numbers, and receiving associated announcements);
      • check for and download software updates at client devices;
      • download client software for participating in meetings;
      • park and retrieve calls;
      • access Web-based services; and
      • join group chats, send and receive group chat information, participate in group chat across federated boundaries.
  • A monitoring service can be deployed externally (outside an organization's network) or internally (on a server inside an organization's network). Although an external service that supports the end user scenarios described above is likely to discover many issues that are impacting internal end users, other cases may not be detected. To address these situations, an enterprise can deploy an internal monitoring service on a server inside the corporate network. This internal server could synch with an external monitoring service, which can reduce setup and maintenance costs, and have one location at which to configure settings and receive alerts and reports.
  • Having a monitoring service on an internal server can have additional advantages. For example, the ability to actually detect if gateways are up and running, even if load balanced, can only be carried out completely with an internal server. In addition, being able to completely inspect configuration information and/or access logs can only be carried out with internal servers.
  • In addition to handing end user scenarios that apply for roaming users and internal users, examples of validation that can be carried out using an internal server include the following:
      • inspecting deployment configuration information and looking for anomalies (e.g., immediate impacts or minor impacts that grow over time);
      • inspecting CDRs for potential trends and issues;
      • inspecting a QoE database for potential trends and issues;
      • validating the range and number of gateways that are available and working correctly; and
      • validating version information of gateway firmware and providing alerts to an administrator if versions are out of date and need to be patched.
  • Stress Testing:
  • One of the key issues for an organization is determining how many users the system can handle. The monitoring service can provide administrators with the ability to not only test specific modalities, but to utilize a mix of these modalities and stress test the environment.
  • Table 11, below, includes a list of features that can be included in a monitoring service. Depending on implementation, a monitoring service may include more features, fewer features, or features that differ from those that are listed in Table 11.
  • TABLE 11
    Example monitoring service features
    Feature Description
    Web This is a web site that an admin can visit. On this site, the admin can learn
    Experience about the service, see tutorials, etc., and sign up for the service. When
    signing up, the admin can create an account (including billing information),
    and specify details about the end user services required.
    Service Once an admin has signed up, and periodically throughout the lifetime of
    Configuration utilizing the service, the admin can specify details about their deployment,
    which is stored in a secure manner. This specification can occur in several
    ways, e.g.:
    Admin utilizes the web experience of the service to view and modify
    details about their deployment.
    Admin utilizes a tool that when used in their deployment
    automatically detects changes, and provides these to the service.
    In addition to specifying details about the admin's deployment, the admin
    can also configure details such as the following:
    Determine end user services and identities within the enterprise that
    are to be used (e.g., validate the IM user experience using two
    identities - service1 and service2, which are deployed on two
    different pools).
    Determine frequency of end user experience verification (e.g. every 5
    minutes).
    Determine what thresholds are acceptable to the deployment (e.g.,
    acceptable for a single IM to fail, but not when 5 fail in an hour).
    Provide prioritized list of who should be alerted, and how, when an
    issue has been discovered.
    Inform the service when it is expected that issues should occur. For
    example, when a server is being patched (maintenance mode), then it
    is likely that more issues will arise.
    Make configuration changes and carry out test for each of the
    available end user experiences, in order to ensure that the
    configuration specified is accurate, and to allow the admin to verify
    any deployment changes and validate any issue resolutions.
    Alerting Make alerts available using current mechanisms that the admin may
    Mechanism utilize (such as SCOM, Tivoli and Openview).
    Provide a web portal view of the service alerts.
    Allow admin to be notified via other modalities, such as SMS, IM, e-
    mail, or interactive voice response (IVR) alerts.
    Confirm receipt of alerts, and if not confirmed, send alerts to a
    secondary set of recipients.
    Provide an incident log portal (identified by a unique ID in an alert),
    which provides more detailed view of the incident, including a
    summary of the issue, details of the logs, potential causes, and any
    previous solutions associated with that specific deployment.
    Provide access to community resources (e.g., FAQs, channels for
    real-time communication between multiple admins).
    Allow admin to re-try the problematic service and verify a fix.
    Reporting Report metrics per end user service.
    Report metrics for specific geographies.
    Report metrics for a specific SIP URI.
    Report on effects/costs of outages.
    Compare deployment-specific information to an anonymized report
    from other enterprises.
    Export relevant data reports and integrate into other
    enterprise-specific reports.
  • The following examples illustrate some of the key features relating to the techniques and tools described herein for enhanced monitoring of performance of UC services.
  • Example 11 Cloud-Based Monitoring for Communication Service Availability
  • Cloud-hosted mechanisms are described for simulating end user real time communications to assess communication service availability or conditions. Resolution mechanisms for specific problems also are described.
  • In this example (illustrated in FIG. 12) a distributed real-time communications (RTC) monitoring system comprises a monitoring service, a cloud database (e.g., a SQL database), and transaction executors (agents). The system illustrated in FIG. 12 is highly scalable; there is no limitation on the number of monitoring service instances or the number of agents to be deployed. Agents can be deployed in various geographical locations (e.g. EMEA (Europe/Middle East/Africa), North America, APAC (Asia-Pacific)) so that real-life communication scenarios of a global organization can be appropriately represented. More agents can be deployed in a private cloud (illustrated in FIG. 12) of the organization and leverage additional synergies with their-real time communications system. The various components shown in FIG. 12 can communicate securely with one another (e.g., via HTTPS).
  • Each agent executes tasks (known as synthetic transactions) which mimic RTC end user behavior (e.g., conference dial-in). Synthetic transaction results are processed by the monitoring service and stored in the cloud database, and appropriate alerts are raised in case of failures. Alerts can include not only diagnostics related information, but also potential root causes and resolution steps, which are extracted from the knowledge base based on historical results.
  • Example 12 Algorithm for Distributing Tasks Among Geo-Distributed Agents
  • In this example, a scheduling algorithm is described that takes a rule schedule (e.g., rule every 15 minutes), puts it in a queue, and assigns it to an agent (also referred to as a transaction executor or TxExecutor) for execution, while considering associated load balancing and resource utilization patterns.
  • In at least one embodiment, to solve the task distribution problem a scheduling mechanism is configured to:
      • periodically generate tasks at the defined intervals;
      • make tasks available for geo-distributed agents;
      • stay resilient to increased\decreased numbers of agents;
      • balance workload between agents; and
      • ensure parallel-executed RTC scenarios are not affecting each other or final results.
  • Conceptually, task scheduling and distribution can be broken into three parts (task scheduling, task distribution, and load balancing), which are discussed below in more detail:
  • a. Task Scheduling
  • In this example, the monitoring service generates tasks based on a rule definition (task template). Each task defines an end user RTC scenario executed in a specified geographical location. Tasks are generated periodically for each rule, with a defined scheduling interval (e.g., every N minutes). Newly generated tasks are added to a task queue.
  • b. Task Distribution
  • In this example, each agent is deployed in a particular geographical location in the cloud and is responsible for simulating end users in that region. After an agent is started, it executes a REGISTER operation and sends its configuration to the monitoring service. The configuration includes agent characteristics (e.g., deployment location) and capabilities (e.g., ability to execute certain tasks, maximum number of tasks to run in parallel, etc.). In response, the monitoring service sends a unique agent ID. The agent is then considered to be registered and can start executing tasks. The registered agent regularly polls the monitoring service for new tasks. The monitoring service, based on the agent's unique ID, looks up its characteristics and capabilities and sends back an appropriate task to be executed.
  • c. Load Balancing and Optimizations
  • Even if a number of agents can be scaled up and down, it is important to use resources wisely and try to distribute work load more or less equally over time. Time slots with the highest number of rules running in parallel will dictate requirements for computing resources.
  • In this example, a rule-load balancing algorithm is responsible for enforcing a “least maximum” of rules to be executed concurrently at the same time slot. Accordingly, in this example, when a new periodically executed rule is added to the system, the rule-load balancing algorithm does the following:
  • (a) for a rule which is executed every N minutes, select the set of time slots starting at minute 1;
  • (b) within this set, identify the time slot which has the highest number of concurrent rules;
  • (c) if the value of this slot is smaller than the “least maximum” which has been identified so far, then that value becomes the new “least maximum” and the set of time slots becomes the set having the “least maximum”; and
  • (d) repeat steps (a)-(c) above until minute N−1.
  • The new rule is added to the set of time slots, which has the least maximum number of rules executed at the same time. FIG. 13 illustrates a version of the algorithm in which a rule which runs every 8 slots (N=8) is added to time slot where only 1 rule is being executed, compared to other time slots in which 2 or more rules are executed in parallel. The 1-rule time slot (time slot index 3) is therefore indicated as being the “first choice” time slot. Time slots in which 2 rules are executed in parallel are indicated as being “second choice,” “third choice,” and “fourth choice,” respectively. FIG. 13 also illustrates where the new rule is added in the sequence. In this example, the new rule is added at time slot index 3.
  • Additional constraints related to RTC specifics could be added to the algorithm. One of these is endpoint MPOP (multiple points of presence) prevention. Since the same RTC accounts could be used to simulate multiple end user behaviors, it is important that scenarios running at the same time are not interfering. An MPOP constraint makes sure that only one endpoint of a given account is running at the same time slot.
  • The check of accounts used in the rules at a given time slot could be performed before rule load balancing algorithm starts examining a current time slot set for a least maximum.
  • Example 13 Aggregation of “Peer” Data for Communications System Availability
  • In this example, benchmarking of availability information is described. Benchmarking can be based on statistical availability, and can be based on “peer group” or industry verticals.
  • For customers running similar scenarios on the same agents, historical information could be used for comparison and benchmarking of their RTC systems. Benchmarking of availability information, audio quality, etc., can be carried out. Benchmarking can be based on statistical availability (e.g., based on “peer group,” industry verticals, etc.).
  • For example, FIG. 14 illustrates a customer's availability (“Customer X”) with a solid line and illustrates average availability for all customers with a dashed line. Comparison of these two lines can provide valuable information about Customer X's availability. For example, during Day 2 a slight drop in average availability (to about 98%) coincides with a very large drop in availability (to about 50%) for Customer X. On the other hand, Customer X's drop in availability (to about 60%) at the end of Day 4 is shown to be about the same as the average for all customers.
  • Example 14 Voice Quality Metrics Based on Synthetic Transactions
  • In this example, voice quality metrics (packet loss, jitter, latency, etc.) are collected for voice-related synthetic transactions (e.g., conference dial-in). This data can be used for raising immediate alerts or discovering audio quality degradation patterns while mining historical data.
  • For example, FIG. 15 shows an alert generated by a monitoring service (“PowerMon”) for a low MOS score. The alert provides a date and time and a descriptive name for the event (“APAC-US Inbound: user07->user01”). The descriptive name includes information such as region (“APAC-US”), caller (“user07”), and callee (“user01”). The alert also provides details (including NMOS scores) for the caller and the callee. The alert also includes a table for reporting packet loss, jitter, latency, and degradation for inbound and outbound streams.
  • Example 15 Global Knowledge Base
  • In this example, a monitoring service maintains a global knowledge base with data related to RTC system availability disruption investigations. In this way, future RTC system availability issues can be solved faster because potential root causes and resolution steps are automatically provided.
  • In one scenario, after executing a task, an agent sends results to the monitoring service. In case of a task failure, the result contains multiple parameters (e.g., execution step, diagnostics code, exception type, SIP code, etc.) describing the failure. The monitoring service uses this set of parameters to classify given failures into buckets. Possible root causes and resolution steps can be entered into system and mapped to the set of parameters (e.g., a particular bucket) after issue investigation. This data immediately becomes available for the classification and investigation of future RTC system availability issues.
  • C. User Survey Service
  • In this section, a user survey service is described that can help an enterprise to obtain information directly from users. For example, the user survey service can be used to obtain information from users about the performance of UC services.
  • Detailed Overview of an Example User Survey Service
  • A user survey service as described herein can include one or more of the features described with reference to Example 16 below. In a broader UC context, a UC system with a comprehensive user survey service can provide at least the following functionality.
      • By contacting a user proactively through the UC system, the end user does not have to authenticate towards the system. Information is automatically collected within the context of a given user, without the need to have that user enter a password or visit a URL with a specific encoded access key.
      • A UC system can provide the user survey service the end user's presence information, and the user survey service can target users based on their availability. Therefore, users can be contacted specifically at a time when they are available, and not be disturbed while they are busy. This should increase the possibility of the user actually completing the survey, as opposed to surveys conducted by e-mail.
      • By reacting to presence information, the user survey service can reach users when they become available. To some users, an immediate request to complete a survey after becoming available may be seen as intrusive. The user survey service can be configured to wait for the duration of a “cooling off” period (e.g., a few minutes) after the user becomes available.
      • The user survey service can track how often individual users are contacted and asked to take a survey. The service can be configured to reduce or stop survey requests when certain conditions are present or certain thresholds are reached. For example, the service can be configured to request a user's participation a maximum of 10 times. As another example, once a user has taken a survey, the service does not contact the user again regarding that survey.
      • Utilizing instant messaging, an inherently stateful communication channel, it is possible for an end user to quit a survey without finishing it. The user survey service can recognize this and allow the user to automatically pick up where the user left off when contacting the user again in order to finish the survey.
      • Useful surveys can be completed in less than a minute, and in some cases, in as little as a few seconds. This can make users more likely to respond to surveys in the future, increasing the overall response rate (compared to lengthy and disruptive e-mail surveys).
  • The following examples illustrate some representative features of a user survey service, according one or more embodiments of the present disclosure.
  • Example 16 Using Presence Information to Survey Users
  • In this example, a user survey service provides an increased number and higher quality of responses to end user surveys by utilizing real-time communication information. After determining if a user is available (e.g., online and involved in a meeting or other activity) based on presence information, the user survey service initiates a survey at that time via a real-time communication mechanism (e.g., instant messaging). The real-time communication is typically more immediate than other communications (e.g., e-mail) and generally increases the chance that the user will respond to the survey.
  • FIG. 16 is a diagram of a UC system that includes a user survey service. As shown in FIG. 16, the UC system includes a survey service server 1606 that communicates with devices associated with a customer site 1602 (e.g., edge server 1608 and user devices associated with a set of end users 1604). (The users/devices that are shown as being within the box in FIG. 16 that represents the customer site 1602 need not be physically present at the customer site 1602, but may be connected to a network associated with the customer site 1602.) As shown in FIG. 16, the edge server 1608 (e.g., a Lync edge server) publishes presence information to the survey service server 1606, which initiates a survey (e.g., via instant messages (IMs)) via the edge server 1608 based on user availability as indicated by the presence information. The edge server 1608 is only one example of a possible deployment choice in a UC system; other servers with different functionality and configurations also may be used.
  • The survey service server can obtain presence information in different ways. For example, a user may be signed in to an application that is trusted by the UC system, or the user's presence can be detected and authenticated when the user signs in to a user account, even if the user is not currently using a trusted application.
  • In this example, users that are determined to be available (indicated by curved arrows) are contacted by IM and asked to complete a survey. Users that are determined to be unavailable (e.g., offline, online but busy, etc.) are not contacted by IM. Users that are not available can be excluded from the survey, or their participation can be requested in some other way (e.g., via e-mail).
  • FIG. 17 is a flow chart that illustrates a survey technique that employs presence information. The technique shown in FIG. 17 can be implemented by the system shown in FIG. 16, or by some other system. In step 1702, a survey service starts up. In step 1704, the service retrieves information (e.g., how to connect to a customer site, and which users associated with the customer site are configured to take surveys) from a database. The service establishes a connection with the remote UC system (e.g., a Lync system) at step 1706, and subscribes to presence information of targeted users in step 1708. The service receives the presence information in step 1710 and begins processing the survey in step 1712. As indicated in step 1714, the service continues processing the survey until all survey processing is completed (see step 1724). In step 1716, the service processes an assigned user. In step 1718, the service determines if the user is available, and returns to step 1714 if not. If the user is available, the service determines if the user has been available for a configurable number of minutes in step 1720, as a “cooling off” period to avoid disturbing a user that has just finished a call, for example. (The service can wait for the configurable interval to elapse in step 1728, if needed, and continue processing for other users, as appropriate.) If the user has been available for the required period of time, the service determines, in step 1722, if a threshold number of attempts to contact the user has been reached. If the threshold has been reached, the user is marked as “processed, no answer” in step 1726. If the threshold has not been reached, the service determines, in step 1730, whether the user has already been contacted within a configurable number of hours, to avoid pestering a user that has been recently contacted. (The service can wait for the configurable interval for this user to elapse in step 1728, if needed, and continue processing for other users, as appropriate.) If the user has not been contacted within the configurable number of hours, the service contacts the user (e.g., via IM) to ask the user to take a survey (or to continue a survey if the user has previously started a survey) in step 1732.
  • In this example, the user can reply positively (e.g., “yes” or some other positive response, such as “start survey”) or negatively (e.g., “no” or some other negative response; a failure to reply within a given amount of time may be interpreted as a negative response). If the user's reply is negative, the service can wait for a configurable interval in step 1728 before attempting to contact the user again. Alternatively, the user may be given more options, such as “contact me later” or “do not contact me again.” If the user chooses not to be contacted again, the service can skip step 1728 and process another user, as appropriate.
  • If the user's reply is positive, the service starts or continues the survey, as appropriate, in step 1734, and the user provides survey responses in step 1736. In step 1738, the service determines whether the survey has been completed. If so, the user is marked as “done” in step 1740, and processing of the survey continues with additional users, as appropriate. If the survey has not been completed, the service marks the user as “contact again later” in step 1742. (The service can wait for the configurable interval for this user to elapse in step 1728, if needed, and continue processing for other users in the meantime, as appropriate.)
  • Many alternatives to the technique shown in FIG. 17 are possible. Various processing steps can be omitted, or the steps shown in FIG. 17 can be supplemented or replaced with other steps. For example, a survey service can omit the “cooling off” period after a user becomes available. As another example, for an important survey that requires participation from a particular user, the service can omit the option of not contacting the user again after a threshold number of attempts have occurred, and instead continue to contact the user until the survey is completed.
  • D. Operating Environment
  • In any of the examples described herein, client devices and administrator devices may be any suitable computing devices, including, but not limited to, laptop computers, desktop computers, smart phones, tablet computers, and/or the like. Servers may include suitable computing devices configured to provide services described in further detail below. As used herein in the context of a server-client relationship, the term “server” refers generally to a computing device that provides information (e.g., video and audio data) and/or services to other devices over a communication link (e.g., a network connection), and is not limited to any particular device configuration. Servers may include one or more suitable devices, such as dedicated server computing devices, or virtualized computing instances or application objects executing on a computing device. The term “client” can be used to refer to a computing device (e.g., a client device, an administrator device) that obtains information and/or accesses services provided by a server over a communication link, and is not limited to any particular device configuration. However, the designation of a particular device as a client device does not necessarily imply or require the presence of a server. At various times, a single device may act as a server, a client, a server and a client, or neither, depending on context and configuration. Actual physical locations of clients and servers are not necessarily important, but the locations can be described as “local” for a client and “remote” for a server to illustrate a common usage scenario in which a client is receiving information provided by a server at a remote location.
  • FIG. 18 is a block diagram that illustrates aspects of an exemplary computing device 1800 appropriate for use in accordance with embodiments of the present disclosure. The description below is applicable to servers, personal computers, mobile phones, smart phones, tablet computers, embedded computing devices, and other currently available or yet-to-be-developed devices that may be used in accordance with embodiments of the present disclosure.
  • In its most basic configuration, the computing device 1800 includes at least one processor 1802 and a system memory 1804 connected by a communication bus 1806. Depending on the exact configuration and type of device, the system memory 1804 may be volatile or nonvolatile memory, such as read only memory (“ROM”), random access memory (“RAM”), EEPROM, flash memory, or other memory technology. Those of ordinary skill in the art and others will recognize that system memory 1804 typically stores data and/or program modules that are immediately accessible to and/or currently being operated on by the processor 1802. In this regard, the processor 1802 may serve as a computational center of the computing device 1800 by supporting the execution of instructions.
  • As further illustrated in FIG. 18, the computing device 1800 may include a network interface 1810 comprising one or more components for communicating with other devices over a network. Embodiments of the present disclosure may access basic services that utilize the network interface 1810 to perform communications using common network protocols. The network interface 1810 may also include a wireless network interface configured to communicate via one or more wireless communication protocols, such as WiFi, 2G, 3G, 4G, LTE, WiMAX, Bluetooth, and/or the like.
  • In the exemplary embodiment depicted in FIG. 18, the computing device 1800 also includes a storage medium 1808. However, services may be accessed using a computing device that does not include means for persisting data to a local storage medium. Therefore, the storage medium 1808 depicted in FIG. 18 is optional. In any event, the storage medium 1808 may be volatile or nonvolatile, removable or nonremovable, implemented using any technology capable of storing information such as, but not limited to, a hard drive, solid state drive, CD-ROM, DVD, or other disk storage, magnetic tape, magnetic disk storage, and/or the like.
  • As used herein, the term “computer-readable medium” includes volatile and non-volatile and removable and non-removable media implemented in any method or technology capable of storing information, such as computer readable instructions, data structures, program modules, or other data. In this regard, the system memory 1804 and storage medium 1808 depicted in FIG. 18 are examples of computer-readable media.
  • For ease of illustration and because it is not important for an understanding of the claimed subject matter, FIG. 18 does not show some of the typical components of many computing devices. In this regard, the computing device 1800 may include input devices, such as a keyboard, keypad, mouse, trackball, microphone, video camera, touchpad, touchscreen, stylus, and/or the like. Such input devices may be coupled to the computing device 1800 by wired or wireless connections including RF, infrared, serial, parallel, Bluetooth, USB, or other suitable connections protocols using wireless or physical connections.
  • In any of the described examples, data can be captured by input devices and transmitted or stored for future processing. The processing may include encoding data streams, which can be subsequently decoded for presentation by output devices. Media data can be captured by multimedia input devices and stored by saving media data streams as files on a computer-readable storage medium (e.g., in memory or persistent storage on a client device, server, administrator device, or some other device). Multimedia input devices may include a video camera. A video camera, when active, may provide a stream of video data. As another example, multimedia input devices may include a microphone. A microphone, when active, may provide a stream of audio data. Input devices can be separate from and communicatively coupled to computing device 1800 (e.g., a client device), or can be integral components of the computing device 1800. In some embodiments, multiple input devices may be combined into a single, multifunction input device (e.g., a video camera with an integrated microphone). Any suitable input device either currently known or developed in the future may be used with described systems described herein.
  • The computing device 1800 may also include output devices such as a display, speakers, printer, etc. The output devices may include video output devices such as a display or touchscreen. The output devices also may include audio output devices such as external speakers or earphones. The output devices can be separate from and communicatively coupled to the computing device 1800, or can be integral components of the computing device 1800. In some embodiments, multiple output devices may be combined into a single device (e.g., a display with built-in speakers). Any suitable output device either currently known or developed in the future may be used with described systems.
  • In any of the described examples, digital signal processors (which can be implemented in hardware, software, or some combination of hardware and software) can be used for processing media data such as audio data and video data. For example, a digital signal processing module can include encoders to encode and/or decoders to decode encoded data in formats such as MP3, Vorbis, AAC, HE-AAC, or Windows Media Audio (WMA) for audio, or MPEG-2/H.262, H.263, VC-1, or H.264 for video.
  • In general, functionality of computing devices described herein may be implemented in computing logic embodied in hardware or software instructions, which can be written in a programming language, such as C, C++, COBOL, JAVA™, PHP, Perl, HTML, CSS, JavaScript, VBScript, ASPX, Microsoft .NET™ languages such as C#, and/or the like. Computing logic may be compiled into executable programs or written in interpreted programming languages. Generally, functionality described herein can be implemented as logic modules that can be duplicated to provide greater processing capability, merged with other modules, or divided into sub-modules. The computing logic can be stored in any type of computer-readable medium (e.g., a non-transitory medium such as a storage medium) or computer storage device and be stored on and executed by one or more general-purpose or special-purpose processors, thus creating a special-purpose computing device configured to provide functionality described herein.
  • E. Extensions and Alternatives
  • Many alternatives to the described systems are possible. For example, although only a single client device and administrator device are shown in FIG. 2 for ease of illustration, the described systems can comprise multiple client devices and administrator devices, which can interact with the system one at a time or simultaneously.
  • Many alternatives to the illustrated techniques are possible. For example, processing stages in techniques described herein can be separated into additional stages or combined into fewer stages. As another example, processing stages in techniques described herein can be omitted or supplemented with other techniques or processing stages. As another example, processing stages illustrated as occurring in a particular order can instead occur in a different order. As another example, processing stages that are described as being performed in a series of steps may instead be handled in a parallel fashion, with multiple modules or software processes concurrently handling one or more of the illustrated processing stages. As another example, processing stages that are indicated as being performed by a particular device or module may instead be performed by one or more other devices or modules.
  • F. Illustrative Embodiments
  • The present disclosure includes descriptions of various aspects of unified communication (UC) systems, including UC management and analysis systems and related tools and techniques. Described systems, tools, and techniques are adapted for enhanced UC data capture, analysis, and reporting; enhanced UC monitoring services; and a user survey service that can be used for conducting user surveys related to UC services.
  • Embodiments disclosed herein include:
      • A computer-implemented method for performing one or more of the above-described techniques.
      • A server computer comprising a processing unit and computer-readable storage media having stored thereon computer-executable instructions configured to cause the server computer to perform one or more of the above-described techniques.
      • A computer-readable storage medium having stored thereon computer-executable instructions configured to cause a computing device to perform one or more of the above-described techniques.
      • A computer system comprising a server that provides one or more of the above-described unified communication services. The computer system may further comprise plural client computing devices and an administrator computing service.
      • An administrator computing device in communication with a server that provides one or more of the above-described unified communication services, the administrator computing device comprising a processing unit and computer-readable storage media having stored thereon computer-executable instructions configured to cause the administrator computing device to perform one or more of the above-described techniques.
      • A client computing device in communication with a server that provides one or more of the above-described unified communication services, the client computing device comprising a processing unit and computer-readable storage media having stored thereon computer-executable instructions configured to cause the client computing device to perform one or more of the above-described techniques.
  • While illustrative embodiments have been illustrated and described, it will be appreciated that various changes can be made therein without departing from the spirit and scope of the claimed subject matter.

Claims (20)

The embodiments of the invention in which an exclusive property or privilege is claimed are defined as follows:
1. In a computer system comprising a server configured to execute a user survey service, a method comprising, by the user survey service:
establishing a connection with a remote unified communication system;
subscribing to presence information of targeted users;
receiving the presence information; and
processing a user survey based on the presence information.
2. The method of claim 1, further comprising, by the user survey service, requesting a user's participation in the user survey.
3. The method of claim 1, further comprising, by the user survey service:
determining whether a user is available; and
if the user is not available, excluding the user from the user survey.
4. The method of claim 1, further comprising, by the user survey service:
determining whether a user is available; and
if the user is available, determining if the user has been available for a period of time.
5. The method of claim 1, further comprising, by the user survey service, determining whether a threshold number of attempts to contact a user has been reached.
6. The method of claim 1, further comprising, by the user survey service, determining whether a user has been contacted within a period of time.
7. The method of claim 1, further comprising, by the user survey service:
determining if a user has been available for a period of time; and
if the user has been available for the period of time, determining whether a threshold number of attempts to contact the user has been reached.
8. The method of claim 1, further comprising, by the user survey service:
determining that a threshold number of attempts to contact a user has not been reached;
determining that the user has not been contacted within a period of time; and
contacting the user with a request to take or continue the user survey.
9. The method of claim 1, further comprising, by the user survey service, contacting a user with a request to take or continue the user survey via a real-time communication mechanism.
10. The method of claim 9, wherein the real-time communication mechanism comprises an instant message.
11. A computer system comprising a server that provides a user survey service, wherein the server is configured to:
establish a connection with a remote unified communication system; and
subscribe to presence information of targeted users;
receive the presence information; and
process a user survey based on the presence information.
12. The computer system of claim 11, wherein the server is further configured to request a user's participation in the user survey.
13. The computer system of claim 11, wherein the server is further configured to:
determine whether a user is available; and
if the user is not available, exclude the user from the user survey.
14. The computer system of claim 11, wherein the server is further configured to:
determine whether a user is available; and
if the user is available, determine if the user has been available for a period of time.
15. The computer system of claim 11, wherein the server is further configured to determine whether a threshold number of attempts to contact a user has been reached.
16. The computer system of claim 11, wherein the server is further configured to determine whether a user has been contacted within a period of time.
17. The computer system of claim 11, wherein the server is further configured to:
determine if a user has been available for a period of time; and
if the user has been available for the period of time, determine whether a threshold number of attempts to contact the user has been reached.
18. The computer system of claim 11, wherein the server is further configured to:
determine that a threshold number of attempts to contact the user has not been reached;
determine that the user has not been contacted within a period of time; and
contact the user with a request to take or continue the survey.
19. The computer system of claim 11, wherein the server is further configured to contact a user with a request to take or continue the survey via a real-time communication mechanism.
20. A unified communication system comprising a survey service server, wherein the survey service server is configured to:
communicate with user devices associated with a customer site via a server configured to publish presence information to the survey service server; and
initiate a survey based on user availability as indicated by the presence information.
US14/178,238 2013-02-12 2014-02-11 User Survey Service for Unified Communications Abandoned US20140229236A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US14/178,238 US20140229236A1 (en) 2013-02-12 2014-02-11 User Survey Service for Unified Communications

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US201361763919P 2013-02-12 2013-02-12
US14/178,238 US20140229236A1 (en) 2013-02-12 2014-02-11 User Survey Service for Unified Communications

Publications (1)

Publication Number Publication Date
US20140229236A1 true US20140229236A1 (en) 2014-08-14

Family

ID=51297434

Family Applications (7)

Application Number Title Priority Date Filing Date
US14/178,238 Abandoned US20140229236A1 (en) 2013-02-12 2014-02-11 User Survey Service for Unified Communications
US14/178,228 Abandoned US20140226800A1 (en) 2013-02-12 2014-02-11 Enhanced Monitoring of Performance for Unified Communication Services
US14/178,209 Active US9071677B2 (en) 2013-02-12 2014-02-11 Enhanced data capture, analysis, and reporting for unified communications
US14/179,476 Active 2034-07-19 US9860368B2 (en) 2013-02-12 2014-02-12 Advanced tools for unified communication data management and analysis
US14/721,813 Active US9503570B2 (en) 2013-02-12 2015-05-26 Enhanced data capture, analysis, and reporting for unified communications
US15/356,361 Active 2034-04-24 US10674007B2 (en) 2013-02-12 2016-11-18 Enhanced data capture, analysis, and reporting for unified communications
US15/827,786 Abandoned US20180084107A1 (en) 2013-02-12 2017-11-30 Advanced tools for unified communication data management and analysis

Family Applications After (6)

Application Number Title Priority Date Filing Date
US14/178,228 Abandoned US20140226800A1 (en) 2013-02-12 2014-02-11 Enhanced Monitoring of Performance for Unified Communication Services
US14/178,209 Active US9071677B2 (en) 2013-02-12 2014-02-11 Enhanced data capture, analysis, and reporting for unified communications
US14/179,476 Active 2034-07-19 US9860368B2 (en) 2013-02-12 2014-02-12 Advanced tools for unified communication data management and analysis
US14/721,813 Active US9503570B2 (en) 2013-02-12 2015-05-26 Enhanced data capture, analysis, and reporting for unified communications
US15/356,361 Active 2034-04-24 US10674007B2 (en) 2013-02-12 2016-11-18 Enhanced data capture, analysis, and reporting for unified communications
US15/827,786 Abandoned US20180084107A1 (en) 2013-02-12 2017-11-30 Advanced tools for unified communication data management and analysis

Country Status (3)

Country Link
US (7) US20140229236A1 (en)
EP (1) EP2957119A4 (en)
WO (1) WO2014127055A2 (en)

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9185589B2 (en) 2009-08-31 2015-11-10 The Nielsen Company (Us), Llc Methods and apparatus to identify wireless carrier performance effects
US9848089B2 (en) 2014-11-05 2017-12-19 The Nielsen Company (Us), Llc Methods and apparatus to generate an overall performance index
US10558988B2 (en) 2016-05-09 2020-02-11 International Business Machines Corporation Survey based on user behavior pattern
US20200344115A1 (en) * 2019-04-25 2020-10-29 Elo Touch Solutions, Inc. Zero touch deployment and dynamic configuration
US11337084B2 (en) 2015-09-29 2022-05-17 Soracom, Inc. Control apparatus for gateway in mobile communication system
CN115086488A (en) * 2022-07-27 2022-09-20 广东创新科技职业学院 Number classification method and device

Families Citing this family (122)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CA2614440C (en) * 2005-07-07 2016-06-21 Sermo, Inc. Method and apparatus for conducting an information brokering service
EP3484135A1 (en) 2008-04-02 2019-05-15 Twilio Inc. System and method for processing telephony sessions
US8837465B2 (en) 2008-04-02 2014-09-16 Twilio, Inc. System and method for processing telephony sessions
US8964726B2 (en) 2008-10-01 2015-02-24 Twilio, Inc. Telephony web event system and method
WO2010101935A1 (en) 2009-03-02 2010-09-10 Twilio Inc. Method and system for a multitenancy telephone network
US9426306B2 (en) * 2009-05-15 2016-08-23 Morgan Stanley Systems and method for determining a relationship rank
US9210275B2 (en) 2009-10-07 2015-12-08 Twilio, Inc. System and method for running a multi-module telephony application
US9203652B2 (en) 2009-12-21 2015-12-01 8X8, Inc. Systems, methods, devices and arrangements for cost-effective routing
US20120208495A1 (en) 2010-06-23 2012-08-16 Twilio, Inc. System and method for monitoring account usage on a platform
US8838707B2 (en) 2010-06-25 2014-09-16 Twilio, Inc. System and method for enabling real-time eventing
US8649268B2 (en) 2011-02-04 2014-02-11 Twilio, Inc. Method for processing telephony sessions of a network
WO2012162397A1 (en) 2011-05-23 2012-11-29 Twilio, Inc. System and method for connecting a communication to a client
US20140044123A1 (en) 2011-05-23 2014-02-13 Twilio, Inc. System and method for real time communicating with a client application
KR20130024739A (en) * 2011-08-31 2013-03-08 성균관대학교산학협력단 System and method for analyzing experience in real time
US9495227B2 (en) 2012-02-10 2016-11-15 Twilio, Inc. System and method for managing concurrent events
US9602586B2 (en) 2012-05-09 2017-03-21 Twilio, Inc. System and method for managing media in a distributed communication network
US9247062B2 (en) 2012-06-19 2016-01-26 Twilio, Inc. System and method for queuing a communication session
US8938053B2 (en) 2012-10-15 2015-01-20 Twilio, Inc. System and method for triggering on platform usage
US10313905B2 (en) 2012-10-29 2019-06-04 T-Mobile Usa, Inc. Contextual quality of user experience analysis using equipment dynamics
US10237144B2 (en) 2012-10-29 2019-03-19 T-Mobile Usa, Inc. Quality of user experience analysis
US10412550B2 (en) 2012-10-29 2019-09-10 T-Mobile Usa, Inc. Remote driving of mobile device diagnostic applications
US10952091B2 (en) 2012-10-29 2021-03-16 T-Mobile Usa, Inc. Quality of user experience analysis
US9538409B2 (en) 2012-10-29 2017-01-03 T-Mobile Usa, Inc. Quality of user experience analysis
US20140171017A1 (en) * 2012-12-17 2014-06-19 Verizon Patent And Licensing, Inc. Billing system user interface tool
US20140229236A1 (en) 2013-02-12 2014-08-14 Unify Square, Inc. User Survey Service for Unified Communications
US9817699B2 (en) * 2013-03-13 2017-11-14 Elasticbox Inc. Adaptive autoscaling for virtualized applications
US9282124B2 (en) 2013-03-14 2016-03-08 Twilio, Inc. System and method for integrating session initiation protocol communication in a telecommunications platform
US9483558B2 (en) * 2013-05-29 2016-11-01 Commvault Systems, Inc. Assessing user performance in a community of users of data storage resources
US9240966B2 (en) 2013-06-19 2016-01-19 Twilio, Inc. System and method for transmitting and receiving media messages
US9137127B2 (en) 2013-09-17 2015-09-15 Twilio, Inc. System and method for providing communication platform metadata
US9274858B2 (en) 2013-09-17 2016-03-01 Twilio, Inc. System and method for tagging and tracking events of an application platform
EP3058490A4 (en) * 2013-10-17 2017-04-26 Hewlett-Packard Enterprise Development LP Storing data at a remote location based on predetermined criteria
US9325624B2 (en) 2013-11-12 2016-04-26 Twilio, Inc. System and method for enabling dynamic multi-modal communication
US9553799B2 (en) 2013-11-12 2017-01-24 Twilio, Inc. System and method for client communication in a distributed telephony network
US9344340B2 (en) * 2013-11-18 2016-05-17 International Business Machines Corporation System and method for notification of QoE issues
US9232048B2 (en) * 2013-12-04 2016-01-05 International Business Machines Corporation Quality of experience determination for multi-party VoIP conference calls that account for focus degradation effects
US20150269567A1 (en) * 2014-03-19 2015-09-24 Mastercard International Incorporated Methods and systems for improving payment card acceptance quality
US10044774B1 (en) * 2014-03-31 2018-08-07 Sonus Networks, Inc. Methods and apparatus for aggregating and distributing presence information
US9398107B1 (en) 2014-03-31 2016-07-19 Sonus Networks, Inc. Methods and apparatus for aggregating and distributing contact and presence information
US10306000B1 (en) 2014-03-31 2019-05-28 Ribbon Communications Operating Company, Inc. Methods and apparatus for generating, aggregating and/or distributing presence information
US9226217B2 (en) 2014-04-17 2015-12-29 Twilio, Inc. System and method for enabling multi-modal communication
US20150310375A1 (en) * 2014-04-28 2015-10-29 Infosys Limited Individual productivity measurement
CN105224558A (en) * 2014-06-16 2016-01-06 华为技术有限公司 The evaluation disposal route of speech business and device
US9251371B2 (en) 2014-07-07 2016-02-02 Twilio, Inc. Method and system for applying data retention policies in a computing platform
US9774687B2 (en) 2014-07-07 2017-09-26 Twilio, Inc. System and method for managing media and signaling in a communication platform
US9246694B1 (en) 2014-07-07 2016-01-26 Twilio, Inc. System and method for managing conferencing in a distributed communication network
US9516101B2 (en) * 2014-07-07 2016-12-06 Twilio, Inc. System and method for collecting feedback in a multi-tenant communication platform
US9847918B2 (en) * 2014-08-12 2017-12-19 Microsoft Technology Licensing, Llc Distributed workload reassignment following communication failure
EP3210350B1 (en) 2014-10-21 2020-05-20 Twilio, Inc. Method for providing a miro-services communication platform
US9882789B2 (en) * 2014-10-29 2018-01-30 At&T Intellectual Property I, L.P. Service assurance platform as a user-defined service
CN105700965A (en) * 2014-11-26 2016-06-22 英业达科技有限公司 System error exclusion method
USD757758S1 (en) * 2014-12-16 2016-05-31 LeGuard, Inc. Display screen with graphical user interface
USD757050S1 (en) * 2014-12-16 2016-05-24 LeGuard, Inc. Display screen with graphical user interface
USD757051S1 (en) * 2014-12-16 2016-05-24 LeGuard, Inc. Display screen with graphical user interface
USD757049S1 (en) * 2014-12-16 2016-05-24 LeGuard, Inc. Display screen with graphical user interface
CN107211300A (en) * 2015-01-26 2017-09-26 诺基亚通信公司 Analysis and classification signaling set or calling
US9477975B2 (en) 2015-02-03 2016-10-25 Twilio, Inc. System and method for a media intelligence platform
JP6576069B2 (en) 2015-03-24 2019-09-18 任天堂株式会社 Information processing system, information processing apparatus, information processing program, and information processing method
JP2016181154A (en) * 2015-03-24 2016-10-13 任天堂株式会社 Information processing system, server system, information processing device, information processing program, and information processing method
WO2016179197A1 (en) * 2015-05-04 2016-11-10 Onepin, Inc. Automatic aftercall directory and phonebook entry advertising
US10419891B2 (en) 2015-05-14 2019-09-17 Twilio, Inc. System and method for communicating through multiple endpoints
US9948703B2 (en) 2015-05-14 2018-04-17 Twilio, Inc. System and method for signaling through data storage
WO2016196044A1 (en) * 2015-05-29 2016-12-08 T-Mobile Usa, Inc. Quality of user experience analysis using echo locate
US10567467B2 (en) 2015-06-22 2020-02-18 Sandvine Corporation System and method for heuristic control of network traffic management
US10282245B1 (en) 2015-06-25 2019-05-07 Amazon Technologies, Inc. Root cause detection and monitoring for storage systems
US10223189B1 (en) 2015-06-25 2019-03-05 Amazon Technologies, Inc. Root cause detection and monitoring for storage systems
US9898357B1 (en) 2015-06-25 2018-02-20 Amazon Technologies, Inc. Root cause detection and monitoring for storage systems
CN105224658B (en) * 2015-09-30 2018-11-30 北京京东尚科信息技术有限公司 A kind of Query method in real time and system of big data
US10582266B2 (en) * 2015-10-07 2020-03-03 Vasona Networks Inc. Rating video-download quality
WO2017100673A1 (en) * 2015-12-09 2017-06-15 Unify Square, Inc. Voice quality dashboard for unified communication system
WO2017100664A1 (en) * 2015-12-09 2017-06-15 Unify Square, Inc. Automated detection and analysis of call conditions in communication system
US10419310B1 (en) 2015-12-17 2019-09-17 8×8, Inc. Monitor device for use with endpoint devices
US10375199B2 (en) * 2015-12-30 2019-08-06 Facebook, Inc. Systems and methods for surveying users
WO2017116853A1 (en) * 2015-12-30 2017-07-06 T-Mobile Usa, Inc. Contextual quality of user experience analysis using equipment dynamics
US10659349B2 (en) 2016-02-04 2020-05-19 Twilio Inc. Systems and methods for providing secure network exchanged for a multitenant virtual private cloud
US10700767B2 (en) 2016-03-16 2020-06-30 Honeywell International Inc. Requesting weather data based on pre-selected events
US10321336B2 (en) * 2016-03-16 2019-06-11 Futurewei Technologies, Inc. Systems and methods for robustly determining time series relationships in wireless networks
US10374882B2 (en) * 2016-03-16 2019-08-06 Futurewei Technologies, Inc. Systems and methods for identifying causes of quality degradation in wireless networks
US10686902B2 (en) 2016-05-23 2020-06-16 Twilio Inc. System and method for a multi-channel notification service
US10063713B2 (en) 2016-05-23 2018-08-28 Twilio Inc. System and method for programmatic device connectivity
US11010717B2 (en) * 2016-06-21 2021-05-18 The Prudential Insurance Company Of America Tool for improving network security
CN107786618B (en) * 2016-08-31 2020-09-08 迈普通信技术股份有限公司 Method and device for selecting login node
US10313406B2 (en) * 2016-11-01 2019-06-04 Microsoft Technology Licensing, Llc Synthetic transaction to determine centroid for cloud hosting
US11216342B2 (en) * 2016-12-12 2022-01-04 Usablenet Inc. Methods for improved auditing of web sites and devices thereof
TR201619423A2 (en) * 2016-12-23 2018-07-23 Turkcell Teknoloji Arastirma Ve Gelistirme Anonim Sirketi A SYSTEM THAT SHOWS THE TRENDS OF CLOSE REAL TIME QUALITY AND PRODUCES ALARM
US11017343B2 (en) * 2017-01-10 2021-05-25 Moduleq, Inc. Personal data fusion
US20180278459A1 (en) * 2017-03-27 2018-09-27 Cisco Technology, Inc. Sharding Of Network Resources In A Network Policy Platform
US9813495B1 (en) * 2017-03-31 2017-11-07 Ringcentral, Inc. Systems and methods for chat message notification
US20180316741A1 (en) * 2017-05-01 2018-11-01 Microsoft Technology Licensing, Llc Synthetic Transaction based on Network Condition
US10701117B1 (en) * 2017-06-02 2020-06-30 Amdocs Development Limited System, method, and computer program for managing conference calls between a plurality of conference call systems
US10628236B2 (en) * 2017-06-06 2020-04-21 Huawei Technologies Canada Co., Ltd. System and method for inter-datacenter communication
US10462306B1 (en) 2017-07-24 2019-10-29 vMOX, LLC Mobile device usage optimization
US10033879B1 (en) * 2017-07-24 2018-07-24 vMOX, LLC Mobile device usage optimization
CN109327320B (en) * 2017-07-31 2020-11-06 华为技术有限公司 Fault delimiting method and equipment
WO2019039632A1 (en) 2017-08-25 2019-02-28 라인 가부시키가이샤 Method and device for connecting user terminals as group and providing service including contents related to group
US11416799B2 (en) * 2017-08-28 2022-08-16 Clari Inc. Method and system for summarizing user activities of tasks into a single activity score using machine learning to predict probabilities of completeness of the tasks
WO2019213427A1 (en) 2018-05-04 2019-11-07 Laibson Benjamin William Emulation of cloud computing service regions
CN108600561A (en) * 2018-05-10 2018-09-28 上海二六三通信有限公司 A kind of communications management system
US10959083B2 (en) 2018-08-06 2021-03-23 vMOX, LLC Application level usage based optimization
US11032329B2 (en) * 2019-01-29 2021-06-08 Fanmio, Inc. Managing engagements in interactive multimedia sessions
US10805361B2 (en) 2018-12-21 2020-10-13 Sansay, Inc. Communication session preservation in geographically redundant cloud-based systems
US10503800B1 (en) * 2018-12-27 2019-12-10 Coupa Software Incorporated System and methods for enabling disintermediated communication associated with search operations
US11328263B2 (en) 2019-02-21 2022-05-10 Microsoft Technology Licensing, Llc User interfaces for filtering electronic calendar data sets
CN109933648B (en) * 2019-02-28 2022-07-05 北京学之途网络科技有限公司 Real user comment distinguishing method and device
US10440175B1 (en) * 2019-03-07 2019-10-08 Securus Technologies, Inc. Dynamic controlled-environment facility resident communication allocation based on call volume
CN110083638A (en) * 2019-04-24 2019-08-02 广东联合电子服务股份有限公司 A kind of regular base construction method of delay and data retention analysis method
US11206289B2 (en) * 2019-05-16 2021-12-21 Level 3 Communications, Llc Monitoring and detection of fraudulent or unauthorized use in telephone conferencing systems or voice networks
US20210065222A1 (en) * 2019-08-26 2021-03-04 Microsoft Technology Licensing, Llc User sentiment metrics
US11196870B2 (en) * 2019-09-27 2021-12-07 Mitel Clouds Services, Inc. Method, system, and device for cloud voice quality monitoring
US11113322B2 (en) 2020-01-07 2021-09-07 Bank Of America Corporation Dynamically generating strategic planning datasets based on collecting, aggregating, and filtering distributed data collections
US10965806B1 (en) * 2020-01-31 2021-03-30 Noble Systems Corporation Auto-correcting voice quality in real-time
US11032170B1 (en) * 2020-02-25 2021-06-08 Bank Of America Corporation Remotely-deployed automated computer network diagnostic tool
US11671484B2 (en) * 2020-09-25 2023-06-06 Verizon Patent And Licensing Inc. Methods and systems for orchestrating a distributed computing service based on latency performance levels
CN112235425B (en) * 2020-12-14 2021-03-09 长沙理工大学 Block chain mine pool forming method, device and system and readable storage medium
US11343373B1 (en) 2021-01-29 2022-05-24 T-Mobile Usa, Inc. Machine intelligent isolation of international calling performance degradation
US11363237B1 (en) 2021-04-09 2022-06-14 Ryan Block System and method for notifying others when a person is on a conference
US20220335357A1 (en) * 2021-04-16 2022-10-20 International Business Machines Corporation Identifying an influencer combination having a root cause to a key performance indicator change
US20220366430A1 (en) * 2021-05-14 2022-11-17 At&T Intellectual Property I, L.P. Data stream based event sequence anomaly detection for mobility customer fraud analysis
US20230101995A1 (en) * 2021-09-29 2023-03-30 Bank Of America Corporation System and methods for proactive protection against malfeasant data collection
US11854028B2 (en) * 2021-11-01 2023-12-26 Microsoft Technology Licensing, Llc Reinforcement learning applied to survey parameter optimization
US20230162843A1 (en) * 2021-11-22 2023-05-25 GE Precision Healthcare LLC Systems and methods for workload management
WO2024039290A1 (en) * 2022-08-15 2024-02-22 Zzivor Pte. Ltd. A system and method for dynamic matching of a request to a member population

Citations (107)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4130881A (en) * 1971-07-21 1978-12-19 Searle Medidata, Inc. System and technique for automated medical history taking
US5058152A (en) * 1989-12-12 1991-10-15 The Telephone Connection Anonymous interactive telephone system having direct connect feature
US5227874A (en) * 1986-03-10 1993-07-13 Kohorn H Von Method for measuring the effectiveness of stimuli on decisions of shoppers
US5456607A (en) * 1989-12-13 1995-10-10 Antoniak; Peter R. Knowledge testing computer game method employing the repositioning of screen objects to represent data relationships
US5496175A (en) * 1991-02-01 1996-03-05 Hitachi, Ltd. Questionnaire system
US5566291A (en) * 1993-12-23 1996-10-15 Diacom Technologies, Inc. Method and apparatus for implementing user feedback
US5590057A (en) * 1993-12-20 1996-12-31 Atlantic Richfield Company Training and certification system and method
US5754939A (en) * 1994-11-29 1998-05-19 Herz; Frederick S. M. System for generation of user profiles for a system for customized electronic identification of desirable objects
US5788508A (en) * 1992-02-11 1998-08-04 John R. Lee Interactive computer aided natural learning method and apparatus
US5791907A (en) * 1996-03-08 1998-08-11 Ramshaw; Bruce J. Interactive medical training system
US5802502A (en) * 1993-05-24 1998-09-01 British Telecommunications Public Limited Company System for selective communication connection based on transaction pricing signals
US5913204A (en) * 1996-08-06 1999-06-15 Kelly; Thomas L. Method and apparatus for surveying music listener opinion about songs
US5915243A (en) * 1996-08-29 1999-06-22 Smolen; Daniel T. Method and apparatus for delivering consumer promotions
US5940471A (en) * 1996-10-04 1999-08-17 Northern Telecom Limited Method and apparatus for obtaining feedback regarding network services
US5999908A (en) * 1992-08-06 1999-12-07 Abelow; Daniel H. Customer-based product design module
US6026387A (en) * 1996-07-15 2000-02-15 Kesel; Brad Consumer comment reporting apparatus and method
US6070145A (en) * 1996-07-12 2000-05-30 The Npd Group, Inc. Respondent selection method for network-based survey
US6092049A (en) * 1995-06-30 2000-07-18 Microsoft Corporation Method and apparatus for efficiently recommending items using automated collaborative filtering and feature-guided automated collaborative filtering
US6236975B1 (en) * 1998-09-29 2001-05-22 Ignite Sales, Inc. System and method for profiling customers for targeted marketing
US20010003214A1 (en) * 1999-07-15 2001-06-07 Vijnan Shastri Method and apparatus for utilizing closed captioned (CC) text keywords or phrases for the purpose of automated searching of network-based resources for interactive links to universal resource locators (URL's)
US20010018673A1 (en) * 1998-03-12 2001-08-30 Steve Goldband Interactive customer support for computer programs using network connection of user machine
US6298119B1 (en) * 2000-01-11 2001-10-02 Siemens Information And Communication Networks, Inc. System and method for digital telephone trouble reporting
US20010037206A1 (en) * 2000-03-02 2001-11-01 Vivonet, Inc. Method and system for automatically generating questions and receiving customer feedback for each transaction
US20010055750A1 (en) * 2000-04-10 2001-12-27 Rasche Jeanette D. Method and apparatus for educating asthma sufferers and caregivers
US20020010628A1 (en) * 2000-05-24 2002-01-24 Alan Burns Method of advertising and polling
US20020029381A1 (en) * 2000-09-06 2002-03-07 Eric Inselberg Method and apparatus for interactive audience participation at a live spectator event
US6370120B1 (en) * 1998-12-24 2002-04-09 Mci Worldcom, Inc. Method and system for evaluating the quality of packet-switched voice signals
US20020049635A1 (en) * 2000-09-06 2002-04-25 Khanh Mai Multiple advertising
US20020052774A1 (en) * 1999-12-23 2002-05-02 Lance Parker Collecting and analyzing survey data
US6421724B1 (en) * 1999-08-30 2002-07-16 Opinionlab, Inc. Web site response measurement tool
US20020103696A1 (en) * 2001-01-29 2002-08-01 Huang Jong S. System and method for high-density interactive voting using a computer network
US20020120501A1 (en) * 2000-07-19 2002-08-29 Bell Christopher Nathan Systems and processes for measuring, evaluating and reporting audience response to audio, video, and other content
US20020120491A1 (en) * 2000-05-31 2002-08-29 Nelson Eugene C. Interactive survey and data management method and apparatus
US20020120504A1 (en) * 2000-07-31 2002-08-29 Intermedia Advertising Group Computerized system and method for increasing the effectiveness of advertising
US20020128898A1 (en) * 1998-03-02 2002-09-12 Leroy Smith Dynamically assigning a survey to a respondent
US20020161779A1 (en) * 2000-03-07 2002-10-31 Brierley Harold M. Method and system for evaluating, reporting, and improving on-line promotion effectiveness
US20030004778A1 (en) * 2001-05-10 2003-01-02 Gareau Brian R. Computer architecture and computer iplemented and/or assisted method of performing a cultural assessment of an organization and making improvements thereon
US20030009759A1 (en) * 2000-01-19 2003-01-09 Denis Khoo Method and system for providing home shopping programs
US20030018517A1 (en) * 2001-07-20 2003-01-23 Dull Stephen F. Providing marketing decision support
US6519571B1 (en) * 1999-05-27 2003-02-11 Accenture Llp Dynamic customer profile management
US20030055722A1 (en) * 2001-09-19 2003-03-20 Jagtec, Inc. Method and apparatus for control of advertisements
US20030070180A1 (en) * 2001-09-28 2003-04-10 Toshio Katayama System for assisting consideration of selection
US20030078804A1 (en) * 2001-10-24 2003-04-24 Palmer Morrel-Samuels Employee assessment tool
US20030088452A1 (en) * 2001-01-19 2003-05-08 Kelly Kevin James Survey methods for handheld computers
US20030101088A1 (en) * 2000-11-27 2003-05-29 Suriyan Lohavichan Web-based survey method for measuring customer service response
US20030105660A1 (en) * 2001-02-20 2003-06-05 Walsh Kenneth Peter Method of relating multiple independent databases
US6581071B1 (en) * 2000-09-12 2003-06-17 Survivors Of The Shoah Visual History Foundation Surveying system and method
US20030115216A1 (en) * 2001-12-19 2003-06-19 First Data Corporation Methods and systems for developing market intelligence
US6597903B1 (en) * 1998-11-02 2003-07-22 Openwave Systems Inc. Online churn reduction and loyalty system
US6606581B1 (en) * 2000-06-14 2003-08-12 Opinionlab, Inc. System and method for measuring and reporting user reactions to particular web pages of a website
US20030163371A1 (en) * 2000-04-11 2003-08-28 Jeremy Beard System and method for presenting information over time to a user
US20030177061A1 (en) * 2002-02-22 2003-09-18 Spady Richard J. Method for presenting opinions and measuring "social (intangible) assets"
US20030182135A1 (en) * 2002-03-21 2003-09-25 Masahiro Sone System and method for customer satisfaction survey and analysis for off-site customer service
US6631184B1 (en) * 2000-07-24 2003-10-07 Comverse Ltd. System for community generated feedback and/or rating
US20030195807A1 (en) * 2000-10-12 2003-10-16 Frank S. Maggio Method and system for verifying exposure to message content via a printed response
US6636590B1 (en) * 2000-10-30 2003-10-21 Ingenio, Inc. Apparatus and method for specifying and obtaining services through voice commands
US20030200135A1 (en) * 2002-04-19 2003-10-23 Wright Christine Ellen System and method for predicting and preventing customer churn
US20030229533A1 (en) * 2002-06-06 2003-12-11 Mack Mary E. System and method for creating compiled marketing research data over a computer network
US20030227870A1 (en) * 2002-06-03 2003-12-11 Wagner Clinton Allen Method and system for automated voice quality statistics gathering
US20040006478A1 (en) * 2000-03-24 2004-01-08 Ahmet Alpdemir Voice-interactive marketplace providing promotion and promotion tracking, loyalty reward and redemption, and other features
US20040015399A1 (en) * 2000-10-12 2004-01-22 Maggio Frank S. Method and system for verifying exposure to message content delivered via outdoor media or in a concentrated format
US20040019688A1 (en) * 2002-07-29 2004-01-29 Opinionlab Providing substantially real-time access to collected information concerning user interaction with a web page of a website
US6690919B1 (en) * 1998-05-05 2004-02-10 Mannesmann Ag Determining the quality of telecommunication services
US20040030697A1 (en) * 2002-07-31 2004-02-12 American Management Systems, Inc. System and method for online feedback
US20040044563A1 (en) * 2000-01-18 2004-03-04 Valuestar, Inc. System and method for real-time updating service provider ratings
US20040059625A1 (en) * 2002-09-20 2004-03-25 Ncr Corporation Method for providing feedback to advertising on interactive channels
US20040073476A1 (en) * 2002-10-10 2004-04-15 Prolink Services Llc Method and system for identifying key opinion leaders
US20040073641A1 (en) * 2002-09-30 2004-04-15 Muneyb Minhazuddin Instantaneous user initiation voice quality feedback
US20040103032A1 (en) * 2000-10-12 2004-05-27 Maggio Frank S. Remote control system and method for interacting with broadcast content
US20040123314A1 (en) * 2002-12-23 2004-06-24 Bova Alfred T. Method and system for integrating television brand advertising with promotional marketing
US20040133463A1 (en) * 2000-03-06 2004-07-08 Theodore Benderev On-line survey method
US6766524B1 (en) * 2000-05-08 2004-07-20 Webtv Networks, Inc. System and method for encouraging viewers to watch television programs
US20040143498A1 (en) * 2002-12-27 2004-07-22 Toshihiko Umeda Service supporting system, service supporting server and service supporting method
US20040143478A1 (en) * 2003-01-18 2004-07-22 Ward Andrew David Method and process for capuring, storing, processing and displaying customer satisfaction information
US20040158455A1 (en) * 2002-11-20 2004-08-12 Radar Networks, Inc. Methods and systems for managing entities in a computing device using semantic objects
US6778807B1 (en) * 2000-09-15 2004-08-17 Documus, Llc Method and apparatus for market research using education courses and related information
US20040172323A1 (en) * 2003-02-28 2004-09-02 Bellsouth Intellectual Property Corporation Customer feedback method and system
US20040181448A1 (en) * 2003-03-14 2004-09-16 Paul Hartsman Marketing network
US20040199574A1 (en) * 1999-09-14 2004-10-07 Franco Louis M. System and method for delivering remotely stored applications and information
US20040210820A1 (en) * 2002-12-27 2004-10-21 Douglas Tarr Automated compensation reports using online surveys and collaborative filtering
US20040220893A1 (en) * 2002-11-20 2004-11-04 Radar Networks, Inc. User interface for managing semantic objects
US20040230676A1 (en) * 2002-11-20 2004-11-18 Radar Networks, Inc. Methods and systems for managing offers and requests in a network
US20040236625A1 (en) * 2001-06-08 2004-11-25 Kearon John Victor Method apparatus and computer program for generating and evaluating feelback from a plurality of respondents
US20040235460A1 (en) * 2001-05-11 2004-11-25 Engstrom G. Eric Method and system for providing an opinion and aggregating opinions with mobile telecommunication device
US6826540B1 (en) * 1999-12-29 2004-11-30 Virtual Personalities, Inc. Virtual human interface for conducting surveys
US20040249712A1 (en) * 2003-06-06 2004-12-09 Brown Sean D. System, method and computer program product for presenting, redeeming and managing incentives
US20040252816A1 (en) * 2003-06-13 2004-12-16 Christophe Nicolas Mobile phone sample survey method
US20040261125A1 (en) * 1998-08-27 2004-12-23 United Video Properties, Inc. Electronic program guide with interactive screen game
US6845485B1 (en) * 1999-07-15 2005-01-18 Hotv, Inc. Method and apparatus for indicating story-line changes by mining closed-caption-text
US20050055275A1 (en) * 2003-06-10 2005-03-10 Newman Alan B. System and method for analyzing marketing efforts
US20050054286A1 (en) * 2001-10-15 2005-03-10 Jawahar Kanjilal Method of providing live feedback
US20050055266A1 (en) * 2003-09-05 2005-03-10 Pitney Bowes Incorporated Method and system for generating information about relationships between an enterprise and other parties and sharing such information among users in the enterprise
US20050065851A1 (en) * 2003-09-22 2005-03-24 Aronoff Jeffrey M. System, method and computer program product for supplying to and collecting information from individuals
US20050075919A1 (en) * 2000-08-23 2005-04-07 Jeong-Uk Kim Method for respondent-based real-time survey
US20050091077A1 (en) * 2003-08-25 2005-04-28 Reynolds Thomas J. Determining strategies for increasing loyalty of a population to an entity
US20050096943A1 (en) * 2003-11-05 2005-05-05 Siegalovsky Ilene L. System and method for correlating market research data based on sales representative activity
US20050114171A1 (en) * 2003-11-05 2005-05-26 Siegalovsky Ilene L. System and method for correlating market research data based on attitude information
US20050119931A1 (en) * 2003-12-01 2005-06-02 Matthew Schall Disparate survey score conversion and comparison method
US20060240803A1 (en) * 2005-04-26 2006-10-26 Xerox Corporation Automated notification systems and methods
US20060291641A1 (en) * 2005-06-02 2006-12-28 Lucent Technologies Inc. Methods and systems for selective threshold based call blocking
US20080182555A1 (en) * 2006-12-08 2008-07-31 Rodrigo Madanes Communication system
US20080291896A1 (en) * 2007-03-28 2008-11-27 Tauri Tuubel Detection of communication states
US20090237240A1 (en) * 2008-03-19 2009-09-24 Microsoft Corporation Determining quality monitoring alerts in unified communication systems
US20100070345A1 (en) * 1992-08-06 2010-03-18 Abelow Daniel H Customer-based product design module
US20110009707A1 (en) * 2008-05-07 2011-01-13 Kaundinya Murali P Telehealth Scheduling and Communications Network
US20110137696A1 (en) * 2009-12-04 2011-06-09 3Pd Performing follow-up actions based on survey results
US8095665B1 (en) * 2005-06-30 2012-01-10 Google Inc. User-friendly features for real-time communications

Family Cites Families (65)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6078953A (en) * 1997-12-29 2000-06-20 Ukiah Software, Inc. System and method for monitoring quality of service over network
US6993494B1 (en) * 1998-06-01 2006-01-31 Harrah's Operating Company, Inc. Resource price management incorporating indirect value
US7212978B2 (en) * 1998-06-01 2007-05-01 Harrah's Operating Company, Inc. Customer valuation in a resource price manager
US6370572B1 (en) * 1998-09-04 2002-04-09 Telefonaktiebolaget L M Ericsson (Publ) Performance management and control system for a distributed communications network
US7653002B2 (en) * 1998-12-24 2010-01-26 Verizon Business Global Llc Real time monitoring of perceived quality of packet voice transmission
US6584191B1 (en) * 1999-08-27 2003-06-24 Aspect Communications Corporation Staffing-based percentage-allocation routing using real-time data
IL130895A (en) * 1999-07-12 2003-10-31 Ectel Ltd Method and system for controlling quality of service over a telecommunication network
US6594277B1 (en) 1999-07-22 2003-07-15 Avaya Technology Corp. Dynamic-rate, differential class-based quality of service agent for internet protocol exchange systems
AU3642101A (en) 1999-11-03 2001-05-14 Michael Peroff Interactive web-based survey method and system
US6570855B1 (en) * 1999-12-30 2003-05-27 At&T Corp. Automatic call manager traffic gate feature
CA2303000A1 (en) * 2000-03-23 2001-09-23 William M. Snelgrove Establishing and managing communications over telecommunication networks
US7237138B2 (en) * 2000-05-05 2007-06-26 Computer Associates Think, Inc. Systems and methods for diagnosing faults in computer networks
US7500143B2 (en) * 2000-05-05 2009-03-03 Computer Associates Think, Inc. Systems and methods for managing and analyzing faults in computer networks
US7212988B1 (en) 2000-07-26 2007-05-01 Feldten Guy W Test screening of videos
US8650064B2 (en) * 2000-10-03 2014-02-11 Louis J. Morsberger System and method for collecting survey information from targeted consumers
US7222166B2 (en) * 2001-01-25 2007-05-22 Bandspeed, Inc. Approach for managing communications channels based on performance and transferring functions between participants in a communications arrangement
US20040073690A1 (en) * 2002-09-30 2004-04-15 Neil Hepworth Voice over IP endpoint call admission
US7746797B2 (en) 2002-10-09 2010-06-29 Nortel Networks Limited Non-intrusive monitoring of quality levels for voice communications over a packet-based network
US7860727B2 (en) * 2003-07-17 2010-12-28 Ventana Medical Systems, Inc. Laboratory instrumentation information management and control network
US20050064874A1 (en) * 2003-09-24 2005-03-24 Lucent Technologies Inc. System and method for brokering wireless communication resources
JP4012498B2 (en) * 2003-11-18 2007-11-21 株式会社日立製作所 Information processing system, information processing apparatus, information processing apparatus control method, and program
US7593586B2 (en) * 2004-06-30 2009-09-22 Aptina Imaging Corporation Method and system for reducing artifacts in image detection
US7185644B2 (en) * 2004-07-02 2007-03-06 Kurtz Jr Gerald Bow press having pivoted bow limb support arm
US8165109B2 (en) * 2004-11-10 2012-04-24 Cisco Technology, Inc. Method for managing the quality of encrypted voice over IP to teleagents
US20060111092A1 (en) * 2004-11-23 2006-05-25 Harris Doug S Alert management apparatus and method
US8438264B2 (en) * 2004-12-28 2013-05-07 At&T Intellectual Property I, L.P. Method and apparatus for collecting, analyzing, and presenting data in a communication network
US20060166669A1 (en) * 2005-01-27 2006-07-27 Holger Claussen Brokering services between wireless device users and operators
US7924732B2 (en) * 2005-04-19 2011-04-12 Hewlett-Packard Development Company, L.P. Quality of service in IT infrastructures
FI20050493A0 (en) * 2005-05-09 2005-05-09 Nokia Corp Connection quality monitoring
US8594311B2 (en) * 2005-06-02 2013-11-26 Virtual Hold Technology, Llc Expected wait time augmentation system and method
US7627093B1 (en) * 2005-12-30 2009-12-01 At&T Corp. Systems and methods for monitoring voice service feature failures in a telecommunication network
US20070233865A1 (en) * 2006-03-30 2007-10-04 Garbow Zachary A Dynamically Adjusting Operating Level of Server Processing Responsive to Detection of Failure at a Server
US8275377B2 (en) 2006-04-20 2012-09-25 Qualcomm Incorporated Wireless handoffs between multiple networks
US20070286351A1 (en) * 2006-05-23 2007-12-13 Cisco Technology, Inc. Method and System for Adaptive Media Quality Monitoring
US8457000B2 (en) * 2006-05-26 2013-06-04 Communications Acquistions, LLC Call quality monitoring
CN2932712Y (en) * 2006-07-07 2007-08-08 富士康(昆山)电脑接插件有限公司 Electric connector
US8493858B2 (en) * 2006-08-22 2013-07-23 Citrix Systems, Inc Systems and methods for providing dynamic connection spillover among virtual servers
WO2008066419A1 (en) * 2006-11-29 2008-06-05 Telefonaktiebolaget Lm Ericsson (Publ) A method and arrangement for controlling service level agreements in a mobile network.
GB0718980D0 (en) 2007-09-27 2007-11-07 Skype Ltd User interface
US7852784B2 (en) * 2008-02-11 2010-12-14 Microsoft Corporation Estimating endpoint performance in unified communication systems
US8503318B2 (en) * 2008-02-11 2013-08-06 Microsoft Corporation Estimating endpoint performance in unified communication systems
US8295191B2 (en) * 2008-03-04 2012-10-23 Microsoft Corporation Endpoint report aggregation in unified communication systems
US8189486B2 (en) * 2008-03-28 2012-05-29 Verizon Patent And Licensing Inc. Method and system for providing holistic, iterative, rule-based traffic management
US8560695B2 (en) * 2008-11-25 2013-10-15 Citrix Systems, Inc. Systems and methods for health based spillover
US9557889B2 (en) 2009-01-28 2017-01-31 Headwater Partners I Llc Service plan design, user interfaces, application programming interfaces, and device management
US8205116B2 (en) * 2009-02-18 2012-06-19 At&T Intellectual Property I, L.P. Common chronics resolution management
US8391456B2 (en) * 2009-03-20 2013-03-05 Microsoft Corporation Dynamic configuration of call controls for communication peripherals
US8837298B2 (en) * 2010-04-16 2014-09-16 Empirix, Inc. Voice quality probe for communication networks
US20110302247A1 (en) * 2010-06-02 2011-12-08 Microsoft Corporation Contextual information dependent modality selection
EP2583432B1 (en) * 2010-06-18 2019-02-20 Nokia Technologies Oy Method and apparatus for generating and handling streaming media quality-of-experience metrics
GB2481254B (en) 2010-06-18 2017-07-12 Skype Determining network quality
US20130219053A1 (en) * 2010-09-17 2013-08-22 Deutsche Telekom Ag Method for improved handling of incidents in a network monitoring system
US8620709B2 (en) * 2011-02-11 2013-12-31 Avaya, Inc Mobile activity manager
US8542810B2 (en) * 2011-03-21 2013-09-24 Microsoft Corporation Automatic rejoining of conferences
US9451651B2 (en) 2011-10-12 2016-09-20 At&T Mobility Ii Llc Management of multiple radio access bearer sessions in communication system
US9860296B2 (en) * 2012-03-23 2018-01-02 Avaya Inc. System and method for end-to-end call quality indication
US8854954B2 (en) * 2012-04-24 2014-10-07 International Businesss Machines Corporation Quality of service prediction and call failover
JP6036827B2 (en) * 2012-08-02 2016-11-30 日本電気株式会社 Traffic data collection device, traffic data collection method, and program
US9189320B2 (en) * 2012-08-15 2015-11-17 International Business Machines Corporation Handling intermittent recurring errors in a network
US8937870B1 (en) * 2012-09-11 2015-01-20 Amazon Technologies, Inc. Network link monitoring and testing
US20140229236A1 (en) * 2013-02-12 2014-08-14 Unify Square, Inc. User Survey Service for Unified Communications
US9118751B2 (en) 2013-03-15 2015-08-25 Marchex, Inc. System and method for analyzing and classifying calls without transcription
US9292402B2 (en) * 2013-04-15 2016-03-22 Century Link Intellectual Property LLC Autonomous service management
US9356962B2 (en) 2013-09-10 2016-05-31 Vmware, Inc. Extensible multi-tenant cloud-management system and methods for extending functionalities and services provided by a multi-tenant cloud-managment system
US9344340B2 (en) * 2013-11-18 2016-05-17 International Business Machines Corporation System and method for notification of QoE issues

Patent Citations (108)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4130881A (en) * 1971-07-21 1978-12-19 Searle Medidata, Inc. System and technique for automated medical history taking
US5227874A (en) * 1986-03-10 1993-07-13 Kohorn H Von Method for measuring the effectiveness of stimuli on decisions of shoppers
US5058152A (en) * 1989-12-12 1991-10-15 The Telephone Connection Anonymous interactive telephone system having direct connect feature
US5456607A (en) * 1989-12-13 1995-10-10 Antoniak; Peter R. Knowledge testing computer game method employing the repositioning of screen objects to represent data relationships
US5496175A (en) * 1991-02-01 1996-03-05 Hitachi, Ltd. Questionnaire system
US5788508A (en) * 1992-02-11 1998-08-04 John R. Lee Interactive computer aided natural learning method and apparatus
US5999908A (en) * 1992-08-06 1999-12-07 Abelow; Daniel H. Customer-based product design module
US20100070345A1 (en) * 1992-08-06 2010-03-18 Abelow Daniel H Customer-based product design module
US5802502A (en) * 1993-05-24 1998-09-01 British Telecommunications Public Limited Company System for selective communication connection based on transaction pricing signals
US5590057A (en) * 1993-12-20 1996-12-31 Atlantic Richfield Company Training and certification system and method
US5566291A (en) * 1993-12-23 1996-10-15 Diacom Technologies, Inc. Method and apparatus for implementing user feedback
US5754939A (en) * 1994-11-29 1998-05-19 Herz; Frederick S. M. System for generation of user profiles for a system for customized electronic identification of desirable objects
US6092049A (en) * 1995-06-30 2000-07-18 Microsoft Corporation Method and apparatus for efficiently recommending items using automated collaborative filtering and feature-guided automated collaborative filtering
US5791907A (en) * 1996-03-08 1998-08-11 Ramshaw; Bruce J. Interactive medical training system
US6070145A (en) * 1996-07-12 2000-05-30 The Npd Group, Inc. Respondent selection method for network-based survey
US6026387A (en) * 1996-07-15 2000-02-15 Kesel; Brad Consumer comment reporting apparatus and method
US5913204A (en) * 1996-08-06 1999-06-15 Kelly; Thomas L. Method and apparatus for surveying music listener opinion about songs
US5915243A (en) * 1996-08-29 1999-06-22 Smolen; Daniel T. Method and apparatus for delivering consumer promotions
US5940471A (en) * 1996-10-04 1999-08-17 Northern Telecom Limited Method and apparatus for obtaining feedback regarding network services
US20020128898A1 (en) * 1998-03-02 2002-09-12 Leroy Smith Dynamically assigning a survey to a respondent
US20010018673A1 (en) * 1998-03-12 2001-08-30 Steve Goldband Interactive customer support for computer programs using network connection of user machine
US6434532B2 (en) * 1998-03-12 2002-08-13 Aladdin Knowledge Systems, Ltd. Interactive customer support for computer programs using network connection of user machine
US6690919B1 (en) * 1998-05-05 2004-02-10 Mannesmann Ag Determining the quality of telecommunication services
US20040261125A1 (en) * 1998-08-27 2004-12-23 United Video Properties, Inc. Electronic program guide with interactive screen game
US6236975B1 (en) * 1998-09-29 2001-05-22 Ignite Sales, Inc. System and method for profiling customers for targeted marketing
US6597903B1 (en) * 1998-11-02 2003-07-22 Openwave Systems Inc. Online churn reduction and loyalty system
US6370120B1 (en) * 1998-12-24 2002-04-09 Mci Worldcom, Inc. Method and system for evaluating the quality of packet-switched voice signals
US6519571B1 (en) * 1999-05-27 2003-02-11 Accenture Llp Dynamic customer profile management
US6845485B1 (en) * 1999-07-15 2005-01-18 Hotv, Inc. Method and apparatus for indicating story-line changes by mining closed-caption-text
US20010003214A1 (en) * 1999-07-15 2001-06-07 Vijnan Shastri Method and apparatus for utilizing closed captioned (CC) text keywords or phrases for the purpose of automated searching of network-based resources for interactive links to universal resource locators (URL's)
US6421724B1 (en) * 1999-08-30 2002-07-16 Opinionlab, Inc. Web site response measurement tool
US20040199574A1 (en) * 1999-09-14 2004-10-07 Franco Louis M. System and method for delivering remotely stored applications and information
US20020052774A1 (en) * 1999-12-23 2002-05-02 Lance Parker Collecting and analyzing survey data
US6826540B1 (en) * 1999-12-29 2004-11-30 Virtual Personalities, Inc. Virtual human interface for conducting surveys
US6298119B1 (en) * 2000-01-11 2001-10-02 Siemens Information And Communication Networks, Inc. System and method for digital telephone trouble reporting
US20040044563A1 (en) * 2000-01-18 2004-03-04 Valuestar, Inc. System and method for real-time updating service provider ratings
US20030009759A1 (en) * 2000-01-19 2003-01-09 Denis Khoo Method and system for providing home shopping programs
US20010037206A1 (en) * 2000-03-02 2001-11-01 Vivonet, Inc. Method and system for automatically generating questions and receiving customer feedback for each transaction
US20040133463A1 (en) * 2000-03-06 2004-07-08 Theodore Benderev On-line survey method
US20020161779A1 (en) * 2000-03-07 2002-10-31 Brierley Harold M. Method and system for evaluating, reporting, and improving on-line promotion effectiveness
US20040006478A1 (en) * 2000-03-24 2004-01-08 Ahmet Alpdemir Voice-interactive marketplace providing promotion and promotion tracking, loyalty reward and redemption, and other features
US20010055750A1 (en) * 2000-04-10 2001-12-27 Rasche Jeanette D. Method and apparatus for educating asthma sufferers and caregivers
US20030163371A1 (en) * 2000-04-11 2003-08-28 Jeremy Beard System and method for presenting information over time to a user
US6766524B1 (en) * 2000-05-08 2004-07-20 Webtv Networks, Inc. System and method for encouraging viewers to watch television programs
US20020010628A1 (en) * 2000-05-24 2002-01-24 Alan Burns Method of advertising and polling
US20020120491A1 (en) * 2000-05-31 2002-08-29 Nelson Eugene C. Interactive survey and data management method and apparatus
US6606581B1 (en) * 2000-06-14 2003-08-12 Opinionlab, Inc. System and method for measuring and reporting user reactions to particular web pages of a website
US20020120501A1 (en) * 2000-07-19 2002-08-29 Bell Christopher Nathan Systems and processes for measuring, evaluating and reporting audience response to audio, video, and other content
US6631184B1 (en) * 2000-07-24 2003-10-07 Comverse Ltd. System for community generated feedback and/or rating
US20020120504A1 (en) * 2000-07-31 2002-08-29 Intermedia Advertising Group Computerized system and method for increasing the effectiveness of advertising
US20050075919A1 (en) * 2000-08-23 2005-04-07 Jeong-Uk Kim Method for respondent-based real-time survey
US20020029381A1 (en) * 2000-09-06 2002-03-07 Eric Inselberg Method and apparatus for interactive audience participation at a live spectator event
US20020049635A1 (en) * 2000-09-06 2002-04-25 Khanh Mai Multiple advertising
US6581071B1 (en) * 2000-09-12 2003-06-17 Survivors Of The Shoah Visual History Foundation Surveying system and method
US6778807B1 (en) * 2000-09-15 2004-08-17 Documus, Llc Method and apparatus for market research using education courses and related information
US20030195807A1 (en) * 2000-10-12 2003-10-16 Frank S. Maggio Method and system for verifying exposure to message content via a printed response
US20040103032A1 (en) * 2000-10-12 2004-05-27 Maggio Frank S. Remote control system and method for interacting with broadcast content
US20040015399A1 (en) * 2000-10-12 2004-01-22 Maggio Frank S. Method and system for verifying exposure to message content delivered via outdoor media or in a concentrated format
US6636590B1 (en) * 2000-10-30 2003-10-21 Ingenio, Inc. Apparatus and method for specifying and obtaining services through voice commands
US20030101088A1 (en) * 2000-11-27 2003-05-29 Suriyan Lohavichan Web-based survey method for measuring customer service response
US20030088452A1 (en) * 2001-01-19 2003-05-08 Kelly Kevin James Survey methods for handheld computers
US20020103696A1 (en) * 2001-01-29 2002-08-01 Huang Jong S. System and method for high-density interactive voting using a computer network
US20030105660A1 (en) * 2001-02-20 2003-06-05 Walsh Kenneth Peter Method of relating multiple independent databases
US20030004778A1 (en) * 2001-05-10 2003-01-02 Gareau Brian R. Computer architecture and computer iplemented and/or assisted method of performing a cultural assessment of an organization and making improvements thereon
US20040235460A1 (en) * 2001-05-11 2004-11-25 Engstrom G. Eric Method and system for providing an opinion and aggregating opinions with mobile telecommunication device
US20040236625A1 (en) * 2001-06-08 2004-11-25 Kearon John Victor Method apparatus and computer program for generating and evaluating feelback from a plurality of respondents
US20030018517A1 (en) * 2001-07-20 2003-01-23 Dull Stephen F. Providing marketing decision support
US20030055722A1 (en) * 2001-09-19 2003-03-20 Jagtec, Inc. Method and apparatus for control of advertisements
US20030070180A1 (en) * 2001-09-28 2003-04-10 Toshio Katayama System for assisting consideration of selection
US20050054286A1 (en) * 2001-10-15 2005-03-10 Jawahar Kanjilal Method of providing live feedback
US20030078804A1 (en) * 2001-10-24 2003-04-24 Palmer Morrel-Samuels Employee assessment tool
US20030115216A1 (en) * 2001-12-19 2003-06-19 First Data Corporation Methods and systems for developing market intelligence
US20030177061A1 (en) * 2002-02-22 2003-09-18 Spady Richard J. Method for presenting opinions and measuring "social (intangible) assets"
US20030182135A1 (en) * 2002-03-21 2003-09-25 Masahiro Sone System and method for customer satisfaction survey and analysis for off-site customer service
US20030200135A1 (en) * 2002-04-19 2003-10-23 Wright Christine Ellen System and method for predicting and preventing customer churn
US20030227870A1 (en) * 2002-06-03 2003-12-11 Wagner Clinton Allen Method and system for automated voice quality statistics gathering
US20030229533A1 (en) * 2002-06-06 2003-12-11 Mack Mary E. System and method for creating compiled marketing research data over a computer network
US20040019688A1 (en) * 2002-07-29 2004-01-29 Opinionlab Providing substantially real-time access to collected information concerning user interaction with a web page of a website
US20040030697A1 (en) * 2002-07-31 2004-02-12 American Management Systems, Inc. System and method for online feedback
US20040059625A1 (en) * 2002-09-20 2004-03-25 Ncr Corporation Method for providing feedback to advertising on interactive channels
US20040073641A1 (en) * 2002-09-30 2004-04-15 Muneyb Minhazuddin Instantaneous user initiation voice quality feedback
US20040073476A1 (en) * 2002-10-10 2004-04-15 Prolink Services Llc Method and system for identifying key opinion leaders
US20040220893A1 (en) * 2002-11-20 2004-11-04 Radar Networks, Inc. User interface for managing semantic objects
US20040158455A1 (en) * 2002-11-20 2004-08-12 Radar Networks, Inc. Methods and systems for managing entities in a computing device using semantic objects
US20040230676A1 (en) * 2002-11-20 2004-11-18 Radar Networks, Inc. Methods and systems for managing offers and requests in a network
US20040123314A1 (en) * 2002-12-23 2004-06-24 Bova Alfred T. Method and system for integrating television brand advertising with promotional marketing
US20040143498A1 (en) * 2002-12-27 2004-07-22 Toshihiko Umeda Service supporting system, service supporting server and service supporting method
US20040210820A1 (en) * 2002-12-27 2004-10-21 Douglas Tarr Automated compensation reports using online surveys and collaborative filtering
US20040143478A1 (en) * 2003-01-18 2004-07-22 Ward Andrew David Method and process for capuring, storing, processing and displaying customer satisfaction information
US20040172323A1 (en) * 2003-02-28 2004-09-02 Bellsouth Intellectual Property Corporation Customer feedback method and system
US20040181448A1 (en) * 2003-03-14 2004-09-16 Paul Hartsman Marketing network
US20040249712A1 (en) * 2003-06-06 2004-12-09 Brown Sean D. System, method and computer program product for presenting, redeeming and managing incentives
US20050055275A1 (en) * 2003-06-10 2005-03-10 Newman Alan B. System and method for analyzing marketing efforts
US20040252816A1 (en) * 2003-06-13 2004-12-16 Christophe Nicolas Mobile phone sample survey method
US20050091077A1 (en) * 2003-08-25 2005-04-28 Reynolds Thomas J. Determining strategies for increasing loyalty of a population to an entity
US20050055266A1 (en) * 2003-09-05 2005-03-10 Pitney Bowes Incorporated Method and system for generating information about relationships between an enterprise and other parties and sharing such information among users in the enterprise
US20050065851A1 (en) * 2003-09-22 2005-03-24 Aronoff Jeffrey M. System, method and computer program product for supplying to and collecting information from individuals
US20050096943A1 (en) * 2003-11-05 2005-05-05 Siegalovsky Ilene L. System and method for correlating market research data based on sales representative activity
US20050114171A1 (en) * 2003-11-05 2005-05-26 Siegalovsky Ilene L. System and method for correlating market research data based on attitude information
US20050119931A1 (en) * 2003-12-01 2005-06-02 Matthew Schall Disparate survey score conversion and comparison method
US20060240803A1 (en) * 2005-04-26 2006-10-26 Xerox Corporation Automated notification systems and methods
US20060291641A1 (en) * 2005-06-02 2006-12-28 Lucent Technologies Inc. Methods and systems for selective threshold based call blocking
US8095665B1 (en) * 2005-06-30 2012-01-10 Google Inc. User-friendly features for real-time communications
US20080182555A1 (en) * 2006-12-08 2008-07-31 Rodrigo Madanes Communication system
US20080291896A1 (en) * 2007-03-28 2008-11-27 Tauri Tuubel Detection of communication states
US20090237240A1 (en) * 2008-03-19 2009-09-24 Microsoft Corporation Determining quality monitoring alerts in unified communication systems
US20110009707A1 (en) * 2008-05-07 2011-01-13 Kaundinya Murali P Telehealth Scheduling and Communications Network
US20110137696A1 (en) * 2009-12-04 2011-06-09 3Pd Performing follow-up actions based on survey results

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
What is unified communications - A Word Definition From the Webopedia Computer Dictionary, archives org, June 23rd 2011https://web.archive.org/web/20110623065101/http://www.webopedia.com/TERM/U/unified_communications.html *
XMPP - Wikipedia, the free encyclopedia, January 2013https://web.archive.org/web/20130131201539/http://en.wikipedia.org/wiki/XMPP *

Cited By (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9185589B2 (en) 2009-08-31 2015-11-10 The Nielsen Company (Us), Llc Methods and apparatus to identify wireless carrier performance effects
US9848089B2 (en) 2014-11-05 2017-12-19 The Nielsen Company (Us), Llc Methods and apparatus to generate an overall performance index
US11337084B2 (en) 2015-09-29 2022-05-17 Soracom, Inc. Control apparatus for gateway in mobile communication system
US11595830B2 (en) 2015-09-29 2023-02-28 Soracom, Inc. Control apparatus for gateway in mobile communication system
US11805429B2 (en) 2015-09-29 2023-10-31 Soracom, Inc. Control apparatus for gateway in mobile communication system
US10558988B2 (en) 2016-05-09 2020-02-11 International Business Machines Corporation Survey based on user behavior pattern
US10726432B2 (en) 2016-05-09 2020-07-28 International Business Machines Corporation Survey based on user behavior pattern
US20200344115A1 (en) * 2019-04-25 2020-10-29 Elo Touch Solutions, Inc. Zero touch deployment and dynamic configuration
US11652689B2 (en) * 2019-04-25 2023-05-16 Elo Touch Solutions, Inc. Zero touch deployment and dynamic configuration
CN115086488A (en) * 2022-07-27 2022-09-20 广东创新科技职业学院 Number classification method and device

Also Published As

Publication number Publication date
US20180084107A1 (en) 2018-03-22
EP2957119A4 (en) 2016-10-26
WO2014127055A2 (en) 2014-08-21
US9503570B2 (en) 2016-11-22
US20140226799A1 (en) 2014-08-14
US20140229614A1 (en) 2014-08-14
US20140226800A1 (en) 2014-08-14
WO2014127055A3 (en) 2015-05-21
US9860368B2 (en) 2018-01-02
US10674007B2 (en) 2020-06-02
US20170149628A1 (en) 2017-05-25
US20150373186A1 (en) 2015-12-24
US9071677B2 (en) 2015-06-30
EP2957119A2 (en) 2015-12-23

Similar Documents

Publication Publication Date Title
US10674007B2 (en) Enhanced data capture, analysis, and reporting for unified communications
US10142391B1 (en) Systems and methods of diagnosing down-layer performance problems via multi-stream performance patternization
US20190073092A1 (en) System and Method of Communication Analysis
US10218588B1 (en) Systems and methods for multi-stream performance patternization and optimization of virtual meetings
US10453012B2 (en) Evaluating communications feature utilization
US10157358B1 (en) Systems and methods for multi-stream performance patternization and interval-based prediction
US10346221B2 (en) Determining life-cycle of task flow performance for telecommunication service order
US20150134404A1 (en) Weighted promoter score analytics system and methods
US8989357B2 (en) Method and inference engine for processing telephone communication data
US20150304372A1 (en) Systems for Providing Services in a Voice Conferencing Environment
US20150278718A1 (en) Systems and methods for communication sharing in a relationship management system
US9148523B2 (en) Method and apparatus for managing a network
US8578211B2 (en) Real time error detection in multimodal communication systems
US20190355015A1 (en) Most influential customer scoring
US20070166685A1 (en) Automated skills assessment
US11157856B2 (en) Systems and methods for quality management system deployment
US20160269258A1 (en) System and method for managing communication sessions in an enterprise
US11893504B2 (en) Enterprise communication channel assistance
Gibeli et al. Construction of baselines for VoIP traffic management on open MANs
Ilag et al. Teams Call Quality Dashboard for Call Quality Troubleshooting
US20230379229A1 (en) Artificial intelligence based service quality response system
Ilag et al. Teams Reporting and Monitoring
Suciu et al. Elastic disaster early warning system using a cloud-based communication center
Ogbo The Impact of Access to Over-the-Top Services on Preferences for Mobile Services: A Conjoint Analysis of Urban and Rural Mobile Users in Nigeria
US20140310057A1 (en) Management of subscribers of a mobile communications network

Legal Events

Date Code Title Description
AS Assignment

Owner name: VENTURE LENDING & LEASING VII, INC., CALIFORNIA

Free format text: SECURITY INTEREST;ASSIGNOR:UNIFY SQUARE, INC.;REEL/FRAME:033663/0492

Effective date: 20140807

AS Assignment

Owner name: UNIFY SQUARE, INC., WASHINGTON

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:AGGARWAL, SUDHANSHU;RAGHAVAN, ARUN;OSBORNE, ROBERT;REEL/FRAME:034848/0778

Effective date: 20150128

AS Assignment

Owner name: UNIFY SQUARE, INC., WASHINGTON

Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:VENTURE LENDING & LEASING VII, INC.;REEL/FRAME:036067/0441

Effective date: 20150713

Owner name: WESTERN ALLIANCE BANK, CALIFORNIA

Free format text: SECURITY INTEREST;ASSIGNOR:UNIFY SQUARE, INC.;REEL/FRAME:036073/0469

Effective date: 20150709

STCB Information on status: application discontinuation

Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION

AS Assignment

Owner name: UNIFY SQUARE, INC., WASHINGTON

Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:WESTERN ALLIANCE BANK;REEL/FRAME:056443/0594

Effective date: 20210604