US20150286478A1 - Application Version Release Management - Google Patents
Application Version Release Management Download PDFInfo
- Publication number
- US20150286478A1 US20150286478A1 US14/741,525 US201514741525A US2015286478A1 US 20150286478 A1 US20150286478 A1 US 20150286478A1 US 201514741525 A US201514741525 A US 201514741525A US 2015286478 A1 US2015286478 A1 US 2015286478A1
- Authority
- US
- United States
- Prior art keywords
- application
- version
- client devices
- users
- performance
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/70—Software maintenance or management
- G06F8/71—Version control; Configuration management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3409—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment for performance assessment
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
- G06F11/3668—Software testing
- G06F11/3672—Test management
- G06F11/3688—Test management for test execution, e.g. scheduling of test suites
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/60—Software deployment
- G06F8/61—Installation
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
Definitions
- a method may include providing a first version of an application to a first subset of users of the application. Performance of the first version of the application may be measured. Next, a second subset of users of the application may be selected based on the performance of the first version of the application. The second subset of users may be larger than the first subset of users. A second version of the application may be provided to the second subset of users of the application. Performance of the second version of the application may be measured, and a third subset of users of the application may be selected based on the performance of the second version of the application. The third subset of users may include both the first subset of users and the second subset of users.
- a method may include providing a first version of an application to a first subset of users of the application. Next, performance of the first version of the application may be measured and a second subset of users of the application may be selected based on the performance of the first version of the application. As a result, a second version of the application may be provided to the second subset of users of the application.
- An implementation of the present disclosure provides a system including a processor configured to provide a first version of an application to a first subset of users of the application. The performance of the first version of the application may be measured. Next, a second subset of users of the application may be selected based on the performance of the first version of the application. As a result, a second version of the application may be provided to the second subset of users of the application.
- implementations of the disclosed subject matter may provide a developer of an application with the ability to release a version of an application, e.g., a test version, an updated version, and the like, to a subset of users (e.g., 5% of users) rather than the general public.
- Performance information associated with the application may be collected and this information may be used to select another subset of users (e.g., 25% of users, 100% of users, etc.) to whom the application may be provided. This may improve a developer's ability to test and improve the quality of an application prior to releasing an application to the general public.
- the developer may be able to test the stability of the application by incrementally releasing the application to users (e.g., 5% of users then 25% of users, then 80% of users, then 100% of users), verifying that the application server is stable under an increasing load of users.
- the disclosed subject matter may allow for improved application quality and user satisfaction. Additional features, advantages, and embodiments of the disclosed subject matter may be set forth or apparent from consideration of the following detailed description, drawings, and claims. Moreover, it is to be understood that both the foregoing summary and the following detailed description are examples and are intended to provide further explanation without limiting the scope of the claims.
- FIG. 1 shows an example process according to an implementation of the disclosed subject matter.
- FIG. 2 shows an example information flow according to an implementation of the disclosed subject matter.
- FIG. 3 shows a computer according to an embodiment of the disclosed subject matter.
- FIG. 4 shows a network configuration according to an embodiment of the disclosed subject matter.
- the present disclosure allows a developer to select or leverage a pre-existing community or population of users who may receive a pre-release version of their application.
- a user may opt into receiving a pre-release version of the application.
- an application developer may wish to launch their new application to some private subset of users or a well-established developer may want to roll out a new version their application out to a small portion of their existing users.
- a developer may be able to observe the application's performance and correct/fix any major performance issues reported (by, for example, 5% of their users) before releasing the version to the general public. This may help to improve the quality of applications and help developers maintain their reputation and a strong rating for their application by giving them the ability to correct/fix performance issues prior to a major public release.
- the present disclosure may reduce testing and development costs by alleviating and addressing application performance problems earlier before releasing an application to the public.
- the present disclosure provides a mechanism for distributing a specific version of an application to one or more subsets of users, a pre-existing community of users, a domain membership of users, a stable pre-release group of users, and the like.
- a developer configuration system may allow developers to configure various configurations of their application's packages to be distributed to a subset of users. Users may be given a user-facing opt-in mechanism, allowing authorized users to become testers of a given application. Specifically, user opt-ins may be recorded and membership in relevant opt-in groups may be confirmed when determining the version of an application to be delivered to a specific subset of users.
- social network communities may be utilized such that developers may maintain a community of test users. Additionally, selection of a stable subset of users for the lifetime of an application rollout (which may be modified with new application versions mid-rollout) may be achieved by recording a random seed to be used for identifying particular users until the rollout is complete.
- a startup game developer may be in the process of building a new application, Grumpy Otters, and they may wish to pre-release the application to a subset of alpha testers.
- Alpha testers may be invited to join the developer's private invite-only social network community.
- An alpha version of the Grumpy Otters application may be configured for distribution to this community, and the alpha testers may be asked to visit a site to opt-in for the testing of the alpha version of the application.
- these alpha testers may able to install the application and may be able send their feedback and comments regarding the application to the developer and/or application provider.
- the Grumpy Otters developer may wish to offer a press-only pre-release of a more stable beta version of the application.
- a process similar to the one above may be followed.
- This beta version release of the application to the press-only subset of users may occur simultaneously as the alpha version test release to the alpha user group described above.
- the developer may receive feedback and comments from both the alpha testers regarding the alpha version and the press-only testers regarding the beta version.
- the developers may be ready to begin testing of the newest version 2 of the application.
- the developer may wish to initiate a testing rollout, which is provided only to a subset of potential recipients of the application. For example, for rollout of version 2, a random 5% of the application user population (who have opted in to such rollout testing) may be selected. Based on this rollout, the developer may become aware of some performance issues of version 2 of the application. The developer may correct/fix these performance issues and provide an update in the form of version 2.1 of the application to the exact same 5% of users who previously received version 2 of the application.
- version 2.1 of the application may be released to 100% of the user population, thereby launching the newest version 2.1 of the application.
- FIG. 1 shows an example process according to an implementation of the disclosed subject matter.
- a method may include providing a first version of an application to a first subset of users of the application, at 101 . Performance of the first version of the application may be measured, at 102 . Next, a second subset of users of the application may be selected based on the performance of the first version of the application, at 103 . As a result, a second version of the application may be provided to the second subset of users of the application, at 104 .
- a version of an application may be a new version of the application, an updated version of the application, a beta version of the application, a test version of the application, an unstable version of the application, a last known working version of the application, a previous version of the application, and the like.
- An application may be created and/or developed based on an application file format.
- An application file format can be a file format used to distribute and/or install application software and middleware onto a platform with a given operating system.
- a version may comprise multiple application file formats of an application, and various application file formats may be provided to a relevant set of devices.
- a version may comprise multiple files, each file having the same file format and each including different content.
- An application developer may specifically configure a version of an application for distribution to a particular subset of users. For example, a developer may configure an alpha version of an application to be provided to the subset of users who have opted in to alpha testing of the developer's applications, the same configuration may apply for a beta version of an application to be provided to a subset of users who have opted in to beta testing.
- the second version of an application may or may not replace the first version of the application on a user's device. For example, a last known working version, a stable version, or an updated version of an application may be automatically provided to users in a subset who experienced performance issues with a first version of the application.
- the first subset of users may have version 1 of an application and simultaneously the second subset of users may have version 2 of the application.
- the first version of the application may be the same as the second version of the application.
- the developer of version 1.0 of a recipe application may wish to test a new version 2.0 of the recipe application.
- New version 2.0 may be provided to 5% of users of version 1.0, who have opted-in to testing of the recipe application.
- Performance of version 2.0 of the recipe application may be measured from the 5% of users, and it may be determined that version 2.0 is not experiencing any performance issues. Because version 2.0 is not experiencing any performance issues, it may be provided to 100% of the users of version 1.0.
- the first and second versions of the application are the same, that is, version 2.0 of the application.
- an application developer or an application provider may create and/or manage one or more communities or subsets of users.
- Users of an application may be a population of users of an application and may be defined in a variety of ways.
- Providing an application to a user may encompass providing the application to one, some, or all the devices associated with a user.
- a population of users may be all users of an application, all users of a specific version of an application, all users of an application associated with a particular demographic, all users who have opted-in to receive a version of an application, and the like.
- a demographic of users may refer to any segment of the population using an application and may refer to users associated with a device, a device manufacturer, OS version, locale, carrier, cellular standard, geographic region, and any other demographic using a version of an application.
- a demographic may be based on the particular device running an application such as Smartphone 4 devices, a specific brand of devices, the OS version on the device and/or any other device characteristic that may affect application performance.
- a demographic may be based on a specific wireless carrier of a device and/or a cellular standard offered by a wireless carrier, such as, CDMA, GSM, 3G, 4G, 4G LTE, and the like.
- a demographic may also be based on the locale and/or geographic region associated with a device and/or user.
- a locale or geographic region may refer to a zip code, area code, city, county, state, country, continent, and/or a region of the world.
- a demographic may include any combination of the above-described characteristics of a device, a carrier, and location and may further include any other characteristics for defining a population and/or subset of application users.
- users may opt-in to receive a version of an application.
- a subset of users of an application may be defined as all the users of an application, a set number of users, a percentage of users ranging from greater than about 0% to about 100%, a fraction of users, a random selection of users, and any other technique for selecting a portion of users.
- the first subset of users may be the same as the second subset of users.
- a new version 2.0 of an application may be provided to 5% of users; this 5% of users may or may not be randomly selected.
- the developer may receive poor performance results for version 2.0 and the developer may develop a new improved beta version 3.0 which corrects the performance issues of version 2.0.
- the developer may provide the new improved beta version 3.0 to the same 5% of users who previously received version 2.0.
- the second subset of users may be randomly selected from among users of the application excluding those users in the first subset.
- the new improved beta version 3.0 may be provided to a random 5% of users, excluding those users who previously received version 2.0.
- the first subset of users may be different from the second subset of users.
- the second subset of users may be larger than the first subset of users or the first subset of users is larger than the second subset of users.
- the first subset of users may have been 5% of users and the developer may provide the new improved beta version 3.0 to 15% of users.
- the first subset of users may overlap with at least a portion of the second subset of users.
- the 15% of users may or may not include some or all of the 5% of user who previously received version 2.0.
- FIG. 2 shows an example information flow according to an implementation of the disclosed subject matter.
- a system may include a processor configured to provide a first version of an application to a first subset of users of the application.
- an application developer may provide a first version of an application to an application provider, at 201 , and the application provider may provide the first version of the application to a first subset of users of the application, at 202 .
- the application provider may receive performance criteria for the first version of the application from the first subset of users, at 203 , and the application provider may measure performance of the first version of the application, at 204 .
- the application provider may provide measured performance information of the first version of the application to the application developer, at 205 .
- Measured performance information may be a determination as to how the first version of the application is performing based on the performance criteria received from the first subset of users.
- the application provider may simply provide performance criteria to the application developer, and/or the application developer may measure performance of the first version of the application based on the received performance criteria.
- the application developer may receive performance criteria directly from the first subset of users.
- Measuring performance of the first version of the application may be performed according to various techniques. For example, measured performance of the first version of the application may be based on one or more performance criteria such as server stability based on the first subset of users, crash reports, activity not responding reports, performance testing reports, battery reports, running services reports, user initiated feedback reports, network reports, application revenue generation, application usage, bug reports, application ranking, number/quality of user ratings, and any other criteria indicating the performance of the first version of the application. Based on the performance criteria received for a first version of an application, a determination may be made as to the measured performance of the first version. The measured performance of the first version of the application may indicate either the presence of performance issues or no performance issues associated with the first version.
- performance criteria such as server stability based on the first subset of users, crash reports, activity not responding reports, performance testing reports, battery reports, running services reports, user initiated feedback reports, network reports, application revenue generation, application usage, bug reports, application ranking, number/quality of user ratings, and any other criteria indicating the performance of the first
- performance criteria may be received from all devices running version 6 of a social network application.
- the performance criteria may include activity not responding reports, user-initiated feedback reports, and performance testing reports from all devices. Based on the reports, it may be determined that version 6 of the social network application is experiencing activity not responding errors on all GSM devices in North America and poor performance test results on tablets in Japan.
- the second version of the application may correct a performance issue for a given demographic. Further, a second version may correct a performance issue by eliminating or reducing the effect of a performance issue. Accordingly, a new version 7 of the social network application may be provided to all GSM devices in North America and a test version 6T of the social network application may be provided to all tablet devices in Japan.
- the application provider may select a second subset of users of the application based on the performance of the first version of the application, at 206 , as shown in FIG. 2 . Selecting a second subset of users based on the performance of the first version of the application may be performed in a variety of ways. In some instances, if the measured performance of the first version of the application indicates no performance issues associated with the first version, the second subset of users may be larger than the first subset of users. For example, if the first version was provided to 10% of users and there were no performance issues associated with the first version, the first version and the second version may be same and the second version may be provided to 100% of users.
- the second subset of users may be selected according to various techniques. For example, the second subset may be randomly selected from among users of the application and may or may not include users in the first subset. In some cases, the second subset may intentionally exclude those users in the first subset, or those users in the first subset who received the poorly performing version. The second subset of users may be selected to include only users in the first subset for which the performance issue was detected, for example to correct the performance issue experienced by particular users in the first subset. In some cases, there may be users in the first subset who may have experienced a performance issue, but from whom performance criteria was not received, for whom the performance issue was not determined, etc.
- the second subset of users may be the same as the first subset of users, for example to correct the performance issue that was experienced by all users in the first subset.
- the second subset of users may be smaller than the first subset of users perhaps to test a new version of the application, for example the first subset of users may have been 100% of users and the second subset of users may be 5% of users of the application or 5% of users for which a performance issue was detected.
- at least a portion of the first subset of users may overlap with at least a portion of the second subset of users.
- the first subset of users may have been 20% of users and perhaps only 5% of those users experienced a performance issue
- the second subset of users may be 10% of users intentionally including some demographic-derived subset of the initial 20% of users, such as the 5% of users who experienced the performance issue.
- a portion or all of the second subset may be randomly selected and may or may not include users in the first subset, who may or may not have experienced a performance issue.
- the second subset of users may be selected to intentionally exclude users in the first subset.
- Several other techniques, not described herein, may be used for selecting the second subset of users.
- the application developer may provide a second version of the application to the application provider, at 207 , as shown in FIG. 2 .
- the application provider may provide the second version of the application to the second subset of users of the application, at 208 .
- the steps described above may be repeated such that a third subset of users of the application may be selected and a third version of the application may be provided to the third subset of users of the application.
- performance of the second version of the application may be measured and the third subset of users may be selected based on the performance of the second version of the application.
- the third subset of users may be selected based upon the performance of the first version of the application.
- each version of an application may be the same as another version, for example, any of the first, second, and/or third versions of an application may be the same.
- a developer may currently offer a weather application version 1.0 on the market and may wish to test out a new weather application version 2.0 prior to a release of version 2.0 to the general public.
- Version 2.0 of the weather application may be provided to a first subset of users, for example 5% of users of version 1.0.
- the performance of new version 2.0 may be measured based on performance criteria received from the 5% of users. The measured performance may indicate no performance issues associated with version 2.0.
- the developer Prior to releasing the well-performing version 2.0 to all users of version 1.0, the developer may wish to release version 2.0 incrementally to an increasing subset of users.
- the second subset of users may be 15% of users of version 1.0, and may include the first subset of 5% of users.
- This incremental release of version 2.0 may allow the developer to confirm the stability of servers, and the like, in view of the increased load of users. If there are no performance or stability issues associated with version 2.0 based on the 15% of users, version 2.0 may be provided to 30% of users, 50% of users, 75% of users, and so on, while measuring performance of version 2.0 and stability prior to releasing to each subsequent subset of users.
- the third subset of users may be selected to exclude users in the first subset, the second subset, or the first and second subsets.
- an application provider may be an application developer.
- the communication between users/devices, an application provider, and/or an application developer may be across one or more bridges between the interfaces.
- the communications between the devices and the application provider may be managed or assisted by a third device, such as, a coordinating device, a local coordinator, a remote server, etc.
- the third device may, for example, receive performance criteria from the devices and determine a performance issue relating to a first version of an application.
- the third device may then provide the performance issue to the application provider, in which case, the application provider may provide a second version of the application to the devices.
- the third device may receive performance criteria from the devices and provide the performance criteria to the application provider.
- the application provider may determine a performance issue based on the performance criteria relating to the first version of the application or provide the performance criteria to the application developer. Accordingly, the application provider or an application developer may provide a second version of the application to the devices.
- the third device may also act as a bridge between the application provider and the devices when providing the second version of the application to the devices.
- more than one intermediate device may be implemented to facilitate communication between devices, an application provider, and an application developer.
- the users may be provided with an opportunity to control whether programs or features collect user information (e.g., user's geographic location, user's applications, user's device type, user application purchase history, user location history, and other user information upon which selection of a user may be based), or to control whether and/or how to receive applications from the application provider and/or developer that may be more relevant to the user.
- user information e.g., user's geographic location, user's applications, user's device type, user application purchase history, user location history, and other user information upon which selection of a user may be based
- certain data may be treated in one or more ways before it is stored or used, so that personally identifiable information is removed.
- a user's identity may be treated so that no personally identifiable information can be determined for the user, or a user's geographic location may be generalized where location information is obtained (such as to a city, ZIP code, or state level), so that a particular location of a user cannot be determined.
- location information such as to a city, ZIP code, or state level
- the user may have control over how information is collected about the user and used by an application provider and/or application developer.
- FIG. 3 is an example computer 20 suitable for implementing embodiments of the presently disclosed subject matter.
- the computer 20 includes a bus 21 which interconnects major components of the computer 20 , such as a central processor 24 , a memory 27 (typically RAM, but which may also include ROM, flash RAM, or the like), an input/output controller 28 , a user display 22 , such as a display screen via a display adapter, a user input interface 26 , which may include one or more controllers and associated user input devices such as a keyboard, mouse, and the like, and may be closely coupled to the I/O controller 28 , fixed storage 23 , such as a hard drive, flash storage, Fibre Channel network, SAN device, SCSI device, and the like, and a removable media component 25 operative to control and receive an optical disk, flash drive, and the like.
- a bus 21 which interconnects major components of the computer 20 , such as a central processor 24 , a memory 27 (typically RAM, but which may also include ROM, flash
- the bus 21 allows data communication between the central processor 24 and the memory 27 , which may include read-only memory (ROM) or flash memory (neither shown), and random access memory (RAM) (not shown), as previously noted.
- the RAM is generally the main memory into which the operating system and application programs are loaded.
- the ROM or flash memory can contain, among other code, the Basic Input-Output system (BIOS) which controls basic hardware operation such as the interaction with peripheral components.
- BIOS Basic Input-Output system
- Applications resident with the computer 20 are generally stored on and accessed via a computer readable medium, such as a hard disk drive (e.g., fixed storage 23 ), an optical drive, floppy disk, or other storage medium 25 .
- a network interface 29 may provide a direct connection to a remote server via a telephone link, to the Internet via an internet service provider (ISP), or a direct connection to a remote server via a direct network link to the Internet via a POP (point of presence) or other technique.
- the network interface 29 may provide such connection using wireless techniques, including digital cellular telephone connection, Cellular Digital Packet Data (CDPD) connection, digital satellite data connection or the like.
- CDPD Cellular Digital Packet Data
- the network interface 29 may allow the computer to communicate with other computers via one or more local, wide-area, or other networks, as shown in FIG. 4 .
- FIG. 3 Many other devices or components (not shown) may be connected in a similar manner (e.g., document scanners, digital cameras and so on). Conversely, all of the components shown in FIG. 3 need not be present to practice the present disclosure. The components can be interconnected in different ways from that shown. The operation of a computer such as that shown in FIG. 3 is readily known in the art and is not discussed in detail in this application. Code to implement the present disclosure can be stored in computer-readable storage media such as one or more of the memory 27 , fixed storage 23 , removable media 25 , or on a remote storage location.
- FIG. 4 shows an example network arrangement according to an embodiment of the disclosed subject matter.
- One or more clients 10 , 11 such as local computers, smart phones, tablet computing devices, and the like may connect to other devices, such as application provider 12 , application developer 14 , and the like via one or more networks 7 .
- the network may be a local network, wide-area network, the Internet, or any other suitable communication network or networks, and may be implemented on any suitable platform including wired and/or wireless networks.
- the clients may communicate with one or more servers 13 , application provider 12 , application developer 14 , and/or databases 15 .
- the devices may be directly accessible by the clients 10 , 11 , or one or more other devices may provide intermediary access such as where a server 13 provides access to resources stored in a database 15 .
- the clients 10 , 11 also may access remote platforms 17 or services provided by remote platforms 17 such as cloud computing arrangements and services.
- the remote platform 17 may include one or more servers 13 , application provider 12 , application developer 14 , and/or databases 15
- various embodiments of the presently disclosed subject matter may include or be embodied in the form of computer-implemented processes and apparatuses for practicing those processes.
- Embodiments also may be embodied in the form of a computer program product having computer program code containing instructions embodied in non-transitory and/or tangible media, such as floppy diskettes, CD-ROMs, hard drives, USB (universal serial bus) drives, or any other machine readable storage medium, wherein, when the computer program code is loaded into and executed by a computer, the computer becomes an apparatus for practicing embodiments of the disclosed subject matter.
- Embodiments also may be embodied in the form of computer program code, for example, whether stored in a storage medium, loaded into and/or executed by a computer, or transmitted over some transmission medium, such as over electrical wiring or cabling, through fiber optics, or via electromagnetic radiation, wherein when the computer program code is loaded into and executed by a computer, the computer becomes an apparatus for practicing embodiments of the disclosed subject matter.
- the computer program code segments configure the microprocessor to create specific logic circuits.
- a set of computer-readable instructions stored on a computer-readable storage medium may be implemented by a general-purpose processor, which may transform the general-purpose processor or a device containing the general-purpose processor into a special-purpose device configured to implement or carry out the instructions.
- Embodiments may be implemented using hardware that may include a processor, such as a general purpose microprocessor and/or an Application Specific Integrated Circuit (ASIC) that embodies all or part of the techniques according to embodiments of the disclosed subject matter in hardware and/or firmware.
- the processor may be coupled to memory, such as RAM, ROM, flash memory, a hard disk or any other device capable of storing electronic information.
- the memory may store instructions adapted to be executed by the processor to perform the techniques according to embodiments of the disclosed subject matter.
Abstract
Description
- The market for applications is rapidly growing, but the ability to test applications prior to public release has lagged behind. A new application developer may launch their application to the general public and some users may experience performance issues associated with the application. Performance issues may also arise based on an unanticipated load on the application server when launching the application to the general public. The poor performance of the application may affect the ratings and reputation of the new developer, and may negatively impact revenue and popularity of the new developer and/or application. Additionally, a well-established developer may release a new updated version of their application to the population of existing users of the application. The new updated version of the application may cause performance issues, for example, that were not occurring in the previous version. This may result in a loss of users, negative feedback/reviews, and/or lost revenue.
- According to an implementation of the disclosed subject matter, a method may include providing a first version of an application to a first subset of users of the application. Performance of the first version of the application may be measured. Next, a second subset of users of the application may be selected based on the performance of the first version of the application. The second subset of users may be larger than the first subset of users. A second version of the application may be provided to the second subset of users of the application. Performance of the second version of the application may be measured, and a third subset of users of the application may be selected based on the performance of the second version of the application. The third subset of users may include both the first subset of users and the second subset of users.
- According to an implementation of the disclosed subject matter, a method may include providing a first version of an application to a first subset of users of the application. Next, performance of the first version of the application may be measured and a second subset of users of the application may be selected based on the performance of the first version of the application. As a result, a second version of the application may be provided to the second subset of users of the application.
- An implementation of the present disclosure provides a system including a processor configured to provide a first version of an application to a first subset of users of the application. The performance of the first version of the application may be measured. Next, a second subset of users of the application may be selected based on the performance of the first version of the application. As a result, a second version of the application may be provided to the second subset of users of the application.
- Prior to releasing an application to the general public, implementations of the disclosed subject matter may provide a developer of an application with the ability to release a version of an application, e.g., a test version, an updated version, and the like, to a subset of users (e.g., 5% of users) rather than the general public. Performance information associated with the application may be collected and this information may be used to select another subset of users (e.g., 25% of users, 100% of users, etc.) to whom the application may be provided. This may improve a developer's ability to test and improve the quality of an application prior to releasing an application to the general public. Additionally, the developer may be able to test the stability of the application by incrementally releasing the application to users (e.g., 5% of users then 25% of users, then 80% of users, then 100% of users), verifying that the application server is stable under an increasing load of users. Overall, the disclosed subject matter may allow for improved application quality and user satisfaction. Additional features, advantages, and embodiments of the disclosed subject matter may be set forth or apparent from consideration of the following detailed description, drawings, and claims. Moreover, it is to be understood that both the foregoing summary and the following detailed description are examples and are intended to provide further explanation without limiting the scope of the claims.
- The accompanying drawings, which are included to provide a further understanding of the disclosed subject matter, are incorporated in and constitute a part of this specification. The drawings also illustrate embodiments of the disclosed subject matter and together with the detailed description serve to explain the principles of embodiments of the disclosed subject matter. No attempt is made to show structural details in more detail than may be necessary for a fundamental understanding of the disclosed subject matter and various ways in which it may be practiced.
-
FIG. 1 shows an example process according to an implementation of the disclosed subject matter. -
FIG. 2 shows an example information flow according to an implementation of the disclosed subject matter. -
FIG. 3 shows a computer according to an embodiment of the disclosed subject matter. -
FIG. 4 shows a network configuration according to an embodiment of the disclosed subject matter. - The present disclosure allows a developer to select or leverage a pre-existing community or population of users who may receive a pre-release version of their application. A user may opt into receiving a pre-release version of the application. For example, an application developer may wish to launch their new application to some private subset of users or a well-established developer may want to roll out a new version their application out to a small portion of their existing users. As a result, a developer may be able to observe the application's performance and correct/fix any major performance issues reported (by, for example, 5% of their users) before releasing the version to the general public. This may help to improve the quality of applications and help developers maintain their reputation and a strong rating for their application by giving them the ability to correct/fix performance issues prior to a major public release. Additionally, the present disclosure may reduce testing and development costs by alleviating and addressing application performance problems earlier before releasing an application to the public.
- In general, the present disclosure provides a mechanism for distributing a specific version of an application to one or more subsets of users, a pre-existing community of users, a domain membership of users, a stable pre-release group of users, and the like. A developer configuration system may allow developers to configure various configurations of their application's packages to be distributed to a subset of users. Users may be given a user-facing opt-in mechanism, allowing authorized users to become testers of a given application. Specifically, user opt-ins may be recorded and membership in relevant opt-in groups may be confirmed when determining the version of an application to be delivered to a specific subset of users. In an implementation, social network communities may be utilized such that developers may maintain a community of test users. Additionally, selection of a stable subset of users for the lifetime of an application rollout (which may be modified with new application versions mid-rollout) may be achieved by recording a random seed to be used for identifying particular users until the rollout is complete.
- As an example, a startup game developer may be in the process of building a new application, Grumpy Otters, and they may wish to pre-release the application to a subset of alpha testers. Alpha testers may be invited to join the developer's private invite-only social network community. An alpha version of the Grumpy Otters application may be configured for distribution to this community, and the alpha testers may be asked to visit a site to opt-in for the testing of the alpha version of the application. Next, these alpha testers may able to install the application and may be able send their feedback and comments regarding the application to the developer and/or application provider.
- As another example, the Grumpy Otters developer may wish to offer a press-only pre-release of a more stable beta version of the application. A process similar to the one above may be followed. This beta version release of the application to the press-only subset of users may occur simultaneously as the alpha version test release to the alpha user group described above. In this case, the developer may receive feedback and comments from both the alpha testers regarding the alpha version and the press-only testers regarding the beta version.
- Furthermore, following the launch of version 1 of the Grumpy Otters application, the developers may be ready to begin testing of the newest version 2 of the application. The developer may wish to initiate a testing rollout, which is provided only to a subset of potential recipients of the application. For example, for rollout of version 2, a random 5% of the application user population (who have opted in to such rollout testing) may be selected. Based on this rollout, the developer may become aware of some performance issues of version 2 of the application. The developer may correct/fix these performance issues and provide an update in the form of version 2.1 of the application to the exact same 5% of users who previously received version 2 of the application. Once the developer is satisfied with performance of version 2.1, including feedback/comments from the 5% of users, the developer may decide to release version 2.1 of the application to increasing percentages of users, such as to 20% of users, and then 50%, in order to verify the stability of their servers under the new load of users. When the developer is ready, version 2.1 of the application may be released to 100% of the user population, thereby launching the newest version 2.1 of the application. The next time the developer wishes to initiate a rollout, they may be able to select a different 5% of the user population so that over time no one particular user consistently receives potentially crash-prone testing versions of an application during testing rollouts.
- Implementations of the disclosed subject matter provide systems and methods for providing various versions of an application to various subsets of users.
FIG. 1 shows an example process according to an implementation of the disclosed subject matter. A method may include providing a first version of an application to a first subset of users of the application, at 101. Performance of the first version of the application may be measured, at 102. Next, a second subset of users of the application may be selected based on the performance of the first version of the application, at 103. As a result, a second version of the application may be provided to the second subset of users of the application, at 104. - In general a version of an application may be a new version of the application, an updated version of the application, a beta version of the application, a test version of the application, an unstable version of the application, a last known working version of the application, a previous version of the application, and the like. An application may be created and/or developed based on an application file format. An application file format can be a file format used to distribute and/or install application software and middleware onto a platform with a given operating system. In some instances, a version may comprise multiple application file formats of an application, and various application file formats may be provided to a relevant set of devices. In other instances, a version may comprise multiple files, each file having the same file format and each including different content. An application developer may specifically configure a version of an application for distribution to a particular subset of users. For example, a developer may configure an alpha version of an application to be provided to the subset of users who have opted in to alpha testing of the developer's applications, the same configuration may apply for a beta version of an application to be provided to a subset of users who have opted in to beta testing. The second version of an application may or may not replace the first version of the application on a user's device. For example, a last known working version, a stable version, or an updated version of an application may be automatically provided to users in a subset who experienced performance issues with a first version of the application. In some instances, within an entire population of users of an application, there may be multiple subsets of users, each subset using different versions of the application at the same time. For example, in some scenarios, the first subset of users may have version 1 of an application and simultaneously the second subset of users may have version 2 of the application. In an implementation, the first version of the application may be the same as the second version of the application. For example, the developer of version 1.0 of a recipe application may wish to test a new version 2.0 of the recipe application. New version 2.0 may be provided to 5% of users of version 1.0, who have opted-in to testing of the recipe application. Performance of version 2.0 of the recipe application may be measured from the 5% of users, and it may be determined that version 2.0 is not experiencing any performance issues. Because version 2.0 is not experiencing any performance issues, it may be provided to 100% of the users of version 1.0. In this case, the first and second versions of the application are the same, that is, version 2.0 of the application.
- As discussed above, an application developer or an application provider may create and/or manage one or more communities or subsets of users. Users of an application may be a population of users of an application and may be defined in a variety of ways. Providing an application to a user may encompass providing the application to one, some, or all the devices associated with a user. A population of users may be all users of an application, all users of a specific version of an application, all users of an application associated with a particular demographic, all users who have opted-in to receive a version of an application, and the like. A demographic of users may refer to any segment of the population using an application and may refer to users associated with a device, a device manufacturer, OS version, locale, carrier, cellular standard, geographic region, and any other demographic using a version of an application. For example, a demographic may be based on the particular device running an application such as Smartphone 4 devices, a specific brand of devices, the OS version on the device and/or any other device characteristic that may affect application performance. As another example, a demographic may be based on a specific wireless carrier of a device and/or a cellular standard offered by a wireless carrier, such as, CDMA, GSM, 3G, 4G, 4G LTE, and the like. A demographic may also be based on the locale and/or geographic region associated with a device and/or user. For example, a locale or geographic region may refer to a zip code, area code, city, county, state, country, continent, and/or a region of the world. Further, a demographic may include any combination of the above-described characteristics of a device, a carrier, and location and may further include any other characteristics for defining a population and/or subset of application users.
- In an implementation, users may opt-in to receive a version of an application. Additionally, a subset of users of an application may be defined as all the users of an application, a set number of users, a percentage of users ranging from greater than about 0% to about 100%, a fraction of users, a random selection of users, and any other technique for selecting a portion of users. In some instances, the first subset of users may be the same as the second subset of users. For example, a new version 2.0 of an application may be provided to 5% of users; this 5% of users may or may not be randomly selected. The developer may receive poor performance results for version 2.0 and the developer may develop a new improved beta version 3.0 which corrects the performance issues of version 2.0. For testing purposes, the developer may provide the new improved beta version 3.0 to the same 5% of users who previously received version 2.0. In an implementation, the second subset of users may be randomly selected from among users of the application excluding those users in the first subset. For example, the new improved beta version 3.0 may be provided to a random 5% of users, excluding those users who previously received version 2.0. In another instance, the first subset of users may be different from the second subset of users. For example, the second subset of users may be larger than the first subset of users or the first subset of users is larger than the second subset of users. With regard to the example above, the first subset of users may have been 5% of users and the developer may provide the new improved beta version 3.0 to 15% of users. In this case, the first subset of users may overlap with at least a portion of the second subset of users. For example, the 15% of users may or may not include some or all of the 5% of user who previously received version 2.0.
- An implementation of the disclosed subject matter provides a system for providing various versions of an application to various subsets of users.
FIG. 2 shows an example information flow according to an implementation of the disclosed subject matter. A system may include a processor configured to provide a first version of an application to a first subset of users of the application. As shown, an application developer may provide a first version of an application to an application provider, at 201, and the application provider may provide the first version of the application to a first subset of users of the application, at 202. Next, the application provider may receive performance criteria for the first version of the application from the first subset of users, at 203, and the application provider may measure performance of the first version of the application, at 204. The application provider may provide measured performance information of the first version of the application to the application developer, at 205. Measured performance information may be a determination as to how the first version of the application is performing based on the performance criteria received from the first subset of users. Alternatively, the application provider may simply provide performance criteria to the application developer, and/or the application developer may measure performance of the first version of the application based on the received performance criteria. In an implementation, the application developer may receive performance criteria directly from the first subset of users. - Measuring performance of the first version of the application may be performed according to various techniques. For example, measured performance of the first version of the application may be based on one or more performance criteria such as server stability based on the first subset of users, crash reports, activity not responding reports, performance testing reports, battery reports, running services reports, user initiated feedback reports, network reports, application revenue generation, application usage, bug reports, application ranking, number/quality of user ratings, and any other criteria indicating the performance of the first version of the application. Based on the performance criteria received for a first version of an application, a determination may be made as to the measured performance of the first version. The measured performance of the first version of the application may indicate either the presence of performance issues or no performance issues associated with the first version. As an example, performance criteria may be received from all devices running version 6 of a social network application. The performance criteria may include activity not responding reports, user-initiated feedback reports, and performance testing reports from all devices. Based on the reports, it may be determined that version 6 of the social network application is experiencing activity not responding errors on all GSM devices in North America and poor performance test results on tablets in Japan. The second version of the application may correct a performance issue for a given demographic. Further, a second version may correct a performance issue by eliminating or reducing the effect of a performance issue. Accordingly, a
new version 7 of the social network application may be provided to all GSM devices in North America and a test version 6T of the social network application may be provided to all tablet devices in Japan. - Further, the application provider may select a second subset of users of the application based on the performance of the first version of the application, at 206, as shown in
FIG. 2 . Selecting a second subset of users based on the performance of the first version of the application may be performed in a variety of ways. In some instances, if the measured performance of the first version of the application indicates no performance issues associated with the first version, the second subset of users may be larger than the first subset of users. For example, if the first version was provided to 10% of users and there were no performance issues associated with the first version, the first version and the second version may be same and the second version may be provided to 100% of users. - In some scenarios, if the measured performance of the first version of the application indicates a performance issue associated with the first version, the second subset of users may be selected according to various techniques. For example, the second subset may be randomly selected from among users of the application and may or may not include users in the first subset. In some cases, the second subset may intentionally exclude those users in the first subset, or those users in the first subset who received the poorly performing version. The second subset of users may be selected to include only users in the first subset for which the performance issue was detected, for example to correct the performance issue experienced by particular users in the first subset. In some cases, there may be users in the first subset who may have experienced a performance issue, but from whom performance criteria was not received, for whom the performance issue was not determined, etc. Alternatively, the second subset of users may be the same as the first subset of users, for example to correct the performance issue that was experienced by all users in the first subset. The second subset of users may be smaller than the first subset of users perhaps to test a new version of the application, for example the first subset of users may have been 100% of users and the second subset of users may be 5% of users of the application or 5% of users for which a performance issue was detected. In some cases, at least a portion of the first subset of users may overlap with at least a portion of the second subset of users. For example, the first subset of users may have been 20% of users and perhaps only 5% of those users experienced a performance issue, the second subset of users may be 10% of users intentionally including some demographic-derived subset of the initial 20% of users, such as the 5% of users who experienced the performance issue. In some cases, a portion or all of the second subset may be randomly selected and may or may not include users in the first subset, who may or may not have experienced a performance issue. Alternatively, the second subset of users may be selected to intentionally exclude users in the first subset. Several other techniques, not described herein, may be used for selecting the second subset of users.
- Next, the application developer may provide a second version of the application to the application provider, at 207, as shown in
FIG. 2 . Subsequently, the application provider may provide the second version of the application to the second subset of users of the application, at 208. Additionally, the steps described above may be repeated such that a third subset of users of the application may be selected and a third version of the application may be provided to the third subset of users of the application. In some instances, performance of the second version of the application may be measured and the third subset of users may be selected based on the performance of the second version of the application. In addition, the third subset of users may be selected based upon the performance of the first version of the application. In an implementation, each version of an application may be the same as another version, for example, any of the first, second, and/or third versions of an application may be the same. As an example, a developer may currently offer a weather application version 1.0 on the market and may wish to test out a new weather application version 2.0 prior to a release of version 2.0 to the general public. Version 2.0 of the weather application may be provided to a first subset of users, for example 5% of users of version 1.0. The performance of new version 2.0 may be measured based on performance criteria received from the 5% of users. The measured performance may indicate no performance issues associated with version 2.0. Prior to releasing the well-performing version 2.0 to all users of version 1.0, the developer may wish to release version 2.0 incrementally to an increasing subset of users. For example, the second subset of users may be 15% of users of version 1.0, and may include the first subset of 5% of users. This incremental release of version 2.0 may allow the developer to confirm the stability of servers, and the like, in view of the increased load of users. If there are no performance or stability issues associated with version 2.0 based on the 15% of users, version 2.0 may be provided to 30% of users, 50% of users, 75% of users, and so on, while measuring performance of version 2.0 and stability prior to releasing to each subsequent subset of users. In some cases, the third subset of users may be selected to exclude users in the first subset, the second subset, or the first and second subsets. - Each of the steps described herein may be performed by an application provider, an application developer, a subset of users/devices (as appropriate), and/or an intermediary entity. Additionally, in some instances, an application provider may be an application developer. In an implementation, the communication between users/devices, an application provider, and/or an application developer may be across one or more bridges between the interfaces. For example, the communications between the devices and the application provider may be managed or assisted by a third device, such as, a coordinating device, a local coordinator, a remote server, etc. In such cases, the third device may, for example, receive performance criteria from the devices and determine a performance issue relating to a first version of an application. The third device may then provide the performance issue to the application provider, in which case, the application provider may provide a second version of the application to the devices. Alternatively, the third device may receive performance criteria from the devices and provide the performance criteria to the application provider. The application provider may determine a performance issue based on the performance criteria relating to the first version of the application or provide the performance criteria to the application developer. Accordingly, the application provider or an application developer may provide a second version of the application to the devices. In an embodiment, the third device may also act as a bridge between the application provider and the devices when providing the second version of the application to the devices. Furthermore, more than one intermediate device may be implemented to facilitate communication between devices, an application provider, and an application developer.
- In situations in which the implementations of the disclosed subject matter collect personal information about users, or may make use of personal information, the users may be provided with an opportunity to control whether programs or features collect user information (e.g., user's geographic location, user's applications, user's device type, user application purchase history, user location history, and other user information upon which selection of a user may be based), or to control whether and/or how to receive applications from the application provider and/or developer that may be more relevant to the user. In addition, certain data may be treated in one or more ways before it is stored or used, so that personally identifiable information is removed. For example, a user's identity may be treated so that no personally identifiable information can be determined for the user, or a user's geographic location may be generalized where location information is obtained (such as to a city, ZIP code, or state level), so that a particular location of a user cannot be determined. Thus, the user may have control over how information is collected about the user and used by an application provider and/or application developer.
- Embodiments of the presently disclosed subject matter may be implemented in and used with a variety of component and network architectures.
FIG. 3 is anexample computer 20 suitable for implementing embodiments of the presently disclosed subject matter. Thecomputer 20 includes abus 21 which interconnects major components of thecomputer 20, such as acentral processor 24, a memory 27 (typically RAM, but which may also include ROM, flash RAM, or the like), an input/output controller 28, auser display 22, such as a display screen via a display adapter, a user input interface 26, which may include one or more controllers and associated user input devices such as a keyboard, mouse, and the like, and may be closely coupled to the I/O controller 28, fixedstorage 23, such as a hard drive, flash storage, Fibre Channel network, SAN device, SCSI device, and the like, and aremovable media component 25 operative to control and receive an optical disk, flash drive, and the like. - The
bus 21 allows data communication between thecentral processor 24 and thememory 27, which may include read-only memory (ROM) or flash memory (neither shown), and random access memory (RAM) (not shown), as previously noted. The RAM is generally the main memory into which the operating system and application programs are loaded. The ROM or flash memory can contain, among other code, the Basic Input-Output system (BIOS) which controls basic hardware operation such as the interaction with peripheral components. Applications resident with thecomputer 20 are generally stored on and accessed via a computer readable medium, such as a hard disk drive (e.g., fixed storage 23), an optical drive, floppy disk, orother storage medium 25. - The fixed
storage 23 may be integral with thecomputer 20 or may be separate and accessed through other interfaces. Anetwork interface 29 may provide a direct connection to a remote server via a telephone link, to the Internet via an internet service provider (ISP), or a direct connection to a remote server via a direct network link to the Internet via a POP (point of presence) or other technique. Thenetwork interface 29 may provide such connection using wireless techniques, including digital cellular telephone connection, Cellular Digital Packet Data (CDPD) connection, digital satellite data connection or the like. For example, thenetwork interface 29 may allow the computer to communicate with other computers via one or more local, wide-area, or other networks, as shown inFIG. 4 . - Many other devices or components (not shown) may be connected in a similar manner (e.g., document scanners, digital cameras and so on). Conversely, all of the components shown in
FIG. 3 need not be present to practice the present disclosure. The components can be interconnected in different ways from that shown. The operation of a computer such as that shown inFIG. 3 is readily known in the art and is not discussed in detail in this application. Code to implement the present disclosure can be stored in computer-readable storage media such as one or more of thememory 27, fixedstorage 23,removable media 25, or on a remote storage location. -
FIG. 4 shows an example network arrangement according to an embodiment of the disclosed subject matter. One ormore clients application provider 12,application developer 14, and the like via one ormore networks 7. The network may be a local network, wide-area network, the Internet, or any other suitable communication network or networks, and may be implemented on any suitable platform including wired and/or wireless networks. The clients may communicate with one ormore servers 13,application provider 12,application developer 14, and/ordatabases 15. The devices may be directly accessible by theclients server 13 provides access to resources stored in adatabase 15. Theclients remote platforms 17 or services provided byremote platforms 17 such as cloud computing arrangements and services. Theremote platform 17 may include one ormore servers 13,application provider 12,application developer 14, and/ordatabases 15. - More generally, various embodiments of the presently disclosed subject matter may include or be embodied in the form of computer-implemented processes and apparatuses for practicing those processes. Embodiments also may be embodied in the form of a computer program product having computer program code containing instructions embodied in non-transitory and/or tangible media, such as floppy diskettes, CD-ROMs, hard drives, USB (universal serial bus) drives, or any other machine readable storage medium, wherein, when the computer program code is loaded into and executed by a computer, the computer becomes an apparatus for practicing embodiments of the disclosed subject matter. Embodiments also may be embodied in the form of computer program code, for example, whether stored in a storage medium, loaded into and/or executed by a computer, or transmitted over some transmission medium, such as over electrical wiring or cabling, through fiber optics, or via electromagnetic radiation, wherein when the computer program code is loaded into and executed by a computer, the computer becomes an apparatus for practicing embodiments of the disclosed subject matter. When implemented on a general-purpose microprocessor, the computer program code segments configure the microprocessor to create specific logic circuits. In some configurations, a set of computer-readable instructions stored on a computer-readable storage medium may be implemented by a general-purpose processor, which may transform the general-purpose processor or a device containing the general-purpose processor into a special-purpose device configured to implement or carry out the instructions. Embodiments may be implemented using hardware that may include a processor, such as a general purpose microprocessor and/or an Application Specific Integrated Circuit (ASIC) that embodies all or part of the techniques according to embodiments of the disclosed subject matter in hardware and/or firmware. The processor may be coupled to memory, such as RAM, ROM, flash memory, a hard disk or any other device capable of storing electronic information. The memory may store instructions adapted to be executed by the processor to perform the techniques according to embodiments of the disclosed subject matter.
- The foregoing description, for purpose of explanation, has been described with reference to specific embodiments. However, the illustrative discussions above are not intended to be exhaustive or to limit embodiments of the disclosed subject matter to the precise forms disclosed. Many modifications and variations are possible in view of the above teachings. The embodiments were chosen and described in order to explain the principles of embodiments of the disclosed subject matter and their practical applications, to thereby enable others skilled in the art to utilize those embodiments as well as various embodiments with various modifications as may be suited to the particular use contemplated.
Claims (21)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US14/741,525 US20150286478A1 (en) | 2013-11-15 | 2015-06-17 | Application Version Release Management |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US14/081,071 US9087156B2 (en) | 2013-11-15 | 2013-11-15 | Application version release management |
US14/741,525 US20150286478A1 (en) | 2013-11-15 | 2015-06-17 | Application Version Release Management |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/081,071 Continuation US9087156B2 (en) | 2013-11-15 | 2013-11-15 | Application version release management |
Publications (1)
Publication Number | Publication Date |
---|---|
US20150286478A1 true US20150286478A1 (en) | 2015-10-08 |
Family
ID=53174618
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/081,071 Active US9087156B2 (en) | 2013-11-15 | 2013-11-15 | Application version release management |
US14/741,525 Abandoned US20150286478A1 (en) | 2013-11-15 | 2015-06-17 | Application Version Release Management |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/081,071 Active US9087156B2 (en) | 2013-11-15 | 2013-11-15 | Application version release management |
Country Status (1)
Country | Link |
---|---|
US (2) | US9087156B2 (en) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20210117172A1 (en) * | 2019-10-21 | 2021-04-22 | Pccw Vuclip (Singapore) Pte. Ltd. | Data-driven consumer journey optimzation system for adaptive consumer applications |
Families Citing this family (14)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10192199B2 (en) | 2011-11-16 | 2019-01-29 | Microsoft Technology Licensing, Llc | Enabling service features within productivity applications |
US9087156B2 (en) * | 2013-11-15 | 2015-07-21 | Google Inc. | Application version release management |
US20150341463A1 (en) * | 2014-05-22 | 2015-11-26 | Microsoft Corporation | Client-side flight version acquisition |
US9921941B2 (en) * | 2015-04-10 | 2018-03-20 | Siemens Aktiengesellschaft | Verification and validation of third party PLC code |
US9727448B1 (en) * | 2015-04-22 | 2017-08-08 | Google Inc. | Method and system for software application testing recommendations |
US10496392B2 (en) | 2015-07-09 | 2019-12-03 | Micro Focus Llc | Staged application rollout |
US10310718B2 (en) | 2016-06-22 | 2019-06-04 | Microsoft Technology Licensing, Llc | Enabling interaction with an external service feature |
US10185626B2 (en) | 2016-09-07 | 2019-01-22 | Microsoft Technology Licensing, Llc | Automatic application error detection and rollback |
US10594839B2 (en) * | 2017-05-09 | 2020-03-17 | Microsoft Technology Licensing, Llc | Virtual assistant skill deployment |
US10339034B2 (en) * | 2017-06-16 | 2019-07-02 | Google Llc | Dynamically generated device test pool for staged rollouts of software applications |
US10977025B2 (en) * | 2018-11-07 | 2021-04-13 | Microsoft Technology Licensing, Llc | Intelligent software asset classification for software update validation |
US11537378B2 (en) * | 2021-03-05 | 2022-12-27 | Microsoft Technology Licensing, Llc | Selecting a sample set of cloud computing resources for A/B testing of a software deployment |
US20220413991A1 (en) * | 2021-06-25 | 2022-12-29 | Microsoft Technology Licensing, Llc | Techniques for Improved Statistically Accurate A/B Testing of Software Build Versions |
US20230061613A1 (en) * | 2021-09-02 | 2023-03-02 | Capital One Services, Llc | Parallel rollout verification processing for deploying updated software |
Citations (24)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20040205748A1 (en) * | 2003-04-11 | 2004-10-14 | Microsoft Corporation | System and method for providing service of automated creation of computer software production images |
US20040261070A1 (en) * | 2003-06-19 | 2004-12-23 | International Business Machines Corporation | Autonomic software version management system, method and program product |
US7085822B1 (en) * | 2000-03-14 | 2006-08-01 | International Business Machines Corporation | Managing pervasive devices |
US7181512B1 (en) * | 2002-05-06 | 2007-02-20 | Accenture Global Services Gmbh | Workstation deployment |
US20070106708A1 (en) * | 2005-10-26 | 2007-05-10 | Dana Rigg | Managing hierarchies of components |
US20070240131A1 (en) * | 2006-03-28 | 2007-10-11 | Go Play Network, Inc. | Application prototyping |
US7555551B1 (en) * | 2005-04-13 | 2009-06-30 | Symantec Corporation | Automatic controllable deployment of software updates |
US20100131084A1 (en) * | 2008-11-25 | 2010-05-27 | Van Camp Kim O | Software deployment manager integration within a process control system |
US20120266155A1 (en) * | 2011-04-13 | 2012-10-18 | Xerox Corporation | Method and system to regulate the electronic availability of application software updates based on information collected regarding installation, usage and support for these updates |
US20120297367A1 (en) * | 2011-05-19 | 2012-11-22 | Verizon Patent And Licensing, Inc. | Testing an application |
US8331901B2 (en) * | 2009-01-28 | 2012-12-11 | Headwater Partners I, Llc | Device assisted ambient services |
US20130054682A1 (en) * | 2011-08-29 | 2013-02-28 | Fiberlink Communications Corporation | Platform for deployment and distribution of modules to endpoints |
US20130067298A1 (en) * | 2011-09-08 | 2013-03-14 | Microsoft Corporation | Automatically allocating clients for software program testing |
US20130073336A1 (en) * | 2011-09-15 | 2013-03-21 | Stephan HEATH | System and method for using global location information, 2d and 3d mapping, social media, and user behavior and information for a consumer feedback social media analytics platform for providing analytic measfurements data of online consumer feedback for global brand products or services of past, present, or future customers, users or target markets |
US8429602B2 (en) * | 2010-03-12 | 2013-04-23 | Hewlett-Packard Development Company, L.P. | Identification of user interface controls |
US20130167138A1 (en) * | 2009-10-09 | 2013-06-27 | Pradeep Cyril Ekka | Method and apparatus for simulating installations |
US20130246849A1 (en) * | 2012-03-16 | 2013-09-19 | James Lee Plamondon | Distributed Testing Of A Software Platform |
US8756318B1 (en) * | 2007-10-09 | 2014-06-17 | Microsoft Corporation | Software deployment using client location |
US8839035B1 (en) * | 2011-09-14 | 2014-09-16 | Amazon Technologies, Inc. | Cloud-based test execution |
US8843914B1 (en) * | 2011-09-19 | 2014-09-23 | Amazon Technologies, Inc. | Distributed update service |
US8856748B1 (en) * | 2013-09-17 | 2014-10-07 | Xamarin Inc. | Mobile application testing platform |
US9087156B2 (en) * | 2013-11-15 | 2015-07-21 | Google Inc. | Application version release management |
US9130801B1 (en) * | 2010-11-22 | 2015-09-08 | Google Inc. | Pushed-based package distribution |
US9223562B1 (en) * | 2005-04-13 | 2015-12-29 | Symantec Corporation | Controllable deployment of software updates |
Family Cites Families (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20090253517A1 (en) | 2008-04-04 | 2009-10-08 | Zipzapplay, Inc. | Open game engine and marketplace with associated game editing and creation tools |
US20100240461A1 (en) | 2009-03-18 | 2010-09-23 | Michael Rasmussen | Methods, systems, and computer readable media for online community-driven computer game development |
US10853826B2 (en) | 2012-02-07 | 2020-12-01 | Yeast, LLC | System and method for evaluating and optimizing media content |
-
2013
- 2013-11-15 US US14/081,071 patent/US9087156B2/en active Active
-
2015
- 2015-06-17 US US14/741,525 patent/US20150286478A1/en not_active Abandoned
Patent Citations (25)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7085822B1 (en) * | 2000-03-14 | 2006-08-01 | International Business Machines Corporation | Managing pervasive devices |
US7181512B1 (en) * | 2002-05-06 | 2007-02-20 | Accenture Global Services Gmbh | Workstation deployment |
US20040205748A1 (en) * | 2003-04-11 | 2004-10-14 | Microsoft Corporation | System and method for providing service of automated creation of computer software production images |
US20040261070A1 (en) * | 2003-06-19 | 2004-12-23 | International Business Machines Corporation | Autonomic software version management system, method and program product |
US7555551B1 (en) * | 2005-04-13 | 2009-06-30 | Symantec Corporation | Automatic controllable deployment of software updates |
US9223562B1 (en) * | 2005-04-13 | 2015-12-29 | Symantec Corporation | Controllable deployment of software updates |
US20070106708A1 (en) * | 2005-10-26 | 2007-05-10 | Dana Rigg | Managing hierarchies of components |
US20070240131A1 (en) * | 2006-03-28 | 2007-10-11 | Go Play Network, Inc. | Application prototyping |
US8756318B1 (en) * | 2007-10-09 | 2014-06-17 | Microsoft Corporation | Software deployment using client location |
US20100131084A1 (en) * | 2008-11-25 | 2010-05-27 | Van Camp Kim O | Software deployment manager integration within a process control system |
US8331901B2 (en) * | 2009-01-28 | 2012-12-11 | Headwater Partners I, Llc | Device assisted ambient services |
US20130167138A1 (en) * | 2009-10-09 | 2013-06-27 | Pradeep Cyril Ekka | Method and apparatus for simulating installations |
US8429602B2 (en) * | 2010-03-12 | 2013-04-23 | Hewlett-Packard Development Company, L.P. | Identification of user interface controls |
US9130801B1 (en) * | 2010-11-22 | 2015-09-08 | Google Inc. | Pushed-based package distribution |
US20120266155A1 (en) * | 2011-04-13 | 2012-10-18 | Xerox Corporation | Method and system to regulate the electronic availability of application software updates based on information collected regarding installation, usage and support for these updates |
US8910142B2 (en) * | 2011-04-13 | 2014-12-09 | Xerox Corporation | Method and system to regulate the electronic availability of application software updates based on information collected regarding installation, usage and support for these updates |
US20120297367A1 (en) * | 2011-05-19 | 2012-11-22 | Verizon Patent And Licensing, Inc. | Testing an application |
US20130054682A1 (en) * | 2011-08-29 | 2013-02-28 | Fiberlink Communications Corporation | Platform for deployment and distribution of modules to endpoints |
US20130067298A1 (en) * | 2011-09-08 | 2013-03-14 | Microsoft Corporation | Automatically allocating clients for software program testing |
US8839035B1 (en) * | 2011-09-14 | 2014-09-16 | Amazon Technologies, Inc. | Cloud-based test execution |
US20130073336A1 (en) * | 2011-09-15 | 2013-03-21 | Stephan HEATH | System and method for using global location information, 2d and 3d mapping, social media, and user behavior and information for a consumer feedback social media analytics platform for providing analytic measfurements data of online consumer feedback for global brand products or services of past, present, or future customers, users or target markets |
US8843914B1 (en) * | 2011-09-19 | 2014-09-23 | Amazon Technologies, Inc. | Distributed update service |
US20130246849A1 (en) * | 2012-03-16 | 2013-09-19 | James Lee Plamondon | Distributed Testing Of A Software Platform |
US8856748B1 (en) * | 2013-09-17 | 2014-10-07 | Xamarin Inc. | Mobile application testing platform |
US9087156B2 (en) * | 2013-11-15 | 2015-07-21 | Google Inc. | Application version release management |
Non-Patent Citations (1)
Title |
---|
Crameri et al., Staged Deployment in Mirage, an Integrated Software Upgrade Testing and Distribution System, ACM (2007) * |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20210117172A1 (en) * | 2019-10-21 | 2021-04-22 | Pccw Vuclip (Singapore) Pte. Ltd. | Data-driven consumer journey optimzation system for adaptive consumer applications |
Also Published As
Publication number | Publication date |
---|---|
US20150143345A1 (en) | 2015-05-21 |
US9087156B2 (en) | 2015-07-21 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US9087156B2 (en) | Application version release management | |
CN101794359B (en) | Methods and systems for enabling community-tested security features for legacy applications | |
US10374930B2 (en) | Off-peak patching for enterprise stability | |
US10339034B2 (en) | Dynamically generated device test pool for staged rollouts of software applications | |
US9183072B1 (en) | Error troubleshooting using a correlated knowledge base | |
US9645914B1 (en) | Apps store with integrated test support | |
WO2016109573A1 (en) | Methods and apparatus to correct age misattribution in media impressions | |
WO2016110234A1 (en) | Cloud platform application-oriented service recommendation method, device and system | |
US9665465B1 (en) | Automated determination of application permissions | |
US10437581B1 (en) | Internet of things platform for handling firmware transfer on machine-to-machine devices | |
US9753784B2 (en) | Cloud delivery platform | |
US20140330647A1 (en) | Application and service selection for optimized promotion | |
US20220147431A1 (en) | Distributed website load testing system running on mobile devices | |
US20140189670A1 (en) | System and method for tailoring applications and application repositories for individual electrical devices | |
CN110737548B (en) | Data request method and server | |
US20150113036A1 (en) | Server and method for sharing application services | |
CN109657164B (en) | Method, device and storage medium for publishing message | |
US20130318036A1 (en) | Zeno: batch-computed news feed updates | |
CN112041818A (en) | System for adjusting application performance based on platform horizontal benchmarking | |
JP6015750B2 (en) | Log collection server, log collection system, and log collection method | |
US20150178341A1 (en) | Application evaluation | |
CN107665135B (en) | Upgrading program pushing method and device | |
US10281973B2 (en) | Application power usage | |
WO2018005244A1 (en) | Constraint based controlled seeding | |
CN109446074B (en) | Pressure testing method and device for on-demand distribution system, computer device and computer storage medium |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: GOOGLE INC., CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:PATTON, BRIAN;FORTESCUE, NICOLAS;SWIGON, PIOTR KRZYSZTOF;AND OTHERS;SIGNING DATES FROM 20131024 TO 20131115;REEL/FRAME:035933/0460 |
|
AS | Assignment |
Owner name: GOOGLE LLC, CALIFORNIA Free format text: CHANGE OF NAME;ASSIGNOR:GOOGLE INC.;REEL/FRAME:044695/0115 Effective date: 20170929 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION |