WO1998027490A1

WO1998027490A1 - Software testing process control system and method

Info

Publication number: WO1998027490A1
Application number: PCT/US1997/020401
Authority: WO
Inventors: David F. Carrier, Iii; R. John K. Gillespie; Janet Kwai Fun Lui; Donald L. Weeks, Jr.
Original assignee: Alcatel Usa Sourcing, L.P.
Priority date: 1996-12-18
Filing date: 1997-11-07
Publication date: 1998-06-25
Also published as: AU5250398A

Abstract

Software test process control system and method (510) include a test controller (532) which receives test cases (540) checked into a version control system (12). The test cases are tagged with a unique label indicative of a specific version and release of a software product. A load built from source modules of the software product and the tagged test cases (540) are then downloaded to a test device. Test results (550) are received and checked into the version control system (12) and stored in a predetermined directory (442).

Description

SOFTWARE TESTING PROCESS CONTROL SYSTEM AND METHOD

TECHNICAL FIELD OF THE INVENTION

This invention is related in general to the field of computer software systems. More particularly, the invention is related to a software testing process control system and method.

BACKGROUND OF THE INVENTION

Large-scale software development efforts require proper coordination and management of teams of software engineers and test engineers. When a software development and testing effort involves a large group of engineers simultaneously working on multiple versions and releases of a large number of different source modules of the software product, confusion and inefficiency easily results if the development process and subsequent product release are not properly managed.

For example, some engineers may be coding source modules for version 3.1 of a software product X, while some engineers may be incorporating newer features into source modules for version 4.0, and still some engineers may be providing a fix to some problems reported in source modules of version 2.1. Note that it is possible to have overlap between the three groups of engineers, so that an engineer may be involved in all three efforts.

Compounding the problem is the fact that each version of a software product must pass through multiple developmental stages prior to its release, where advancing to the next stage requires the passing of some predetermined testing and approval process. To be tested, some or all the source modules for . that version of the software product must be collected and built into a load. The process of load building is also called compiling and linking of all the source modules. The resultant load or build is the software product to be tested or to be delivered to the customers when all the developmental stages have been passed. The test inputs or test cases must be tailored to the specific version and release of the load being tested or directed toward specific fixes or features that were incorporated into the load.

Previously, the process of determining which source modules to collect, collecting the source modules, building the load therefrom, and testing the load is tedious and time-consuming. Many hours were spent in meetings with software engineers and managers to determine which source modules should be included in which version, and what reported problems were fixed by which source modules, and whether the source modules with new features should be included in the present build. The end result is a list of source modules that is used to identify and pull the source modules, typically from a version control subsystem into which engineers have checked in their work. The source modules are then manually identified and compiled into a load, which are tested with test data or test cases. Testing is also done on a much smaller scale with one or more source modules. These subsets of source modules must also be compiled and linked into a load to be tested. It may be seen that because the process of testing a software load is repeatedly performed in the development of the software product, considerable savings in time, energy, and funds are possible if the process is automated and made more efficient.

SUMMARY OF THE INVENTION

Accordingly, there is a need for apparatus and method for software testing control that automates and stream lines the process. In accordance with the present invention, apparatus and method for software testing control are provided which eliminates or substantially reduces the disadvantages associated with prior systems and methods .

In one aspect of the invention, a software test process control method has the steps of receiving and storing check-in data related to test cases checked into a version control system, and tagging the test cases with a unique label indicative of a specific version and release of a software product. The load built from source modules of the software product and the tagged test cases are then downloaded to a test device.

In another aspect of the invention, an automated method for software file release with test process control, having the steps of receiving and storing check- in data related to source modules checked into a version control system, and receiving and storing check- in data related to test cases checked into the version control system. The source modules are then tagged with a unique label indicative of a specific version and release of a software product, and the test cases are also tagged with the unique label. A load is then built from the tagged source modules and the load and the tagged test cases are downloaded to a test device for testing.

In yet another aspect of the invention, the system includes test cases having data input for testing a load checked into a version control system. A test controller is adapted for tagging the test cases with a unique label indicative of a specific version and release of a software product, for downloading the load and tagged test cases to a test device, and for receiving test results from the test device and storing the test results in a predetermined directory in the version control system. BRIEF DESCRIPTION OF THE DRAWINGS

For a better understanding of the present invention, reference may be made to the accompanying drawings, in which: FIGURE 1 is a simplified block diagram of an exemplary software release control system constructed according to the teachings of the present invention;

FIGURES 2A and 2B are an exemplary flowchart of a software release control process according to the teachings of the present invention;

FIGURE 3 is a block diagram illustrating the software release control process;

FIGURE 4 is a diagram illustrating the users of software release control system; FIGURE 5 is an exemplary flowchart of a process to attach source modules to release forms;

FIGURE 6 is an exemplary flowchart of a process to assign release forms to a build;

FIGURE 7 is an exemplary flowchart of a create new product process;

FIGURE 8 is an exemplary flowchart of a process to create a build;

FIGURE 9 is an exemplary flowchart of a process for defect tracking; FIGURE 10 is a block diagram illustrating defect tracking; and

FIGURE 11 is a simplified block diagram of a remote access to software release control system; and FIGURE 12 is a simplified block diagram of a software testing control system according to the teachings of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

The preferred embodiment (s) of the present invention is (are) illustrated in FIGURES 1-12, where like reference numerals being used to refer to like and corresponding parts of the various drawings. Referring to FIGURE 1, a block diagram of a software release control system 10 constructed according to the teachings of the present invention is shown. Software release control system 10 uses a version control subsystem 12 to manage the numerous versions of software modules or files developed by software engineers. Version control subsystem 12 may be coupled to software release control system 10 and a number of workstations, personal computers, terminals, or any suitable data display and entry device 14, via a computer network 16. At least two databases or files are included in version control subsystem 12 - source files 18 for storing source modules being developed or modified, and third party files 20 for storing source modules of third party software that will be incorporated into the load. During code development, engineers check out source files to work on and check them back in at the end of the work session. Version control subsystem 12 generally keeps track of what changes were made to the file. Examples of available version control subsystems include ClearCase by Pure Atria Software of Shaumburg, Illinois; Control Version System by Free Software Foundation of Cambridge, Massachusetts; DSEE by Hewlett Packard of Palo Alto, California. Further coupled to software release control system 10 is another data display and entry device 22 provided for configuration administrator (s) , who has authority to initiate and oversee the load building process. A problem reporting tool 24, represented in FIGURE 1 by a data display and entry device, is also coupled to software release control system 10. Problem reporting tool 24 is used to record problems reported by customers, test engineers, and other sources. Each problem report is assigned a unique problem report number for tracking and auditing purposes.

Software release control system 10 includes a number of tools, including a load builder 30 and a defect tracker 32. A number of files or databases are also included for storing a variety of data: check-in data 40, approved files 42, deferred files 44, submitted files 46, load list 50, and build report 52. Check-in data database 40 stores records associated with source modules that have been checked into version control subsystem 12. Check-in data 40 may include the developer's name, file name, check-in number, product, release, check-in time, total number of lines, number of lines changed, etc. Approved files database 42 stores data associated with source modules that have received approval for inclusion into a build, while deferred files database 44 stores data associated with source modules that have been denied inclusion into a build. Submitted files database 46 stores data associated with those source modules that have been attached to release forms. Release forms are logical groupings of source modules collected and submitted for the build process. Load list file 50 contains a list of built modules and third party software that have been identified to go onto deliverable media. The load list is used during generation of the deliverable media. Build report database 52 stores data generated from the load building process. Hard copy reports may then be generated from data stored in build report database 52.

After a load is built, it may be downloaded to a portable storage medium 60, such as a tape 62 and compact disc 64. Storage medium 60 containing a load may then be tested on a test bed 66. In addition, the load may be electronically transferred to a remote site 68, via telecommunications networks or computer networks, for operations and/or testing.

FIGURES 2A and 2B is a flowchart describing the process of software release control 100. References may also be made to various system components shown in FIGURE 1 and to the diagram in FIGURE 3 providing an illustration of the process.

A user, typically a software engineer engaged in the development, modification, or testing of software code, logs into system 10 and selects a software product from a displayed list of existing or developing software products, as shown in block 102. If the source module that the engineer desires to work on is already checked into version control subsystem 12, then it is checked out therefrom, as shown in block 104. The engineer then codes or modifies the source module, as shown in block 106. At the end of the work session, the source module is checked back into version control subsystem 12 in block 108. When a source module is checked into version control subsystem 12, a trigger sends check- in data to software release control system 10, as shown in block 110. This check-in process is shown in FIGURE 3, where source modules 160 are checked into version control subsystem 12 and causing triggers to be invoked and received by software release control system 10.

In block 112 of the flowchart, the check-in data are stored by software release control system 10. If the source module is completed, then its associated check-in data may be attached to open release form 166 and 168, as shown in block 116 and illustrated in FIGURE 3. A release form is a logical grouping of check-in data associated with source modules checked into version control subsystem 12. A release form is typically associated with a particular problem report number or feature specification documentation number. When a release form is associated with a problem report number, the source modules associated therewith are developed or modified in response to the problem reported. When a release form is associated with a feature specification documentation number, the source modules associated therewith are typically developed for a new release. Once the release forms are complete, they are submitted as candidates for a particular build 170, as shown in block 118 and illustrated in FIGURE 3.

Software release control system 10 preferably provides a graphical and menu-driven user interface for prompting for data entry and displaying options. For example, to attach check-in data of source modules to a release form, the user may select from a list of possible functions related to release forms, such as listing of currently open release forms, creating a new release form, attaching check-in data to a release form, submitting a release form, etc . A pointing device such as a mouse may be used to select the desired option from the list. To attach check- in data to release forms, the user may select the appropriate option, which brings up a list of currently open release forms for selection. The selection of a release form then causes a list of unattached check-in data for the software product in question that are associated with the particular user. The user may then select one or more check-in data for attachment to the release form. The user may also be prompted to provide additional data for each check-in data selected, such as the date of any preliminary logical inspection of the source module (such as a Fagan inspection) , a problem report number or a feature specification documentation number, etc. Returning to FIGURES 2A and 2B, in block 120, when release forms are submitted for a build, an electronic notification message may be automatically generated and sent to designated personnel having the responsibility of assigning and approving submitted release form(s) to a particular build, such as a configuration administrator. The configuration administrator may then assign the release form to a build and approve the build, as shown in block 122. The release forms are sorted according to approval - those release forms approved for a build and those release forms not approved for a build, as shown in blocks 124 and 126. The release forms not approved for a build may be deferred for a later build, and the associated data are stored, as shown in block 128. Although not shown explicitly, release forms may also be disapproved or rejected for a number of reasons, which may be unsubmitted or deleted from system 10 after a predetermined time period. In block 130, the approved release forms are then provided in a list, which is used to tag or label all source modules 160 stored in database 18 that are to be included in build 170, as shown in block 132 and FIGURE 3. The build label identifies the product, version, and build in which the tagged source modules will be incorporated. System 10 further provides for the permanent identification 171 of versions of source modules 160 with a specific build 170 of a product.

A configuration administrator may initiate a build after all necessary release forms have been submitted, approved, assigned, and tagged. Software release control system 10 first retrieves source modules that bear the appropriate build label from version control subsystem 12, as shown in block 134, and also retrieves any third party software from version control subsystem 12, as shown in block 136. A build script is then invoked and executed to compile and link the source modules, third party software, and any other files necessary for the execution of the resultant software product, as shown in block 138. As shown in FIGURE 3, the built load may be in any one of development stages 176: development, integration test, system test, verification, and field deployment, and is so identified in its load number or part number identifier. For example, a load number may indicate, sequentially, the customer identifier, part number, release, point release, maintenance release, feature release, development state V, and version number. Development state V may indicate the verification stage, for example. In block 142, the build may then be downloaded to a portable storage medium for delivery to a customer, electronically transferred to a desired destination for delivery or testing, or downloaded to a test bed for testing purposes. A build report summarizing information related to the build may also be generated. It may be seen that software release control system 10 preferably provides a number of security levels to users having differing needs and assigned tasks. Referring to FIGURE 4, users 200 of system 10 may be assigned roles, such as system 202, administration 204, manager 206, and regular 208. For example, regular users 208 may have access to those functions related to creating release forms, attaching the source modules to release forms, and submitting release forms. Manager users 206 may have the additional ability to assign release forms to builds for those software products for which they have authorization during selected development stages, such as development and integration test. Administration users 204 may additionally have access to build administration functions, such as initiating the tagging source modules with build labels and specifying third party software. System users 202 may have unrestricted access to all resources.

FIGURE 5 provides additional details on an exemplary process to attach a source module to a release form 220. From a display of a list of open release forms, the user may select one or more release forms, as shown in block 222. A listing of source modules associated with the user and checked into version control subsystem 12 are then displayed for the user's selection, as shown in block 224. For each selected source module, the user is prompted to provide certain predetermined information, such as whether a preliminary inspection, for example Fagan inspection, was performed and date of the inspection, as shown in block 226. The user will also be prompted to enter problem report number (s) or feature specification document number (s) associated with the source module, as shown in block 228. Software release control system 10 then groups the selected source module with the selected release form. The attachment process ends in block 230.

As described above, after source modules are attached to release forms primarily by the engineers or developers, the release forms are then assigned to a build, primarily by build administrators. Referring to FIGURE 6, an exemplary assignment process 240 is shown. A list of release forms submitted for a given product is displayed, as shown in block 242. The user, such as build administrator, selects a release form from the list, as shown in block 244. In block 246, details of check-in data associated with the selected release form are then displayed, so that the user may review the details and indicate his/her approval, as shown in block 248. If the user disapproves of the release form, then an electronic message may be generated and sent to the developer (s) associated with the source modules so that appropriate action may be taken, as shown in block 250. If the release forms are approved, then a list of builds for the selected software product is displayed, as shown in block 254. The user may then select the build that will incorporate the approved release form, as shown in block 256. The process ends in block 258 thereafter.

Referring to FIGURE 7, an exemplary process 270 for creating a new software product in software release control system 10 is shown. As described above, it is preferable to limit access to this function to users with system authority. When this option is selected, system 10 prompts for the product identifier and release number, as shown in block 272. A sequence number and a build index number are then set by system 10, as shown in blocks 274 and 276. For example, sequence number may be set to 1 and the build index number may be set to 0. The process ends in block 278. When the user enters product and release number of an existing product, error messages may be displayed.

FIGURE 8 shows an exemplary process for creating a new build 290. When this function is selected by the user, the screen displays a prompt for the user to identify the owner of the build, as shown in block 292. system 10 then queries and obtains from the check-in database information on software products and previous builds known to be related to the owner identifier entered by the user, as shown in block 294. The result is displayed for the user to select the software product and a build, as shown in block 296. Dependencies such as operating system and environment variables which specify certain values to be used in the build are either copied from an existing build and modified or entered by the user for the new build, as shown in block 298. Other information such as specifying third party software may also be entered at this time. A build label for the new build is then generated, based on the information given, as shown in block 300. The create new build process ends in block 302.

As described above, software release control system 10 is also capable of tracking defects in previous versions and how these defects are fixed in subsequent versions. An exemplary process 320 performed by defect tracker 32 (FIGURE 1) is shown in FIGURE 9. References are also made to FIGURE 10, which provides a graphical illustration of the process. A product and a build are first specified or selected from lists displayed by system 10, as shown in blocks 322 and 324. With this information, defect tracker 32 obtains a list of all check-in data of all source modules associated with the selected build and product, as shown in block 326. Problem report numbers that are specified in the check- in data are then obtained, as shown in block 328. As identified by the problem report numbers, those problem reports stored in problem reporting tool 24

(FIGURE 1) are labeled with the build label of the present build, as shown in block 330. It may be seen in FIGURE 10 that source module check- in data 360 and 362, a release form 364 that includes check-in data 360 and 362, and load 366 are all tagged with a build label 370-376. In determining which check-in data and correspondingly, which source module contains the code for fixing a known defect, a problem report number 384 associated with check-in data 360 is used to identify a problem report 368 with the problem report number identifier 380. Problem report 368 is then tagged with a build label 388 that corresponds to load 366. Therefore, the proper association between a load and problem report (s) is made.

FIGURE 11 shows that a software release control system 400 may also be accessed from a site 410 located remotely from system 400. Software release control system 400 includes a load builder 401, a defect tracker 402, and a database 403, which stores a number of files as described above in conjunction with FIGURE 1. A version control subsystem 404 having a database storing source files 405 is coupled to system 400. A workstation 406 may also be coupled to system 400 to provide administrator access thereto. At remote site 410, a second version control subsystem 411 storing source files 412 generated and/or maintained by software engineers at remote site 410 is provided. Version control subsystem 411 is coupled to developer workstations, personal computers, and other suitable tools 416 which facilitate code development.

At the end of a work session, a software engineer checks in source modules to remote version control subsystem 411, which are then stored in source files database 412. As the source modules are checked in, a trigger is invoked and sent to software release control system 400. Similar to the local site application as shown in FIGURE 1, the trigger includes check-in data. The trigger may be transmitted over telecommunications networks, computer networks, or a dedicated line, as deemed suitable. In a periodic manner, packets containing the contents of source files database 412 are electronically copied to source files database 405 of local version control subsystem 404 to synchronize the databases, so that the contents thereof are essentially identical over time. When a user has completed the source modules, he/she may attach them to one or more release forms and then submit the release forms for a build, in the same manner as described above. When release forms are approved for a build, the corresponding source modules stored in remote source files database 412 that make up the release form are tagged with the appropriate build label. The load building process obtains approved files or source modules from remote version control subsystem 411 to build the load. Defect tracking for the remote site is performed in a similar manner as described above, where a build label becomes associated with problem reports. Constructed in this manner, developers may submit source files from one or more remote sites to one local software release control system for load building and defect tracking.

Referring to FIGURE 12, a testing process control subsystem 510 of the file release control system is shown. System 510 includes a load builder 530 described in detail above, and additionally a test controller 532. Compiled and linked source modules and other files generated during the load building process are stored in a build directory 544 at the end of the load building process. Build directory 544 functions as a staging area for downloading the files to deliverable storage media (not shown) , or test devices 560 such as a desk top test machine 562, or an automated test driver 564 that drives testing executed on a test bed 566. System 510 further includes test cases 540 and project file documents 542. Project file documents 542 may include a number of documents that are associated with the software product under development and testing. For example, project file documents 542 may include meeting minutes, engineering specification documents, engineering memoranda, test cases 540, test reports 550, and build documents, some of which are shown explicitly in FIGURE 12, some are not. Test cases 540 are test inputs and test reports 550 are the result or output of the tests. Build report and build log 552 were generated during the load building process, where the build log records events, messages, actions, and results during the build process, and the build report is a summary of selected information from the build log. Project file documents 542 are stored in version control subsystem 12, of which test cases 520 are shown explicitly.

In operation, during the load building process described above, source modules for a specific version and release are compiled and linked. The compiled and linked executable source files are then stored in build directory 544. The load building process also copies other related files to building area, including test cases 540 and third party software. The test cases are similarly controlled in version control subsystem 12 and similarly tagged with the same or similar build label as the source modules to associate files related to the same version and release as shown in FIGURES 3, 8, and 10. During the test control process, test controller 532 downloads the built load from build directory 544 and test cases 540 to test device 560. Typically, desk top test machine 562 is used to test portions of the software product, and automated test driver 564 and test bed 566 are used to test the entire software product. Test device 560 may be co-located with system 510 or may be located remotely in which the transfer of the load and test cases 540 may be done electronically via a computer or telecommunications network. Test device 560 receives the load and test cases 540 and performs the test or tests. Results of the tests are sent back to test controller 532 and checked into version control subsystem 12. Typically, test reports 550 are stored at predetermined locations or subdirectories in project file documents 542.

Constructed and automated in this manner, software testing is a process integrated with file release control which uses build labels to associate all files related to the same version and release of the software product. Although the present invention and its advantages have been described in detail, it should be understood that various changes, substitutions and alterations can be made therein without departing from the spirit and scope of the invention as defined by the appended claims .

Claims

WHAT IS CLAIMED IS:

1. A software test process control method, comprising the steps of: receiving and storing check- in data related to test cases checked into a version control system; tagging the test cases with a unique label indicative of a specific version and release of a software product; and downloading a load built from source modules of the software product and the tagged test cases to a test device .

2. The method, as set forth in claim 1, further comprising the step of copying the load and test cases to a build directory.

3. The method, as set forth in claim 1, wherein the downloading step comprises the step of downloading the load and the tagged test cases to a desk top test machine.

4. The method, as set forth in claim 1, wherein the downloading step comprises the step of downloading the load and the tagged test cases to an automated test driver and a test bed.

5. The method, as set forth in claim 1, further comprising the steps of : receiving test results from the test device; checking in the test results in to the version control system; and storing the test results in a predetermined directory pre-allocated for the specific version and release of the software product .

6. The method, as set forth in claim 1, further comprising the steps of: receiving test results from the test device; checking in the test results in to the version control system; and storing the test results in a test results subdirectory of a project file directory pre-allocated for the specific version and release of the software product.

7. An automated method for software file release with test process control, comprising the steps of: receiving and storing check- in data related to source modules checked into a version control system; receiving and storing check-in data related to test cases checked into the version control system; tagging the source modules with a unique label indicative of a specific version and release of a software product ; tagging the test cases with the unique label; building a load with the tagged source modules; and downloading the load and the tagged test cases to a test device.

8. The method, as set forth in claim 7, further comprising the step of copying the load and test cases to a build directory.

9. The method, as set forth in claim 7, wherein the downloading step comprises the step of downloading the load and the tagged test cases to a desk top test machine.

10. The method, as set forth in claim 7, wherein the downloading step comprises the step of downloading the load and the tagged test cases to an automated test driver and a test bed.

11. The method, as set forth in claim 7, further comprising the steps of: receiving test results from the test device; checking in the test results in to the version control system; and storing the test results in a predetermined directory pre-allocated for the specific version and release of the software product .

12. The method, as set forth in claim 7, further comprising the steps of: receiving test results from the test device; checking in the test results in to the version control system; and storing the test results in a test results subdirectory of a project file directory pre-allocated for the specific version and release of the software product.

13. A system for software testing process control, comprising: test cases having data input for testing a load checked into a version control system; and a test controller adapted for tagging the test cases with a unique label indicative of a specific version and release of a software product, for downloading the load and tagged test cases to a test device, and for receiving test results from the test device and storing the test results in a predetermined directory in the version control system.

14. The system, as set forth in claim 13, wherein the predetermined directory comprises a subdirectory for storing the test results received from the test device.