Perpetual Testing

Complex software systems can benefit from a virtually limitless amount of testing and analysis. This project is developing technology for assuring that deployed software continues to undergo test and analysis. This continued effort, coordinated by the development environment, can be more sharply focussed by observing actual system utilization. Process technology is used to marshal diverse testing and analysis approaches and harness increased testing resources that may be accessible through the deployment environment.

A major goal of Perpetual Testing is improving the effectiveness of testing while reducing the human effort required to test (and re-test) software. Currently, many hours of effort are expended to test software ``thoroughly'' in the development environment (often based only on generally incomplete and inaccurate models and only presumptions of the deployment environment) usually offset by only limited resources. One important advantage of Perpetual Testing is that it extends indefinitely the time during which testing and analysis can occur, rather than restricting this time to a fixed, arbitrary "testing phase." We expect to continue fielding systems tested to a level of thoroughness that is common today, but also to continually improve testing thoroughness after deployment. By gathering metrics about the software, discovered faults, and the cost and effectiveness of various techniques, plans that incorporate the most appropriate analysis and testing capabilities can be formulated, executed, and continuously improved.

While Perpetual Testing improves the trustworthiness of software, it also supports incremental recertification of evolving software. The dependability and evolvability of systems tend to degrade over time, as changes degrade system structure and invalidate implicit assumptions. The perpetual testing approach can improve evolvability by exploiting a wealth of amassed information and using it to determine the scope of changes and to direct the retesting and reanalysis process, thus contributing to rapid evolvability. This vision of Perpetual Testing implies that software be encased in an environment that supports its development and evolution and that takes a pro-active role in assuring that its testing and analysis continue. This requires coordination of multiple information sources in the development and field environments, and maintenance of a rich information web in a form that can be exploited for change impact analysis as well as improved assurance. Clearly this can not be achieved without a highly automated testing and analysis process integrating tools and information sources and exploiting idle computing resources.

The perpetual testing project is developing technologies to support seamless, perpetual analysis and testing of software through deployment and evolution. Whereas the current dominant paradigm treats testing as a phase that succeeds development and precedes delivery, we are building the foundation for treating analysis and testing as on-going activities to improve quality assurance without pause through several generations of product, in the development environment as well as the deployed environment. Software in the deployed environment is monitored not only to check conformance to required properties but also to validate and refine the models and assumptions on which quality assurance activities in the development environment depend. The degree of monitoring and transmission of information to the development environment differs depending on performance and security requirements of the end-user and is always be under user control.

Perpetual testing is necessarily incremental. Analysis and testing processes are carried out in response to changes in software artifacts or associated information or in anticipation of change. Improvements to existing technologies focus largely on scalability and incrementality for large evolving systems. Analysis and testing is aimed at attaining and maintaining adequate adherence of all software artifacts to relations captured by a rich web of hypercode links, including dependence relations among software components and among properties and analysis techniques.

© 2006 University of Massachusetts Amherst. Site Policies.
This site is maintained by the Laboratory for Advanced Software Engineering Research.