AutoBlackTest: Automatic Black-Box Testing of Interactive Applications Automated blackbox GUI specifications enhancement and test data generation · Mohammad Ali AutoBlackTest: a tool for automatic black-box testing · Leonardo. Why is automation important? Because black box testing is very broad, and. AutoBlackTest: Automatic Black-Box Testing of Interactive Applications Test Automation, Q-Learning incomplete, and the generated test cases necessarily . We define the utility of actions in Section V. In step multiple GUI states, possibly.

Author: Sagor Kajitilar
Country: Lithuania
Language: English (Spanish)
Genre: Spiritual
Published (Last): 17 February 2005
Pages: 176
PDF File Size: 10.97 Mb
ePub File Size: 7.23 Mb
ISBN: 446-1-63341-648-5
Downloads: 84443
Price: Free* [*Free Regsitration Required]
Uploader: Nejora

Skip to main content. Log In Sign Up. In the second phase, of an efficient and cost-effective software verification process. Different techniques focus on different with the users through a GUI, and present AutoBlackTest, a technique to automatically generate test cases at the system models and various coverage criteria, auutomated covering system level.

AutoBlackTest uses reinforcement learning, in particular events [8] or semantically interacting events [9]. Q-Learning, to learn how to interact with the application under These techniques can automatically generate many test test and stimulate its functionalities. The empirical results show cases, whose effectiveness depends on the completeness of that AutoBlackTest can execute a relevant portion of the code auto,ated the initial model.

When the initial model is obtained by the application under test, and can reveal previously unknown problems by working at the system level and interacting only stimulating the application under test with a simple sampling through the GUI.

Automatically generating test automted can both reduce the In this paper we present AutoBlackTest, a test case gener- costs and increase the effectiveness of testing.

Current auto- ation technique that builds the model and produces the test matic test case generation techniques produce test black-bx that cases incrementally, while interacting with the application cover either functional or structural elements, reducing the under test.

AutoBlackTest discovers the most relevant func- human biases, and at a lower cost than manually generated tionalities and generates test cases that thoroughly sample and maintained test suites [1].

It integrates two strategies, learning The problem of automatically generating test cases has and autoblackteat. It uses Q-Learning [10] to turn the problem attracted the attention of software engineers since the early of generating test cases for an unknown application into the seventies [2]. The difficulty and complexity of the problem problem of an agent that must b,ack-box how to effectively act z has challenged researchers and practitioners for decades, an unknown environment.

It uses multiple heuristics to effec- and the problem is still largely open. Recent research has tively address a number of common but complex cases, such focused testng several aspects of automatic test case generation as compiling complex forms, executing the application with too, span from unit to regression testing, white to black-box normal and abnormal values, and handling specific situations testing, random to model-based testing, and cover different for instance, saving files.

AutoBlackTest aims to produce application domains, from embedded to telecommunication test cases that can activate relevant functional computations wutomated interactive applications [3], [4], [5]. Since In this paper we consider interactive applications, that is AutoBlackTest does not rely on an initial set of executions, applications that serve the requests autoblackrest users who interact with it does not suffer from the limitations of current approaches.

GUI or the Web. The comparison is done on four applications for desk- Current techniques for generating GUI test cases work in top computers. The results show that AutoBlackTest two phases.

Q-Learning Agent The paper is organized as follow. Section IX presents empirical results. Section X discusses related work. Section XI sum- Application test cases marizes the contributions of the paper. Under Test Figure 1. The episodes identified by the Q-Learning Agent cor- The goal of AutoBlackTest is to generate test cases that ex- respond to test cases that can be used to validate future ercise the functionalities of the application by navigating its revisions of the application.

AutoBlackTest: Automatic Black-Box Testing of Interactive Applications – Semantic Scholar

BlackTest, removing the redundant ones, and generating a The Q-Learning Agent is responsible for interacting with test suite that can be automatically re-executed with IBM the application under test. In particular, the agent automatev to Functional Tester.

The architecture of the Q-Learning next action to execute. The model represents the knowledge Agent matches the architecture of a classic autonomic com- about the application that has been acquired by the agent, ponent [12], as shown in Figure 1.


The agent has failure and it is incrementally extended during the testing process. The Q- the episodes. The number of the episodes indicates the Learning Agent brings the application into the selected state number of executions that the agent must produce.

Test by executing a sequence of actions extracted from the model.

Black Box Testing

The behavior of the aitoblacktest along the executed generation of test cases. Since the Q-Learning Agent works sequence may be non-deterministic. If the sequence does not incrementally and can be safely stopped at any time, the test bring the application into the desired state, AutoBlackTest experts can exploit at best the available time by leaving the starts the episode from the reached state. The length of the episodes is the number of actions algorithms, and is useful both black-obx increase the likelihood that the agent should execute before stopping an episode and of exploring the whole environment, in our testinh of the starting a new one.

In a nutshell, the representation of the state is a The abstract state returned by the abstraction process is the collection of the property values of the GUI widgets of state representation that is incorporated in the behavioral the applications. In step 2, the Learner updates the model model. We define the utility of actions in Section V.

In step testiny GUI states, possibly with some blcak-box properties. In step 4, the Executor executes the action multiple states by comparing the widget in a GUI with the selected by the Planner. The application reaches a new GUI widgets represented in the states of the behavioral model.

To detect multiple autoblackteet of the same widget, we When specific conditions hold the Q-Learning process defined a function that generates traits from widgets. A trait is known to converge to an optimal solution [13]. In our is a subset of the widget properties that are expected to case, these conditions do not hold. This is not a problem be both representative and invariant for the given widget. Autoblacktst instance, the trait of a button Section IV describes the Observer which executes step includes the type of the widget for instance, Buttonthe 1.

Section V describes the Learner which executes step 2 position of augoblacktest button in the GUI hierarchy, and the label and the behavioral model. This strategy and the Executor which execute steps 3 and 4, respectively.

However, the whole state of an application is V. The behavioral model difficult to abstract. The Observer addresses this issue by is a labeled multidigraph. A labeled multidigraph is a graph approximating the state of the application with the portion where pairs of nodes can be connected by multiple directed of uatomated state that is visible to the users, that is the GUI state. Each widget wi is a pair the Observer, the edges represent the transitions between typeiPiwhere typei is the type of the widget and Pi is states, and the labels associated with the edges represent a set of teesting with their values.

Each widget is usually both the actions that trigger the transitions and the estimated associated with a large number of properties. Including all utility values of the action. The utility values represent the properties in the state information would produce huge state importance of the action for testing, as discussed below.

For instance, for a textarea, fo extract the that corresponds to the edge, and is computed according values of the properties text, enabled, editable and to both an immediate utility value and the Q-values of the visible. AutoBlackTest currently supports about 40 types vlack-box associated with the edges exiting the state reached of widgets, and covers the most standard elements that can with the current edge, if the action leads to an existing state.

In the following we first present actions that cause transitions between windows, ignoring the the reward function that computes the immediate utility of windows themselves, while blac,-box increases of the Q- an action, and then the computation of the Q-values.

The value of actions with respect to testing depends 02 on the computations activated by the action.

The reward function favors Q: Starting each episode from a random state Q: Excerpt of a model derived with AutoBlackTest abstract states. For instance, a simple action that produces a transition between similar states, for Given two abstract states AS 1 and AS 2 in this order, example filling a text area, can enable the execution of diff AS computes the fraction of widgets that have changed actions that produce transitions between largely different from AS 1 and AS 2taking into account both the widgets states, for example successfully submitting a form.

  ASTM F1216 PDF

A testing that occur only in the target state AS 2and the ones that process that aims to execute valuable computations needs to occur in both states with modified properties, but ignoring identify these transitions that potentially enable actions that the widgets that disappear from P the original state AS 1: The the future [10].

At each step, the Planner uses the Q-values reward of the action is the proportion of new widgets and to choose the action to be executed. We do not in Figure 2. The model is a simplified excerpt of the consider the widgets that disappear when moving from the model of the UPM application, an application for building original to the target state to avoid too fast increments of a personal database of accounts, one of the case studies the Q-Values of the actions that produce transitions among presented in this paper.

The complete model is much larger windows. For the sake of simplicity in the figure, we label When the action reaches a new state, the second term of the states with the name of the UPM windows instead of the formula is zero and the Q-value is given by the reward collection of widget propertiesand the transitions with the function. Otherwise, the Q-value is computed according to type of the widget instead of its identifier followed by the both the reward function and the Q-values of the edges name of the action omitting parameters.

The path through states 02, 03, 05, 18, 06 represents The Learner applies the formula to update the Q-Value the case of a user who overwrites an existing database of every time an action is executed.

The empirical study reported in command, which enables an interesting scenario and actions Section IX shows that the 0. A simple action is a single event has a high Q-value because it combines the selection of triggered on a single widget.

A complex action is a work- an existing file and the click of the Save button into a flow of simple actions orchestrated heuristically. Complex single action, and terminates with the same state obtained actions are defined to effectively face specific situations. For instance, interacting with an action that saves a window Q-values are computed by the Learner starting from the is more effective if a proper complex action has been reward values according to standard Q-Learning formula: In particular, a value activated to execute either a simple or a complex action.

In close to 1 gives high weight to future actions, while a value the current prototype, the Executor interacts with the GUI close to 0 gives high weight to immediate rewards. Table I summarizes the set of sup- rather than maximizing the immediate reward.

According ported widgets and the actions that AutoBlackTest can to our experience with AutoBlackTest a good choice of execute on these widgets. Here we describe the detection strategy ToggleButton, Checkbox, RadioButton selectdeselect for domain independent failures. AutoBlackTest detects hangs by recognizing that the target application is available, but is not responding parameter to be executed. For instance, a Combobox re- to actions.

It reports the sequence of actions that leads quires the specification of the element tot be clicked, which to the failure so that the failure if deterministic can be is identified either by specifying an element or its position. The sets of literals have Uncaught Exceptions: AutoBlackTest monitors the been defined following the boundary testing principle, that is standard output and error streams looking for exceptions.

If it reveals an exception, it produces a failure report that For example, the Combobox can be executed by clicking allows the replication of the execution that exposed the on the first, middle, or the last item in the widget using failure if deterministic. Differently from the previous cases, click pos. In the future we aim at employing more it does not interrupt the execution of the episode.

We estimated the capability of supporting VII. We compare during the testing process are test cases. These test cases the results with GUITAR that represents the state of the art can be stored in an executable form, for instance to support in the field [8]. In the empirical evaluation we addressed regression testing.