Temas para doutoramento:
Projecto Amber-iTest
(An Automated Model-Based User
Interface Testing Environment)
Ana Paiva & João Pascoal Faria
{apaiva,jpf}@fe.up.pt
Grupo de Engenharia de Software
www.fe.up.pt/~softeng
Departamento de Engenharia Informática
www.fe.up.pt/si/unidades_geral.visualizar?p_unidade=151
Faculdade de Engenharia da Universidade do Porto
www.fe.up.pt
ProDEI 29/2/2008
Research interests
Software Quality – our society depends ever more on
software, so its crucial to assure its quality through various
means

Product quality – testing and certification of software products


(the customer perspective)

particularly model-based user interface testing

partnerships: MSR/FSE, etc.
Process quality – improving the software development process to
assure the production of high-quality software in a predictable and
cost-effective manner (the supplier perspective)

particularly the combination of CMMI/PSP/TSP/Agile methods
for fast process improvement

partnerships: SEI, etc.
Formal methods – mathematically based software specification
and verification methods

particularly lightweight and automated methods integrated with
commonly used development environments

partnerships: UM, etc.
Ana Paiva & João Pascoal Faria, ProDEI – 29/2/2008
2
Model-based User Interface Testing
UI testing is a fundamental but currently very time
consuming and poorly systematized V&V activity

Microsoft: number of testers ≈ number of developers
Model-based testing can help to systematize and
automate the UI testing process (through automatic
test case generation from formal models), but poses
specific challenges:




Test case explosion problem: how to generate a test suite of
manageable size that still assures adequate UI testing?
How to bridge the gap between the model and the
implementation?
How to model GUIs?
How to reduce the effort in the construction of the model?
Ana Paiva & João Pascoal Faria, ProDEI – 29/2/2008
3
An example of a MBUIT environment
Model-based testing tool: Spec Explorer (FSE/MSR)


Automates test case generation and execution for API
conformance testing
Test sequences are derived from a FSM which is generated by
bounded exploration of the model
Formal modelling language: Spec# (FSE/MSR)


Extends C# with contracts and high-level constructs
Can be used as modelling or programming language
GUI testing extensions (from previous work at FEUP)



GUI modelling techniques, promoting abstraction and reuse
FSM reduction tool for hierarchical GUIs, to handle the
state/test case explosion problem
GUI mapping tool, to bridge the gap between the model and
its implementation
Ana Paiva & João Pascoal Faria, ProDEI – 29/2/2008
4
GUI modelling with Spec#: example
class FindDialog {
string findWhat = "";
bool matchCase = false;
UpDown direction = Down;
[Action] void SetFindWhat(string value)
requires IsEnabled("FindDialog");
{
findWhat = value;
}
(imaginar em inglês)
[Action] void FindNext()
requires IsEnabled("FindDialog") && findWhat != "";
{
Notepad.Find(findWhat, direction, matchCase);
}
Janela -> classe
Botão ou opção menu -> acção
Controlo com estado -> var. estado
+ acções
}
[Action] void Cancel()
requires IsEnabled(“FindDialog”)
{
WindowManager.RemoveWindow("FindDialog");
}
…
Ana Paiva & João Pascoal Faria, ProDEI – 29/2/2008
5
Current issues
Time and effort required to build the GUI models
Reluctance of GUI testers and modellers in writing textual
formal specifications (they strongly prefer graphical notations
with which they are familiar)

UML models better accepted by testers, but the detail and rigour of
UML models is insufficient for being used as test oracles
Effort required to configure the test generation process to
guarantee the quality of the test cases generated.


There is the need for coverage criteria best adapted for GUI testing,
such as coverage of navigation maps
With Spec Explorer, the bounded exploration process is based on
parameter domain values provided by the tester, but there is no
feedback mechanism to evaluate the quality of the test cases
obtained based on appropriate test adequacy criteria for GUIs
Ana Paiva & João Pascoal Faria, ProDEI – 29/2/2008
6
AMBER-iTest project Goals
Develop a set of tools and techniques to
automate specification based GUI testing,
solving the shortcomings found in our previous
work, so that it can be used in industrial
environments.
Industrial partner: Critical Software
Duration: January 2008 – December 2010
Ana Paiva & João Pascoal Faria, ProDEI – 29/2/2008
7
Amber-iTest overview
2. Ferramenta para
mapear acções do
modelo com objectos de
interacção da GUI
3. Engenharia Reversa
Requisitos
Modelação
≈30%
linhas de código
1. Modelação
visual e teste
4. Definição de
critérios de
cobertura para teste
de GUIs
Microsoft Research:
• Spec Explorer
• Spec#/AsmL
Modelo do
SW
Mapear
acções do
modelo
Geração
automática de
casos de teste
Informação de
mapeamento
Casos de
teste
(abstractos)
SW a
testar
Execução
automática
dos testes
Lista de erros
NModel
5. Validação industrial da abordagem
Ana Paiva & João Pascoal Faria, ProDEI – 29/2/2008
8
Theme/Task 1. Development of a visual GUI
modelling front-end
study different graphical notations used to model GUIs,
and adapt (extend or combine) one notation or a
combination of notations according to the requirements
already described;
since it is not a goal of the project to develop a new visual
editor to support the edition of the graphical notation, a
freely available visual editor will be selected and adapted
to support the graphical view of the GUI model;
definition of a set of rules for the translation between
the graphical notation and the formal textual notation;
construction of a tool for the automatic translation and
round-trip engineering between the two notations.
Preliminary results: Paper MBT’07
Ana Paiva & João Pascoal Faria, ProDEI – 29/2/2008
9
Visual GUI modelling front-end overview
Visual to
formal model
translation
UML visual
model
Test
coverage
analysis
Coverage
information
new
Refinement
Spec
Explorer
m()
pre
post
{body}
Spec# formal model
(explicit spec)
Spec# formal model
(implicit spec)
Test case
generation
new
m()
pre
post
oracle
Visual
modelling
GUI mapping
code & data
(adapter)
Model to
implementation
mapping
Test
Execution
Test
suite
(abstract)
Spec
Explorer
GUI
Application
Under Test
Test results
Ana Paiva & João Pascoal Faria, ProDEI – 29/2/2008
10
Theme/Task 2. Development (generalization) of a
model to implementation mapping tool
support the majority of the Windows controls used in
Windows applications;
support Java SWT applications;
support Web applications (HTML, DHTML, etc.);
support a one-to-many mapping between logical
model actions and physical GUI actions, so that a
model action can correspond to a sequence of
physical actions; the sequence of physical actions will
be captured in way similar to the recording
mechanism of capture-replay testing tools.
Previous work: Paper ICFEM’05
Ana Paiva & João Pascoal Faria, ProDEI – 29/2/2008
11
Theme/Task 3. Development of a GUI reverse
engineering tool
In a first phase, discover the navigation map of the application
and represent it by a state machine.
In a second phase, the goal will be to discover state variables
within each window or page and effects of user actions on state
variables.
In a third phase, it will be introduced the ability to mix
automatic exploration with manual exploration, in order to be
able to access parts of the application functionality that are
protected by a key or that depend on specific input data (this
feature represents an improvement to the state-of-art).
The reverse engineering tool will generate, at the same time,
the mapping information between the model and the
implementation that is needed for test execution (see task 2).
Preliminary results: Paper FMICS’07
Ana Paiva & João Pascoal Faria, ProDEI – 29/2/2008
12
Reverse engineering example
// Main window
[Action] void AddressBook_NewContact()
requires IsEnabled(“AddressBook”);
ensures IsEnabled(“Contact”);
{ AddWindow(“Contact”); }
//Contact dialog
[Action] void SetLastName(string str)
requires IsEnabled(“Contact”);
{ LastName = str; }
//…
// RECORD MODE
[Action(Kind=ActionAttributeKind.Scenario)])
void PasswordScenario()
requires IsEnabled(“Password");
{
TextBox = “mypassword";
Ok();
}
// Find dialog
[Action] void SetFindWhat(string str)
requires IsEnabled(“Find”);
ensures FindButton.IsEnabled();
{ FindButton.Enabled = true; }
Ana Paiva & João Pascoal Faria, ProDEI – 29/2/2008
13
Theme/Task 4. Development of GUI test coverage
analysis and enforcement tools
identification of test coverage criteria that are best suited for
model-based GUI testing, and that can be used to evaluate the
quality and guide the generation of test cases;
techniques and tools to determine the degree of coverage
reached by a given test suite, according to the criteria
previously identified;
techniques and tools to control and guide the generation of
test cases in order to meet specified test coverage goals:



stop the generation of test cases (or the intermediate FSM) as soon
as the coverage goals are met;
remove redundant test cases (or redundant parts of the FSM) with
respect to the coverage goals specified, extending our previous
work presented at ASM´05;
automatically generate domain values that allow achieving the
coverage goals specified (e.g., using constraint satisfaction
algorithms);
Previous work: paper ASM’05
Ana Paiva & João Pascoal Faria, ProDEI – 29/2/2008
14
Amber-iTest - Vantagens
Projecto financiado pela FCT
2 ofertas de bolsa ainda em aberto, duração 2 anos
Já existe trabalho feito em muitas das tarefas
Envolvimento com a indústria
Projecto na área da qualidade de sw (área em
expansão no mundo empresarial)
Interacção/integração com/no Grupo de Engenharia
de Software (investigação e ensino)
Obrigada pela vossa atenção!
Mais informação:
http://paginas.fe.up.pt/~softeng/wiki/doku.php?id=
projects:amber_itest:start
Ana Paiva & João Pascoal Faria, ProDEI – 29/2/2008
15
References and further information
“Reverse Engineered Formal Models for GUI Testing”. Ana C. R. Paiva,
João C. P. Faria, Pedro M. C. Mendes. 12th International Workshop on
Formal Methods for Industrial Critical Systems (FMICS 2007). July 1-2,
2007, LNCS vol. 4916, Berlin, Germany.
"Towards the Integration of Visual and Formal Methods for GUI Testing"
Ana C. R. Paiva, João C. P. Faria, Raul F. A. M. Vidal. MBT'06 - Third
Workshop on Model Based Testing, 31 March - 1 April, 2007, Braga,
Portugal
"A Model-to-implementation Mapping Tool for Automated Model-based
GUI Testing". Ana C. R. Paiva, João C. P. Faria, Nikolai Tillmann, Raul F.
A. M. Vidal.ICFEM 2005 - Seventh International Conference on Formal
Engineering Methods, 1-4 November 2005, Manchester, UK.
"Modeling and Testing Hierarchical GUIs". Ana C. R. Paiva, João C. P.
Faria, Raul F. A. Moreira, Nikolai Tillmann. ASM 2005 - 12th
International Workshop on Abstract State Machines, March 8-11, 2005,
Paris, France.
Automated Specification-Based Testing of Graphical User Interfaces,
PhD Thesis, Ana Cristina Paiva Pimenta, FEUP, November 2006
AMBER iTest Project:
http://paginas.fe.up.pt/~softeng/wiki/doku.php?id=projects:amber_it
est:start
Ana Paiva & João Pascoal Faria, ProDEI – 29/2/2008
16