68 research outputs found
Testing of Smart TV Applications: Key Ingredients, Challenges and Proposed Solutions
Smart TV applications are software applications that have been designed to
run on smart TVs which are televisions with integrated Internet features.
Nowadays, the smart TVs are going to dominate the television market, and the
number of connected TVs is growing exponentially. This growth is accompanied by
the increase of consumers and the use of smart TV applications that drive these
devices. Due to the increasing demand for smart TV applications especially with
the rise of the Internet of Things (IoT) services, it is essential to building
an application with a certain level of quality. Despite the analogy between the
smart TV and mobile apps, testing smart TV applications is different in many
aspects due to the different nature of user interaction and development
environment. To develop the field and formulate the concepts of smart TV
application testing, this paper aims to provide the essential ingredients,
solutions, answers to the most critical questions, and open problems. In
addition, we offer initial results and proof of concepts for a creeper
algorithm to detect essential views of the applications. This paper serves as
an effort to report the key ingredients and challenges of the smart TV
application testing systematically to the research community.Comment: FTC 2018 Conferenc
Internet of Things: Current Challenges in the Quality Assurance and Testing Methods
Contemporary development of the Internet of Things (IoT) technology brings a
number of challenges in the Quality Assurance area. Current issues related to
security, user's privacy, the reliability of the service, interoperability, and
integration are discussed. All these create a demand for specific Quality
Assurance methodology for the IoT solutions. In the paper, we present the state
of the art of this domain and we discuss particular areas of system testing
discipline, which is not covered by related work sufficiently so far. This
analysis is supported by results of a recent survey we performed among ten IoT
solutions providers, covering various areas of IoT applications.Comment: 10 pages Internet of Things (IoT
Tapir: Automation Support of Exploratory Testing Using Model Reconstruction of the System Under Test
For a considerable number of software projects, the creation of effective
test cases is hindered by design documentation that is either lacking,
incomplete or obsolete. The exploratory testing approach can serve as a sound
method in such situations. However, the efficiency of this testing approach
strongly depends on the method, the documentation of explored parts of a
system, the organization and distribution of work among individual testers on a
team, and the minimization of potential (very probable) duplicities in
performed tests. In this paper, we present a framework for replacing and
automating a portion of these tasks. A screen-flow-based model of the tested
system is incrementally reconstructed during the exploratory testing process by
tracking testers' activities. With additional metadata, the model serves for an
automated navigation process for a tester. Compared with the exploratory
testing approach, which is manually performed in two case studies, the proposed
framework allows the testers to explore a greater extent of the tested system
and enables greater detection of the defects present in the system. The results
show that the time efficiency of the testing process improved with framework
support. This efficiency can be increased by team-based navigational strategies
that are implemented within the proposed framework, which is documented by
another case study presented in this paper
Prioritized Process Test: An Alternative to Current Process Testing Strategies
Testing processes and workflows in information and Internet of Things systems
is a major part of the typical software testing effort. Consistent and
efficient path-based test cases are desired to support these tests. Because
certain parts of software system workflows have a higher business priority than
others, this fact has to be involved in the generation of test cases. In this
paper, we propose a Prioritized Process Test (PPT), which is a model-based test
case generation algorithm that represents an alternative to currently
established algorithms that use directed graphs and test requirements to model
the system under test. The PPT accepts a directed multigraph as a model to
express priorities, and edge weights are used instead of test requirements. To
determine the test-coverage level of test cases, a test-depth-level concept is
used. We compared the presented PPT with five alternatives (i.e., the Process
Cycle Test, a naive reduction of test set created by the Process Cycle Test,
Brute Force algorithm, Set-covering Based Solution and Matching-based Prefix
Graph Solution) for edge coverage and edge-pair coverage. To assess the
optimality of the path-based test cases produced by these strategies, we used
fourteen metrics based on the properties of these test cases and 59 models that
were created for three real-world systems. For all edge coverage, the PPT
produced more optimal test cases than the alternatives in terms of the majority
of the metrics. For edge-pair coverage, the PPT strategy yielded similar
results to those of the alternatives. Thus, the PPT strategy is an applicable
alternative, as it reflects both the required test coverage level and the
business priority in parallel
Dynamic Data Consistency Tests Using a CRUD Matrix as an Underlying Model
In testing of software and Internet of Things (IoT) systems, one of necessary
type of tests has to verify the consistency of data that are processed and
stored in the system. The Data Cycle Test technique can effectively do such
tests. The goal of this technique is to verify that the system processes data
entities in a system under test in a correct way and that they remain in a
consistent state after operations such as create, read, update and delete.
Create, read, update and delete (CRUD) matrices are used for this purpose. In
this paper, we propose an extension of the Data Cycle Test design technique,
which is described in the TMap methodology and related literature. This
extension includes a more exact definition of the test coverage, a reflection
of the relationships between the tested data entities, an exact algorithm to
select and combine read and update operations in test cases for a particular
data entity, and verification of the consistency of the produced test cases. As
verified by our experiments, in comparison to the original Data Cycle Test
technique, this proposed extension helps test designers to produce more
consistent test cases that reduce the number of undetected potential data
consistency defects.Comment: Paper accepted at 2020 European Symposium on Software Engineering
(ESSE 2020), Rome, Italy, Nov. 6 - 8, 2020, http://www.esse.org
Employment of Multiple Algorithms for Optimal Path-based Test Selection Strategy
Executing various sequences of system functions in a system under test
represents one of the primary techniques in software testing. The natural way
to create effective, consistent and efficient test sequences is to model the
system under test and employ an algorithm to generate the tests that satisfy a
defined test coverage criterion. Several criteria of test set optimality can be
defined. In addition, to optimize the test set from an economic viewpoint, the
priorities of the various parts of the system model under test must be defined.
Using this prioritization, the test cases exercise the high priority parts of
the system under test more intensely than those with low priority. Evidence
from the literature and our observations confirm that finding a universal
algorithm that produces an optimal test set for all test coverage and test set
optimality criteria is a challenging task. Moreover, for different individual
problem instances, different algorithms provide optimal results. In this paper,
we present a path-based strategy to perform optimal test selection. The
strategy first employs a set of current algorithms to generate test sets; then,
it assesses the optimality of each test set by the selected criteria, and
finally, chooses the optimal test set. The experimental results confirm the
validity and usefulness of this strategy. For individual instances of 50 system
under test models, different algorithms provided optimal results; these results
varied by the required test coverage level, the size of the priority parts of
the model, and the selected test set optimality criteria
Code Coverage Aware Test Generation Using Constraint Solver
Code coverage has been used in the software testing context mostly as a
metric to assess a generated test suite's quality. Recently, code coverage
analysis is used as a white-box testing technique for test optimization. Most
of the research activities focus on using code coverage for test prioritization
and selection within automated testing strategies. Less effort has been paid in
the literature to use code coverage for test generation. This paper introduces
a new Code Coverage-based Test Case Generation (CCTG) concept that changes the
current practices by utilizing the code coverage analysis in the test
generation process. CCTG uses the code coverage data to calculate the input
parameters' impact for a constraint solver to automate the generation of
effective test suites. We applied this approach to a few real-world case
studies. The results showed that the new test generation approach could
generate effective test cases and detect new faults.Comment: 9 page
Aspects of Quality in Internet of Things (IoT) Solutions: A Systematic Mapping Study
Internet of Things (IoT) is an emerging technology that has the promising
power to change our future. Due to the market pressure, IoT systems may be
released without sufficient testing. However, it is no longer acceptable to
release IoT systems to the market without assuring the quality. As in the case
of new technologies, the quality assurance process is a challenging task. This
paper shows the results of the first comprehensive and systematic mapping study
to structure and categories the research evidence in the literature starting in
2009 when the early publication of IoT papers for IoT quality assurance
appeared. The conducted research is based on the most recent guidelines on how
to perform systematic mapping studies. A set of research questions is defined
carefully regarding the quality aspects of the IoT. Based on these questions, a
large number of evidence and research papers is considered in the study (478
papers). We have extracted and analyzed different levels of information from
those considered papers. Also, we have classified the topics addressed in those
papers into categories based on the quality aspects. The study results carry
out different areas that require more work and investigation in the context of
IoT quality assurance. The results of the study can help in a further
understanding of the research gaps. Moreover, the results show a roadmap for
future research directions
A Comprehensive View on Quality Characteristics of the IoT Solutions
Categorization of quality characteristics helps in a more effective
structuring of the testing process and in the determination of properties,
which can be verified in the system under test. In the emerging area of
Internet of Things (IoT) systems, several individual attempts have been made to
summarize these aspects, but the previous work is rather heterogenic and
focuses on specific subareas. Hence, we consolidated the quality
characteristics into one unified view, which specifically emphasizes the
aspects of security, privacy, reliability and usability, as these aspects are
of often quoted as major challenges in the quality of contemporary IoT systems.
The consolidated view also covers other areas of system quality, which are
relevant for IoT system testing and quality assurance. In the paper, we also
discuss relevant synonyms of particular quality characteristics as presented in
the literature or being used in the current industry praxis. The consolidated
view uses two levels of characteristics to maintain a suitable level of
granularity and specificity in the discussed quality characteristics.Comment: Paper accepted at Urb-IoT 2018 - 3rd EAI International Conference on
IoT in Urban Space. November 21-23, 2018, Guimares, Portugal.
http://urbaniot.org/full-program
Avocado: Open-Source Flexible Constrained Interaction Testing for Practical Application
This paper presents the outcome of a research collaboration between academia
and industry to implement and utilize the capabilities of constrained
interaction testing for an open-source tool for industrial-scale application.
The project helps promote flexibility in generating constrained interaction
test suites, executing them, and setting up a test oracle to report them--all
within the same tool called Avocado. Avocado employs a constraint solver with
computational algorithms to generate constrained interaction test suites. The
environment of the application under test can be set up to execute the
generated test suite with minimum effort. A test oracle can be set up by the
tool to report the status and the results of the executed test cases. Avocado
represents a comprehensive and flexible solution for conducting combinatorial
interaction testing (CIT) and constrained CIT on an industrial application. In
this paper, we present the structure of the tool and our method of implementing
the algorithms in detail.Comment: 6 page
- …