Testing of Smart TV Applications: Key Ingredients, Challenges and Proposed Solutions

Smart TV applications are software applications that have been designed to run on smart TVs which are televisions with integrated Internet features. Nowadays, the smart TVs are going to dominate the television market, and the number of connected TVs is growing exponentially. This growth is accompanied by the increase of consumers and the use of smart TV applications that drive these devices. Due to the increasing demand for smart TV applications especially with the rise of the Internet of Things (IoT) services, it is essential to building an application with a certain level of quality. Despite the analogy between the smart TV and mobile apps, testing smart TV applications is different in many aspects due to the different nature of user interaction and development environment. To develop the field and formulate the concepts of smart TV application testing, this paper aims to provide the essential ingredients, solutions, answers to the most critical questions, and open problems. In addition, we offer initial results and proof of concepts for a creeper algorithm to detect essential views of the applications. This paper serves as an effort to report the key ingredients and challenges of the smart TV application testing systematically to the research community.Comment: FTC 2018 Conferenc

Tapir: Automation Support of Exploratory Testing Using Model Reconstruction of the System Under Test

For a considerable number of software projects, the creation of effective test cases is hindered by design documentation that is either lacking, incomplete or obsolete. The exploratory testing approach can serve as a sound method in such situations. However, the efficiency of this testing approach strongly depends on the method, the documentation of explored parts of a system, the organization and distribution of work among individual testers on a team, and the minimization of potential (very probable) duplicities in performed tests. In this paper, we present a framework for replacing and automating a portion of these tasks. A screen-flow-based model of the tested system is incrementally reconstructed during the exploratory testing process by tracking testers' activities. With additional metadata, the model serves for an automated navigation process for a tester. Compared with the exploratory testing approach, which is manually performed in two case studies, the proposed framework allows the testers to explore a greater extent of the tested system and enables greater detection of the defects present in the system. The results show that the time efficiency of the testing process improved with framework support. This efficiency can be increased by team-based navigational strategies that are implemented within the proposed framework, which is documented by another case study presented in this paper

Handling Constraints in Combinatorial Interaction Testing in the presence of Multi Objective Particle Swarm and Multithreading

Context: Combinatorial testing strategies have lately received a lot of attention as a result of their diverse applications. In its simple form, a combinatorial strategy can reduce several input parameters (configurations) of a system into a small set based on their interaction (or combination). In practice, the input configurations of software systems are subjected to constraints, especially in case of highly configurable systems. To implement this feature within a strategy, many difficulties arise for construction. While there are many combinatorial interaction testing strategies nowadays, few of them support constraints. Objective: This paper presents a new strategy, to construct combinatorial interaction test suites in the presence of constraints. Method: The design and algorithms are provided in detail. To overcome the multi-judgment criteria for an optimal solution, the multi-objective particle swarm optimization and multithreading are used. The strategy and its associated algorithms are evaluated extensively using different benchmarks and comparisons. Results: Our results are promising as the evaluation results showed the efficiency and performance of each algorithm in the strategy. The benchmarking results also showed that the strategy can generate constrained test suites efficiently as compared to state-of-the-art strategies. Conclusion: The proposed strategy can form a new way for constructing of constrained combinatorial interaction test suites. The strategy can form a new and effective base for future implementations.Comment: 35 page

Fuzzy Adaptive Tuning of a Particle Swarm Optimization Algorithm for Variable-Strength Combinatorial Test Suite Generation

Combinatorial interaction testing is an important software testing technique that has seen lots of recent interest. It can reduce the number of test cases needed by considering interactions between combinations of input parameters. Empirical evidence shows that it effectively detects faults, in particular, for highly configurable software systems. In real-world software testing, the input variables may vary in how strongly they interact, variable strength combinatorial interaction testing (VS-CIT) can exploit this for higher effectiveness. The generation of variable strength test suites is a non-deterministic polynomial-time (NP) hard computational problem \cite{BestounKamalFuzzy2017}. Research has shown that stochastic population-based algorithms such as particle swarm optimization (PSO) can be efficient compared to alternatives for VS-CIT problems. Nevertheless, they require detailed control for the exploitation and exploration trade-off to avoid premature convergence (i.e. being trapped in local optima) as well as to enhance the solution diversity. Here, we present a new variant of PSO based on Mamdani fuzzy inference system \cite{Camastra2015,TSAKIRIDIS2017257,KHOSRAVANIAN2016280}, to permit adaptive selection of its global and local search operations. We detail the design of this combined algorithm and evaluate it through experiments on multiple synthetic and benchmark problems. We conclude that fuzzy adaptive selection of global and local search operations is, at least, feasible as it performs only second-best to a discrete variant of PSO, called DPSO. Concerning obtaining the best mean test suite size, the fuzzy adaptation even outperforms DPSO occasionally. We discuss the reasons behind this performance and outline relevant areas of future work.Comment: 21 page

Adopting A Particle Swarm-Based Test Generator Strategy For Variable-Strength And T-Way Testing

Recently, researchers have started to explore the use of Artificial Intelligence (AI)-based algorithms as t-way (where t indicates the interaction strength) and variable-strength testing strategies. Many AI-based strategies have been developed, such as Ant Colony, Simulated Annealing, Genetic Algorithm, and Tabu Search. Although useful, most existing AI-based strategies adopt complex search processes and require heavy computations. For this reason, existing AI-based strategies have been confined to small interaction strengths (i.e., t≤3) and small test configurations. Recent studies demonstrate the need to go up to t=6 in order to capture most faults. This thesis presents the design and implementation of a new interaction test generation strategy, known as the Particle Swarm-based Test Generator (PSTG), for generating t-way and variable-strength test suites. Unlike other existing AI-based strategies, the lightweight computation of the particle swarm search process enables PSTG to support high interaction strengths of up to t=6. The performance of PSTG is evaluated using several sets of benchmark experiments. Comparatively, PSTG consistently outperforms its AI counterparts and other existing strategies as far as the size of the test suite is concerned. Furthermore, the case study demonstrates the usefulness of PSTG for detecting faulty interactions of the input components

An experimental study of hyper-heuristic selection and acceptance mechanism for combinatorial t-way test suite generation

Recently, many meta-heuristic algorithms have been proposed to serve as the basis of a t -way test generation strategy (where t indicates the interaction strength) including Genetic Algorithms (GA), Ant Colony Optimization (ACO), Simulated Annealing (SA), Cuckoo Search (CS), Particle Swarm Optimization (PSO), and Harmony Search (HS). Although useful, metaheuristic algorithms that make up these strategies often require specific domain knowledge in order to allow effective tuning before good quality solutions can be obtained. Hyperheuristics provide an alternative methodology to meta-heuristics which permit adaptive selection and/or generation of meta-heuristics automatically during the search process. This paper describes our experience with four hyper-heuristic selection and acceptance mechanisms namely Exponential Monte Carlo with counter (EMCQ), Choice Function (CF), Improvement Selection Rules (ISR), and newly developed Fuzzy Inference Selection (FIS),using the t -way test generation problem as a case study. Based on the experimental results, we offer insights on why each strategy differs in terms of its performance