Using Dependability Benchmarks To Support Iso/iec Square

The integration of Commercial-Off-The-Shelf (COTS) components in software has reduced time-to-market and production costs, but selecting the most suitable component, among those available, remains still a challenging task. This selection process, typically named benchmarking, requires evaluating the behaviour of eligible components in operation, and ranking them attending to quality characteristics. Most existing benchmarks only provide measures characterising the behaviour of software systems in absence of faults ignoring the hard impact that both accidental and malicious faults have on software quality. However, since using COTS to build a system may motivate the emergence of dependability issues due to the interaction between components, benchmarking the system in presence of faults is essential. The recent ISO/IEC 25045 standard copes with this lack by considering accidental faults when assessing the recoverability capabilities of software systems. This paper proposes a dependability benchmarking approach to determine the impact that faults (noted as disturbances in the standard) either accidental or malicious may have on the quality features exhibited by software components. As will be shown, the usefulness of the approach embraces all evaluator profiles (developers, acquirers and third-party evaluators) identified in the ISO/IEC 25000 "SQuaRE" standard. The feasibility of the proposal is finally illustrated through the benchmarking of three distinct software components, which implement the OLSR protocol specification, competing for integration in a wireless mesh network. © 2011 IEEE.2837IEEE,IEEE Computer Society,ifip,JPLEgyed, A., Balzer, R., Integrating COTS software into systems through instrumentation and reasoning (2006) Automated Software Engineering, 13 (1), pp. 41-64. , DOI 10.1007/s10515-006-5466-4Wallnau, K., Hissam, S., Seacord, R., Building systems from commercial components (2002) SEI Series in Software Eng., , Addison-WesleyDonzelli, P., Zelkowitz, M., Basili, V., Allard, D., Meyer, K.N., Evaluating COTS component dependability in context (2005) IEEE Software, 22 (4), pp. 46-53. , DOI 10.1109/MS.2005.91Misra, K.B., (2008) Handbook of Performability Engineering, , 1st ed. Springer Publishing Company, IncorporatedKanoun, K., Spainhower, L., (2008) Dependability Benchmarking for Computer Systems, , Wiley and IEEE Computer Society PressVoas, J.M., McGraw, G., (1997) Software Fault Injection: Inoculating Programs Against Errors, , John Wiley & Sons, Inc(2010) ISO/IEC 25000. Software Engineering - Software Product Quality Requirements and Evaluation (SQuaRE) - Guide to SQuaRE, , Geneve ISO(2010) ISO/IEC 25045. Systems and Software Engineering - Systems and Software Quality Requirements and Evaluation (SQuaRE) - Evaluation Module for Recoverability, , Geneve ISO(2011), http://www.tpc.org, onlineArlat, J., Aguera, M., Amat, L., Crouzet, Y., Fabre J.-Charles, Laprie J.-Claude, Martins, E., Powell, D., Fault injection for dependability validation: A methodology and some applications (1990) IEEE Transactions on Software Engineering, 16 (2), pp. 166-182. , DOI 10.1109/32.44380Avizienis, A., Laprie, J.-C., Randell, B., Landwehr, C., Basic concepts and taxonomy of dependable and secure computing (2004) IEEE Transactions on Dependable and Secure Computing, 1 (1), pp. 11-33. , DOI 10.1109/TDSC.2004.2Dujmovic, J., Elnicki, R., (1982) A DMS Cost/Benefit Decision Model: Mathematical Models for Data Management System Evaluation, Comparison, and Selection, , National Bureau of Standards, Washington D.C., No. GCR 82-374. NTIS No. PB 82-170150Morris, M.F., Kiviat graphs: Conventions and figures of merit (1974) ACM/Sigmetrics Performance Evaluation Review, 3 (3), pp. 2-8(2011), http://ruralnet.sourceforge.net/, online(2010) Hillsdale WMN, , http://dashboard.openmesh.com/overview2.php?id=Hillsdale, OnlineChhabra, J., Real-world experiences with an interactive ad hoc sensor network (2002) Proceedings of the 2002 International Conference on Parallel Processing Workshops, pp. 143-151Lu, W., Seah, W.K.G., Peh, E.W.C., Ge, Y., Communications support for disaster recovery operations using hybrid mobile ad-hoc networks Proceedings of the 32nd IEEE Conference on Local Computer Networks, 2007, pp. 763-770Akyildiz, I.F., Wireless mesh networks: A survey (2005) IEEE Radio Communications, 43, pp. S23-S30Friginal, J., De Andres, D., Ruiz, J.-C., Gil, P., Using performance, energy consumption, and resilience experimental measures to evaluate routing protocols for ad hoc networks 10th IEEE Symposium on Network Computing and Applications (NCA), 2011Andrés, D., Friginal, J., Ruiz, J.-C., Gil, P., An attack injection approach to evaluate the robustness of ad hoc networks IEEE Pacific Rim International Symposium on Dependable Computing (PRDC), 2009, pp. 228-233(2010) IEC 61508. Functional Safety of Electrical/ Electronic/programmable Electronic Safety-related Systems, , IEC(1992) DO-178B. Software Considerations in Airborne Systems and Equipment Certification, , Radio Technical Commission for Aeronautics(2001) EN 50128. Railway Applications. Communications, Signalling and Processing Systems. Software for Railway Control and Protection Systems, , British Standards InstitutionSkramstad, T., Assessment of Safety Critical Systems with COTS Software and Software of Uncertain Pedigree (SOUP) Proceedings from ERCIM Workshop on Dependable Systems, ERCIM Conference on Dependable Software Intensive Embedded Systems, Italy, 2005Friginal, J., De Andres, D., Ruiz, J.-C., Gil, P., Attack injection to support the evaluation of ad hoc networks IEEE Symposium on Reliable Distributed Systems (SRDS), 2010, pp. 21-2