Technology Readiness Levels at 40: a study of state-of-the-art use, challenges, and opportunities

The technology readiness level (TRL) scale was introduced by NASA in the 1970s as a tool for assessing the maturity of technologies during complex system development. TRL data have been used to make multi-million dollar technology management decisions in programs such as NASA's Mars Curiosity Rover. This scale is now a de facto standard used for technology assessment and oversight in many industries, from power systems to consumer electronics. Low TRLs have been associated with significantly reduced timeliness and increased costs across a portfolio of US Department of Defense programs. However, anecdotal evidence raises concerns about many of the practices related to TRLs. We study TRL implementations based on semi-structured interviews with employees from seven different organizations and examine documentation collected from industry standards and organizational guidelines related to technology development and demonstration. Our findings consist of 15 challenges observed in TRL implementations that fall into three different categories: system complexity, planning and review, and validity of assessment. We explore research opportunities for these challenges and posit that addressing these opportunities, either singly or in groups, could improve decision processes and performance outcomes in complex engineering projects

Product development risk management and the role of transparency

Thesis (S.M.)--Massachusetts Institute of Technology, Dept. of Mechanical Engineering, 2012.Cataloged from PDF version of thesis.Includes bibliographical references (p. 73-77).Risks in product development lead to schedule and cost over-runs and poor product quality. While numerous risk management frameworks have been published and research on specific risk management practices and methods has been conducted, there is little understanding of what the key characteristics of successful risk management in product development are. This research consists of two phases: an empirical study of the best practices in product development risk management, and a qualitative study of the role of transparency in the same. The results of a survey of over 200 product development practitioners in industry were analyzed. Of the 170 practices from the literature addressed in the survey, 36 best practices in product development risk management were identified. These best practices were categorized in to six groups: 1- Risk Management Personnel and Resources; 2- Tailoring and Integration of the Risk Management Process; 3- Risk-based Decision Making; 4- Specific Mitigation Actions; 5- Monitoring and Review, and; 6- ISO 31000 Principles. The best practices in these categories show strong evidence not only for achieving effective risk management, but also the ability to positively affect overall project stability and the achievement of the project cost, schedule, performance and customer satisfaction targets. All eleven of the ISO 31000:2009 Risk Management Standard principles (ISO 2009b) were found to be best practices of product development risk management, suggesting the standard is applicable to product development The practice with the highest correlation with product development success was found to be one of the eleven ISO principles: "risk management is transparent and inclusive." The second phase of this research aimed to qualitatively validate the observed correlation between transparency and product development success, through twelve semi-structured interviews with product development practitioners from industry. Transparency was found to be an essential feature of product development risk management Transparency of risk management is beneficial to product development in many ways: it is a vehicle for an accurate shared representation of the current state of the product development project, it facilitates stakeholder collaboration; it is a means of aligning efforts towards critical tasks. Requirements for and barriers to transparency were also explored. These results not only inform current product development practitioners on where to focus risk management efforts, but also contribute an empirical evaluation of the impact of specific risk management practices on product development success.by Alison L. Olechowski.S.M

No Risk, No Reward: Towards An Automated Measure of Psychological Safety from Online Communication

The data created from virtual communication platforms presents the opportunity to explore automated measures for monitoring team performance. In this work, we explore one important characteristic of successful teams - Psychological Safety - or the belief that a team is safe for interpersonal risk-taking. To move towards an automated measure of this phenomenon, we derive virtual communication characteristics and message keywords related to elements of Psychological Safety from the literature. Using a mixed methods approach, we investigate whether these characteristics are present in the Slack messages from two design teams - one high in Psychological Safety, and one low. We find that some usage characteristics, such as replies, reactions, and user mentions, might be promising metrics to indicate higher levels of Psychological Safety, while simple keyword searches may not be nuanced enough. We present the first step towards the automated detection of this important, yet complex, team characteristic.Comment: Published as an extended abstract in the ACM Conference on Human Factors in Computing Systems (CHI) 202

Technology-based design and sustainable economic growth

This paper seeks to analyze how design creates economic value. The literature on knowledge-based economic development has primarily focused on innovation as the analytical lens, whereas design is the original action that leads to innovation. Despite the fundamental importance of design, existing design research has offered few insights and little guidance for national strategies due to the lack of focus on and analysis of design in an economic context. This paper addresses such gaps by linking design research and economic development theory. We first elaborate on the relationship among design, invention and innovation, describing the necessity of design activity for invention and innovation. Our analysis of the fundamental characteristics of design across contexts sheds light on the strategic importance of the accumulative nature of technology-based design for sustaining economic growth. Through the lens of technology-based design, we further quantitatively compare Singapore and three similarly-sized countries (South Korea, Finland and Taiwan). Based upon interview data, we also qualitatively examine Singapore's national strategy focusing on design. The quantitative and qualitative results align well with the Singaporean government's use of design as a strategic lever to pursue innovation-driven economic growth, and also reveal its achievements and shortfalls which indicate possible directions for strategic adjustment. Keywords: Technology-based design; Invention; Innovation; Design capability; Economic growthSUTD-MIT International Design Centre (IDC

"Just a little bit on the outside for the whole time": Social belonging confidence and the persistence of Machine Learning and Artificial Intelligence students

The growing field of machine learning (ML) and artificial intelligence (AI) presents a unique and unexplored case within persistence research, meaning it is unclear how past findings from engineering will apply to this developing field. We conduct an exploratory study to gain an initial understanding of persistence in this field and identify fruitful directions for future work. One factor that has been shown to predict persistence in engineering is belonging; we study belonging through the lens of confidence, and discuss how attention to social belonging confidence may help to increase diversity in the profession. In this research paper, we conduct a small set of interviews with students in ML/AI courses. Thematic analysis of these interviews revealed initial differences in how students see a career in ML/AI, which diverge based on interest and programming confidence. We identified how exposure and initiation, the interpretation of ML and AI field boundaries, and beliefs of the skills required to succeed might influence students' intentions to persist. We discuss differences in how students describe being motivated by social belonging and the importance of close mentorship. We motivate further persistence research in ML/AI with particular focus on social belonging and close mentorship, the role of intersectional identity, and introductory ML/AI courses.Comment: Published in the 2023 Annual Conference of the American Society for Engineering Educatio

Advancing a Model of Students' Intentional Persistence in Machine Learning and Artificial Intelligence

Machine Learning (ML) and Artificial Intelligence (AI) are powering the applications we use, the decisions we make, and the decisions made about us. We have seen numerous examples of non-equitable outcomes, from facial recognition algorithms to recidivism algorithms, when they are designed without diversity in mind. Thus, we must take action to promote diversity among those in this field. A critical step in this work is understanding why some students who choose to study ML/AI later leave the field. While the persistence of diverse populations has been studied in engineering, there is a lack of research investigating factors that influence persistence in ML/AI. In this work, we present the advancement of a model of intentional persistence in ML/AI by surveying students in ML/AI courses. We examine persistence across demographic groups, such as gender, international student status, student loan status, and visible minority status. We investigate independent variables that distinguish ML/AI from other STEM fields, such as the varying emphasis on non-technical skills, the ambiguous ethical implications of the work, and the highly competitive and lucrative nature of the field. Our findings suggest that short-term intentional persistence is associated with academic enrollment factors such as major and level of study. Long-term intentional persistence is correlated with measures of professional role confidence. Unique to our study, we show that wanting your work to have a positive social benefit is a negative predictor of long-term intentional persistence, and women generally care more about this. We provide recommendations to educators to meaningfully discuss ML/AI ethics in classes and encourage the development of interpersonal skills to help increase diversity in the field.Comment: Presented at the 2022 Annual Conference of the American Society for Engineering Educatio

Characteristics of successful risk management in product design

The paper reports results of one of the largest empirical studies to-date on the impact of design risk management practices on product design success. Through a survey of 224 practices, 38 (in 7 categories) where found to be statistically significant for at least 3 out of 4 performance metrics. The categories are: 1. Organizational Design Experience; 2. Risk Management Personnel and Resources; 3. Tailoring and Integration of Risk Management Process; 4. Risk-Based Decision Making; 5. Specific Mitigation Actions; 6. Monitoring and Review; and 7. Other ISO Risk Management Principles.King Fahd University of Petroleum and Minerals (Center for Clean Water and Clean Energy at MIT and KFUPM (R11-DMN-09))Massachusetts Institute of Technology. Lean Advancement Initiativ

Characteristics and Enablers of Transparency in Product Development Organizations

Risks in product development lead to schedule and cost overruns and poor product quality. While many risk management frameworks have been published and research on specific practices has been conducted, little is understood of key characteristics of successful risk management in product development and how they manifest in real development projects. This research consists of two phases. The first phase is a survey on 171 best practices in risk management. Analysis of over 200 responses from industry practitioners identified transparency as a key characteristic of successful risk management in product development. Due to the limited exploration of the concept of transparency in the literature, the second phase of this work consisted of a qualitative investigation of transparency through interviews with 15 industry practitioners. Analysis of the interview results suggests a hierarchical structure which decomposes transparency into several characteristics and identifies enablers for each of these characteristics. We propose that transparency can be a valuable lever for product developers and managers. Future work is needed to validate the generalizability of the observations provided

Improving the Systems Engineering Process with Multilevel Analysis of Interactions

The systems engineering V (SE-V) is an established process model to guide the development of complex engineering projects (INCOSE, 2011). The SE-V process involves decomposition and integration of system elements through a sequence of tasks that produce both a system design and its testing specifications, followed by successive levels of build, integration, and test activities. This paper presents a method to improve SE-V implementation by mapping multilevel data into design structure matrix (DSM) models. DSM is a representation methodology for identifying interactions either between components or tasks associated with a complex engineering project (Eppinger & Browning, 2012). Multilevel refers to SE-V data on complex interactions that are germane either at multiple levels of analysis, e.g. component versus subsystem conducted either within a single phase or across multiple time phases, e.g. early or late in the SE-V process. This method extends conventional DSM representation schema by incorporating multilevel test coverage data as vectors into the off diagonal cells. These vectors provide a richer description of potential interactions between product architecture and SE-V integration test tasks than conventional domain mapping matrices (DMMs). We illustrate this method with data from a complex engineering project in the offshore oil industry. Data analysis identifies potential for unanticipated outcomes based on incomplete coverage of SE-V interactions during integration tests. Additionally, assessment of multilevel features using maximum and minimum function queries isolates all the interfaces that are associated with either early or late revelations of integration risks based on the planned suite of SE-V integration tests