Operationalizing Machine Learning: An Interview Study

Organizations rely on machine learning engineers (MLEs) to operationalize ML, i.e., deploy and maintain ML pipelines in production. The process of operationalizing ML, or MLOps, consists of a continual loop of (i) data collection and labeling, (ii) experimentation to improve ML performance, (iii) evaluation throughout a multi-staged deployment process, and (iv) monitoring of performance drops in production. When considered together, these responsibilities seem staggering -- how does anyone do MLOps, what are the unaddressed challenges, and what are the implications for tool builders? We conducted semi-structured ethnographic interviews with 18 MLEs working across many applications, including chatbots, autonomous vehicles, and finance. Our interviews expose three variables that govern success for a production ML deployment: Velocity, Validation, and Versioning. We summarize common practices for successful ML experimentation, deployment, and sustaining production performance. Finally, we discuss interviewees' pain points and anti-patterns, with implications for tool design.Comment: 20 pages, 4 figure

COLLAPSE: Collaborative full-stack platform for data science development

Os projectos de data science diferem do projecto habitual de desenvolvimento de software devido ao seu foco na experimentação. Desde tentar diferentes conjuntos de dados, características e tarefas de pré-processamento, algoritmos e hiper parâmetros, um projecto de data science baseia-se na utilização de uma metodologia para melhorar continuamente os resultados. É essencial neste tipo de projecto poder comparar experiências, bem como ser capaz de reproduzir os resultados associados a cada experiência, pelo que surgem desafios relacionados com a versão dos dados, o seguimento da experiência e a garantia da reprodutibilidade dos resultados. Há também uma necessidade de anotação de dados, a fim de facilitar o desenvolvimento de abordagens supervisionadas e a implantação de modelos em ambientes diversos. Existem várias ferramentas e plataformas em desenvolvimento que são capazes de enfrentar alguns destes desafios, mas não há uma única que pretenda resolvê-los todos, pelo que o objectivo é desenvolver uma solução integrada que forneça às equipas de ciência de dados a anotação de dados, colaboração, versionamento de dados, rastreio de experiências, reprodutibilidade de resultados e implementação de modelos. Isto será conseguido com base no software já existente e desenvolvendo uma camada intermédia entre o utilizador e algumas características destas plataformas com o objectivo final de resolver todos os desafios acima mencionados. Uma plataforma como esta permitirá às equipas de data science cooperar de uma forma mais fácil, bem como ter um desenvolvimento mais eficiente dos projectos. Esta plataforma será validada através do teste do seu desempenho num projecto real.Data science projects differ from the usual software development project because of their focus on experimentation. From trying different datasets, features and pre-processing tasks, algorithms and hyper parameters, a data science project relies on using a methodology to continuously improving the results. It is essential in this kind of project to be able to compare experiments, as well as to be able to reproduce the results associated with each experiment, so challenges related to data versioning, experiment tracking and guaranteeing the reproducibility of the results arise. There is also a necessity for data annotation in order to facilitate the development of supervised approaches and deployment of models in diverse environments. As of now there are several tools and platforms in development that are able to tackle some of these challenges however there is not a singular one that is aiming to solve them all, so the objective is to develop an integrated solution that provides data science teams with data annotation, collaboration, data versioning, experiment tracking, results reproducibility and models deployment. This will be achieved by drawing upon the already existing software and developing a middle layer between the user and some features from these platforms with the final objective being to resolve all the aforementioned challenges. A platform like this will allow data science teams to cooperate in an easier way as well as have a more efficient development of projects. This platform will be validated by testing its performance on a real project

Software Process Modeling with Eclipse Process Framework

The software development industry is constantly evolving. The rise of the agile methodologies in the late 1990s, and new development tools and technologies require growing attention for everybody working within this industry. The organizations have, however, had a mixture of various processes and different process languages since a standard software development process language has not been available. A promising process meta-model called Software & Systems Process Engineering Meta- Model (SPEM) 2.0 has been released recently. This is applied by tools such as Eclipse Process Framework Composer, which is designed for implementing and maintaining processes and method content. Its aim is to support a broad variety of project types and development styles. This thesis presents the concepts of software processes, models, traditional and agile approaches, method engineering, and software process improvement. Some of the most well-known methodologies (RUP, OpenUP, OpenMethod, XP and Scrum) are also introduced with a comparison provided between them. The main focus is on the Eclipse Process Framework and SPEM 2.0, their capabilities, usage and modeling. As a proof of concept, I present a case study of modeling OpenMethod with EPF Composer and SPEM 2.0. The results show that the new meta-model and tool have made it possible to easily manage method content, publish versions with customized content, and connect project tools (such as MS Project) with the process content. The software process modeling also acts as a process improvement activity.Ohjelmistoprosessin mallinnus Eclipse Process Frameworkilla ja SPEM 2.0 metamallilla Ohjelmistot ja ohjelmistoteollisuus kehittyvät jatkuvasti. Ketterien menetelmien tulo 1990-luvun loppupuolella, uudet kehitystyökalut ja teknologiat vaativat yhä enemmän huomiota alalla työskenteleviltä ihmisiltä. Organisaatioilla on kuitenkin ollut sekalainen kirjo prosesseja ja erilaisia prosessikuvauskieliä, koska standardia kuvauskieltä ei ole ollut saatavilla. Prosessimetamalli SPEM 2.0 julkaistiin hiljattain. Tätä mallia hyödyntää mm. Eclipse Process Framework Composer (EPFC) –työkalu, joka on suunniteltu prosessien ja menetelmäsisällön kehittämiseen ja ylläpitoon. Työkalun tavoitteena on tukea useita erilaisia projektityyppejä ja kehitystyylejä. Tässä työssä esitellään seuraavat aiheet ja käsitteet: ohjelmistoprosessit, mallit, perinteiset ja ketterät lähestymistavat, metoditekniikkaa sekä prosessien kehittäminen. Lisäksi tutustutaan muutamiin tunnetuimmista metodologioista (RUP, OpenUP, OpenMethod, XP ja Scrum) ja vertaillaan näitä. Työssä tutkitaan tarkemmin Eclipse Process Framework Composer –työkalua, SPEM 2.0 metamallia, näiden ominaisuuksia, käyttöä sekä mallintamista. Esitän tutkimustulokset ja tutkimuksenkulun OpenMethodin mallintamisesta EPFC –työkalulla sekä SPEM 2.0 -metamallilla. Tulokset osoittavat, että uusi metamalli ja työkalu helpottavat prosessin ja menetelmäsisällön hallintaa, mahdollistavat räätälöityjen julkaisujen teon sisällöstä, sekä yhdistävät prosessin projektityökaluihin kuten MS Projectiin. Mallinnus voidaan lisäksi ymmärtää osana prosessin kehittämistä.Siirretty Doriast

Salary History and Employer Demand: Evidence from a Two-Sided Audit

We study how salary history disclosures affect employer demand, and how salary history bans shape hiring and wages. We show how these effects depend on the properties of the labor market, and we measure the key properties using a novel, two-sided field experiment. Our field experiment features hundreds of recruiters reviewing more than 2, 000 job applications. We randomize the presence of salary history questions as well as job candidates’ disclosures. We find that employers make negative inferences about nondisclosing candidates, and that they anticipate positive selection into disclosure. Recruiters view salary history as a stronger signal about competing options than about worker quality. Disclosures by men (and other highly paid candidates) yield higher salary offers; however, they are negative signals of the value (net of salary) that a worker brings to the firm, and thus they yield fewer callbacks. While disclosures (especially of high amounts) generally increase recruiter beliefs about quality and competing offers, male wage premiums are regarded as a weaker signal of quality than other sources (such as the premiums from working at higher-paying firms or being well paid compared to peers). Recruiters correctly anticipate that women are less likely to disclose salary history at any level, and thus they punish women less than men for silence. When we simulate the effect of salary history bans using our results, we find muted effects on callbacks. Gender inequality in salary offers is reduced; however, equality comes at the expense of lower salaries overall (and especially for men)

Service-Oriented Architecture (SOA), Agile Development Methods and Quality Assurance (QA) : a case study

Dissertação (mestrado)—Universidade de Brasília, Instituto de Ciências Exatas, Departamento de Ciência da Computação, 2019.Este trabalho propõe um framework batízado de NatVI e apresenta um estudo de caso que lidam com a interface entre Service-Oriented Architecture (SOA). Agile Development e Quality Assurance (QA). O framework NatVI busca apresentar uma solução para todo o ciclo de desenvolvimento de software. neste caso com foco em aplicações baseadas em serviços. NatVI foi resultado de uma revisão da literatura onde os 'trade off' conhecidos entre SOA e Métodos Ágeis foram identificados e as soluções possíveis avaliadas e incorporadas ao produto final Também foram consideradas as melhores práticas baseadas tanto nos princípios de SOA quando nos princípios ágeis. Muito importante neste cenário foi não perder QA de vista. uma necessidade intrínseca aos projetos de software. Tudo isso para responder ao aumento no dinamismo dos ambientes de negócios que está aumentando a cada dia devido ao próprio dinamismo do avanço tecnológico. As organizações são chamadas a entregar valores com rapidez e confiança neste ambiente onde as possibilidades de soluções evoluem quase que diariamente. E os governos não são diferentes, obrigados a prestar mais e melhores serviços aos cidadãos e às empresas. O governo brasileiro não é uma exceção. As formas tradicionais de pensar o processo de engenharia de software vêm apresentando algumas dificuldades para lidar com este novo cenário. principalmente porque não são adequadas para lidar com constantes mudanças nos requisitos e entregas rápidas, conceitos que SOA e Métodos de Desenvolvimento Ágeis prometem ser capazes de responder. O estudo de caso foi realizado em uma pequena unidade do governo federal brasileiro. Um órgão responsável pela supervisão de um campo de interesse institucional para o país. O framework NatVI proposto foi aplicado em um ambiente onde SOA já estava em uso, apesar de ínslpiente. O estudo de caso avaliou a evolução da qualidade de software por meio do acompanhamento de métricas de erro no código fonte Avaliou a evolução do entendimento sobre os métodos ágeis bem como o engajamento no processo por parte da equipe de desenvolvimento. Avaliou ainda a satisfação dos clientes com o novo processo de desenvolvimento. Durante o estudo de caso. aproveitou-se um treinamento em desenvolvimento ágil que foi ministrado pela instituição à equipe e TI e alguns clientes Algumas limitações foram identificadas. Por exemplo, o tamanho da equipe de TI envolvida e a quantidade de clientes que participaram foi considerada pequena para uma inferência estatística Uma avaliação subjetiva teve que ser feita para melhorar o entendimento dos números. Desta forma, entrevistas semiestruturada foram feitas. Os resultados encontrados indicam que o caminho é promissor. mas indica também que muitos estudos ainda necessitam ser feitos, o que não é ruim, pois abre um campo vasto para pesquisas, ainda mais considerando outros ingredientes que foram identificados durante este trabalho. que podem muito bem fazer parte de estudos futuros. como containers e DevOps, por exemplo.Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES).This work proposes a framework named NatVI and presents a case study that deals with the interface between the Service-Oríented Architecture (SOA) with Agile Development Methods and Qualíty Assurance (QA). The NatVI framework seeks to present a solution for the entire software development cycle, in this case focusing on service-based applications. The framework was the result of a literature review where the known trade-offs between SOA and Agile Methods were identified and the possible solutions evaluated and incorporated unto the final product. Best practices based on both the SOA principles and agile principles were also considered. It was very important in this scenario not to cose QA of view, an intrinsic need for software projects. All this to respond to the increase in the dynamism of business environments that ís increasing every day due to the very dynamism of technological advancement. Organizations are called to deliver values quickly and confidently in this environment where solutíons possibilities evolve almost daily. And governments are no different. obliged to provide more and better services to citizens and busínesses. The Brazilian government is no exception. The traditional ways of thinking the software engineering process have presented some difficulties in dealing with this new scenario. mainly because they are not adequate to deal with constant changes in requirements and fast deliveries. concepts that SOA and Agite Development Methods promise to be able to respond. The case study was carríed out in a small unit of the Brazilian federal government. A unit that is responsible for supervisíng a field of institutional interest to the country. The proposed NatVI framework was applied in an environment where SOA was already in use, though ít was insipid The case study evaluated the evolution of software quality through the monitoring of errar metrics in the source cede. It evaluated the evolution of the understandíng of the agíle methods as well as the engagement in the process by the development team. It also evaluated customer satisfaction with the new development process. During the case study, an agile development training was used that was given by the institution to the team and IT and some clíents. Some limitations have been identified. For example, the size of the IT staff involved and the number of customers who partícipated was considered small for an inference statistics. A subjective assessment had to be made to improve the understanding of numbers. In this way. semi- structured interviews were made. The results indicate that the way ís promising, but it also índicates that many studíes still need to be done. which ís not bad, lince it opens up a vast fleld for research. even more considering other concepts that were identlfied during this work, which can greatly well be part of future studies. such as containers and DevOps, for example

Developing the price competitiveness in project business : Case: International industrial company

The purpose of the the research is to determine how can the price competitiveness of project businesses be further increased by developing the key drivers of the presale period. Thesis is important, because it brings new perspectives to the concept of price competitiveness and it can be used as reference in researching industrial price competitiveness, especially in project-based businesses. The research problem were divided in sub areas: what are the key drivers influencing price competitiveness in project-based business, which drivers can be developed before the actual sale and finally how key drivers should be developed in order to increase price competitiveness. Theoretical section of the thesis go through models and factors for price competitiveness. In the conclusion and discussion section a new model is introduced for analysing price competitiveness in project-based businesses. This model is an extension of the Kim & Maubourne's 2004 generic model. The new model is taking more factors into consideration regarding price competitiveness and gives examples how these can be developed. The new model goes deeper into project-based business and examines how different factors affect each other and opens the theories behind them as well. Theoretical part of the thesis can also be used as a guideline which factors to take into considerations when building new competitiveness businesses and how to rethink the whole process. The empirical part of the thesis was done by interviewing key people inside the case company. Based on the theoretical section, a large number of detailed questions and themes had been selected. The author itself had a long and deep background in the subjects and good understanding of the factors and models in real-life, because he had led project-based businesses and business development in project-based businesses his whole career. The research identifies which factors influence price competitiveness in general. These were studied which are specific ones for project-based businesses. Three factors are highlighted in project-based businesses: the role of supply chain, pricing model and the importance of development activities to secure the contract and correct price levels. In these three areas, the research offers a wide range of ways to develop the price competitiveness

Large Language Models for Social Networks: Applications, Challenges, and Solutions

Large Language Models (LLMs) are transforming the way people generate, explore, and engage with content. We study how we can develop LLM applications for online social networks. Despite LLMs' successes in other domains, it is challenging to develop LLM-based products for social networks for numerous reasons, and it has been relatively under-reported in the research community. We categorize LLM applications for social networks into three categories. First is knowledge tasks where users want to find new knowledge and information, such as search and question-answering. Second is entertainment tasks where users want to consume interesting content, such as getting entertaining notification content. Third is foundational tasks that need to be done to moderate and operate the social networks, such as content annotation and LLM monitoring. For each task, we share the challenges we found, solutions we developed, and lessons we learned. To the best of our knowledge, this is the first comprehensive paper about developing LLM applications for social networks

Developing the price competitiveness in project business : Case: International industrial company

The purpose of the the research is to determine how can the price competitiveness of project businesses be further increased by developing the key drivers of the presale period. Thesis is important, because it brings new perspectives to the concept of price competitiveness and it can be used as reference in researching industrial price competitiveness, especially in project-based businesses. The research problem were divided in sub areas: what are the key drivers influencing price competitiveness in project-based business, which drivers can be developed before the actual sale and finally how key drivers should be developed in order to increase price competitiveness. Theoretical section of the thesis go through models and factors for price competitiveness. In the conclusion and discussion section a new model is introduced for analysing price competitiveness in project-based businesses. This model is an extension of the Kim & Maubourne's 2004 generic model. The new model is taking more factors into consideration regarding price competitiveness and gives examples how these can be developed. The new model goes deeper into project-based business and examines how different factors affect each other and opens the theories behind them as well. Theoretical part of the thesis can also be used as a guideline which factors to take into considerations when building new competitiveness businesses and how to rethink the whole process. The empirical part of the thesis was done by interviewing key people inside the case company. Based on the theoretical section, a large number of detailed questions and themes had been selected. The author itself had a long and deep background in the subjects and good understanding of the factors and models in real-life, because he had led project-based businesses and business development in project-based businesses his whole career. The research identifies which factors influence price competitiveness in general. These were studied which are specific ones for project-based businesses. Three factors are highlighted in project-based businesses: the role of supply chain, pricing model and the importance of development activities to secure the contract and correct price levels. In these three areas, the research offers a wide range of ways to develop the price competitiveness

Modelling Security Requirements Through Extending Scrum Agile Development Framework

Security is today considered as a basic foundation in software development and therefore, the modelling and implementation of security requirements is an essential part of the production of secure software systems. Information technology organisations are moving towards agile development methods in order to satisfy customers' changing requirements in light of accelerated evolution and time restrictions with their competitors in software production. Security engineering is considered difficult in these incremental and iterative methods due to the frequency of change, integration and refactoring. The objective of this work is to identify and implement practices to extend and improve agile methods to better address challenges presented by security requirements consideration and management. A major practices is security requirements capture mechanisms such as UMLsec for agile development processes. This thesis proposes an extension to the popular Scrum framework by adopting UMLsec security requirements modelling techniques with the introduction of a Security Owner role in the Scrum framework to facilitate such modelling and security requirements considerations generally. The methodology involved experimentation of the inclusion of UMLsec and the Security Owner role to determine their impact on security considerations in the software development process. The results showed that overall security requirements consideration improved and that there was a need for an additional role that has the skills and knowledge to facilitate and realise the benefits of the addition of UMLsec