Safurai 001: New Qualitative Approach for Code LLM Evaluation

This paper presents Safurai-001, a new Large Language Model (LLM) with significant potential in the domain of coding assistance. Driven by recent advancements in coding LLMs, Safurai-001 competes in performance with the latest models like WizardCoder [Xu et al., 2023], PanguCoder [Shen et al., 2023] and Phi-1 [Gunasekar et al., 2023] but aims to deliver a more conversational interaction. By capitalizing on the progress in data engineering (including latest techniques of data transformation and prompt engineering) and instruction tuning, this new model promises to stand toe-to-toe with recent closed and open source developments. Recognizing the need for an efficacious evaluation metric for coding LLMs, this paper also introduces GPT4-based MultiParameters, an evaluation benchmark that harnesses varied parameters to present a comprehensive insight into the models functioning and performance. Our assessment shows that Safurai-001 can outperform GPT-3.5 by 1.58% and WizardCoder by 18.78% in the Code Readability parameter and more.Comment: 22 pages, 1 figure, 3 table

A web-oriented framework for the development and deployment of academic facing administrative tools and services

The demand for higher education has increased dramatically in the last decade. At the same time, institutions have faced continual pressure to reduce costs and increase quality of education, while delivering that education to greater numbers of students. The introduction of software systems such as virtual learning environments, online learning resources and centralised student record systems has become routine in attempts to address these demands. However, these approaches suffer from a variety of limitations: They do not take all stakeholders’ needs into account. They do not seek to reduce administrative overheads in academic processes. They do not reflect institution-specific academic policies. They do not integrate readily with other information systems. They are not capable of adequately modelling the complex authorisation roles and organisational structure of a real institution. They are not well suited to rapidly changing policies and requirements. Their implementation is not informed by sound software engineering practises or data architecture design. Crucially, as a consequence of these drawbacks such systems can increase administrative workload for academic staff. This thesis describes the research, development and deployment of a system which seeks to address these limitations, the Module Management System (MMS). MMS is a collaborative web application targeted at streamlining and minimising administrative tasks. MMS encapsulates a number of user-facing tools for tasks including coursework submission and marking, tutorial attendance tracking, exam mark recording and final grade calculation. These tools are supported by a framework which acts as a form of “university operating system”. This framework provides a number of different services including an institution abstraction layer, role-based views and privileges, security policy support integration with external systems

Web interaction environments : characterising Web accessibility at the large

Tese de doutoramento, Informática (Engenharia Informática), Universidade de Lisboa, Faculdade de Ciências, 2012Accessibility quality on the Web is essential for providing a good Web experience to people with disabilities. The existence of virtual ramps aid these users grasping and interacting withWeb content, just like the experience of those who are unimpaired. However, more often than not, Web pages impose accessibility barriers, usually centred on the unavailability of tailored content to specific perceptual abilities (e.g., textual description of images, enabling grasping information with assistive technologies), as well as on proper HTML structural elements that adequate the semantics of a Web page. When evaluating the accessibility quality of Web pages, the resulting analysis is often focused on a small sample set (e.g., a single Web page or a selection of pages from a Web site). While this kind of analysis gets the gist of accessibility quality, it misses the big picture on the overall accessibility quality of the Web. This thesis addresses the challenge of observing accessibility phenomena on the Web, through the experimental evaluation of large collections of Web pages. This resulted on new findings about the accessibility quality of the Web, such as its correlation with HTML element count, and the erroneous perception of accessibility quality by developers. Small-scale experiments have been verified also at large scale, such as the correlation between the usage of HTML templates and accessibility quality. Based on the challenges raised by the experimental evaluation, this thesis proposes a novel approach for large scale Web accessibility evaluation based on Linked Data, as well as the establishment of metrics to assess the truthfulness and coverage of automated evaluation methods.A qualidade da acessibilidade é um factor crucial para as pessoas com deficiências terem uma boa experiência de interacção com a Web.A qualidade da acessibilidade é um factor crucial para as pessoas com deficiências terem uma boa experiência de interacção com a Web. A existência de rampas virtuais ajuda estas pessoas a compreender e interagir com conteúdos Web, a par do que o utilizador comum já experiencia. Porém, a maioria das páginas Web ainda contêm barreiras à acessibilidade. Estas barreiras centram-se normalmente na indisponibilidade de conteúdos perceptíveis por diferentes tipos de capacidades (e.g., descrições textuais de imagens), bem como no uso incorrecto de elementos HTML de acordo com a semântica de uma página Web. Nos dias de hoje, a avaliação da qualidade de acessibilidade de páginas Web é ainda efectuada em pequena escala (e.g., uma página Web ou, no melhor caso, um conjunto de páginas representativas de um sítio Web). Apesar deste tipo de avaliações resultarem na compreensão de alguns fenómenos do estado da acessibilidade na Web, ainda não se sabe qual o seu impacto em larga escala. Esta tese discute os principais desafios na observação da acessibilidade da Web, tendo por base um conjunto de avaliações experimentais de colecções de grande dimensão de páginas Web. Destes estudos destacam-se as seguintes contribuições e resultados:a diferença drástica na interpretação dos avisos resultantes de avaliações de acessibilidade Web: um dos resultados principais da avaliação experimental em larga escala destaca a diferença na interpretação dos avisos (warnings) da aplicação de técnicas da norma WCAG, onde a interpretação optimista (i.e., a visão da maioria dos criadores de páginas Web) se distancia amplamente da interpretação conservadora (onde os avisos são interpretados como erros); a correlação entre a qualidade da acessibilidade de uma página Web e a sua complexidade: este mesmo estudo de larga escala revelou uma correlação entre a complexidade de uma página Web (no que diz respeito ao número de elementos HTML que contém) e a qualidade da acessibilidade. Quanto menor a complexidade de uma página Web, mais certa se torna a alta qualidade da acessibilidade dessa página; o benefício do uso de templates e sistemas de gestão de conteúdos na melhoria da acessibilidade de páginas Web: em ambos os estudos experimentais de acessibilidade foi detectada uma correlação entre a qualidade de acessibilidade das páginas Web e o uso de templates e sistemas de gestão de conteúdo. Esta propriedade foi verificada quer em pequena escala (sobre uma colecção de páginas Web da Wikipedia), quer em larga escala; o incumprimento das regras mais elementares e mais conhecidas da acessibilidade: estes estudos experimentais permitiram também verificar que, apesar de toda a envagelização e educação sobre as questões de acessibilidade na Web, a maioria das regras de acessibilidade são incessantemente quebradas pela maioria das páginas Web.Esta problemática verifica-se, em particular, nas regras de cumprimento de acessibilidade mais conhecidas, tal como por exemplo a disponibilidade de textos alternativos a conteúdos multimédia. Com base nestas experiências e resultados, esta tese apresenta um novo modelo de estudo da acessibilidade na Web, tendo por base o ciclo de estudos da Web em larga escala. Deste modelo resultaram as seguintes contribuições: um modelo para a avaliação distribuída de acessibilidade Web, baseado em propriedades tecnológicas e topológicas: foi concebido um modelo de avaliação de acessibilidade Web que permite a concepção de sistemas de avaliação com base em propriedades tecnológicas e topológicas. Este modelo possibilita, entre outras características, o estudo da cobertura de plataformas e avaliadores de acessibilidade, bem como da sua aplicação em larga escala; uma extensão às linguagens e modelos EARL e Linked Data, bem como um conjunto de definições para extrair informação destes: este modelo de avaliação de acessibilidade Web foi sustentado também pela sua concretização em linguagens e modelos já existentes para o estudo de acessibilidade (EARL) e da Web em larga escala (Linked Data), permitindo assim a sua validação; definição dos limites da avaliação de acessibilidade Web: por fim, este modelo de avaliação de acessibilidade permitiu também delinear uma metodologia de meta-avaliação da acessibilidade, na qual se poderão enquadrar as propriedades dos avaliadores de acessibilidade existentes. Todas estas contribuições resultaram também num conjunto de publicações científicas, das quais se destacam: Rui Lopes and Luís Carriço, A Web Science Perspective of Web Accessibility, in submission for the ACM Transactions on Accessible Computing (TACCESS), ACM, 2011; Rui Lopes and Luís Carriço, Macroscopic Characterisations of Web Accessibility, New Review of Hypermedia and Multimedia – Special Issue on Web Accessibility. Taylor & Francis, 2010; Rui Lopes, Karel Van Isacker and Luís Carriço, Redefining Assumptions: Accessibility and Its Stakeholders, The 12th International Conference on Computers Helping People with Special Needs (ICCHP), Vienna, Austria, 14-16 July 2010; Rui Lopes, Daniel Gomes and Luís Carriço, Web Not For All: A Large Scale Study of Web Accessibility, W4A: 7th ACM International Cross-Disciplinary Conference on Web Accessibility, Raleigh, North Carolina, USA, 26-27 April 2010; Rui Lopes, Konstantinos Votis, Luís Carriço, Dimitrios Tzovaras, and Spiridon Likothanassis, The Semantics of Personalised Web Accessibility Assessment, 25th Annual ACM Symposium on Applied Computing (SAC), Sierre, Switzerland, 22-26 March, 2010 Konstantinos Votis, Rui Lopes, Dimitrios Tzovaras, Luís Carriço and Spiridon Likothanassis, A Semantic Accessibility Assessment Environment for Design and Development for the Web, HCI International 2009 (HCII 2009), San Diego, California, USA, 19-24 July 2009 Rui Lopes and Luís Carriço, On the Gap Between Automated and In-Vivo Evaluations of Web Accessibility, HCI International 2009 (HCII 2009), San Diego, California, USA, 19-24 July 2009; Rui Lopes, Konstantinos Votis, Luís Carriço, Spiridon Likothanassis and Dimitrios Tzovaras, Towards the Universal Semantic Assessment of Accessibility, 24th Annual ACM Symposium on Applied Computing (SAC),Waikiki Beach, Honolulu, Hawaii, USA, 8-12 March 2009; Rui Lopes and Luís Carriço, Querying Web Accessibility Knowledge from Web Graphs, Handbook of Research on Social Dimensions of Semantic Technologies, IGI Global, 2009; Rui Lopes, Konstantinos Votis, Luís Carriço, Spiridon Likothanassis and Dimitrios Tzovaras, A Service Oriented Ontological Framework for the Semantic Validation of Web Accessibility, Handbook of Research on Social Dimensions of Semantic Technologies, IGI Global, 2009; Rui Lopes and Luís Carriço, On the Credibility of Wikipedia: an Accessibility Perspective, Second Workshop on Information Credibility on the Web (WICOW 2008), Napa Valley, California, USA, 2008; Rui Lopes, Luís Carriço, A Model for Universal Usability on the Web, WSW 2008: Web Science Workshop, Beijing, China, 22 April 2008; Rui Lopes, Luís Carriço, The Impact of Accessibility Assessment in Macro Scale Universal Usability Studies of the Web, W4A: 5th ACM International Cross-Disciplinary Conference on Web Accessibility, Beijing, China, 21-22 April 2008. Best paper award; Rui Lopes, Luís Carriço, Modelling Web Accessibility for Rich Document Production, Journal on Access Services 6 (1-2), Routledge, Taylor & Francis Group, 2009; Rui Lopes, Luís Carriço, Leveraging Rich Accessible Documents on the Web, W4A: 4th ACM International Cross-Disciplinary Conference on Web Accessibility, Banff, Canada, 7-8 May 2007.Fundação para a Ciência e a Tecnologia (FCT, SFRH/BD/29150/2006

Developing and Securing Software for Small Space Systems

The space systems industry is moving towards smaller multi-vendor satellites, known as Small Space. This shift is driven by economic and technological factors that necessitate hardware and software components that are modular, reusable, and secure. This research addresses two problems associated with the development of modular, reusable, and secure space systems: developing software for space systems (the Development Problem) and securing space systems (the Security Problem). These two problems are interrelated and this research addresses them together. The Development Problem encompasses challenges that space systems developers face as they try to address the constraints induced by reduced budgets, design and development lifecycles, maintenance allowances, multi-vendor component integration and testing timelines. In order to satisfy these constraints a single small satellite might incorporate hardware and software components from dozens of organizations with independent workforces and schedules. The Security Problem deals with growing need to ensure that each one of these software or hardware components behaves according to policy or system design as well as the typical cybersecurity concerns that face any information system. This research addresses the Development Problem by exploring the needs and barriers of Small Space to find the best path forward for the space systems industry to catch up with the methodology advancements already being widely used in other software fields. To do this exploration a series of five surveys, referred to as SISDPA, was conducted to assess current attitudes and state of practice among space system developers. This crystallized a need in space system development — modular reusable open networks can help Small Space realize its potential, but there is still need to address certain security threats. This research addresses the Security Problem by augmenting a modular reusable open-network software development framework, called SSM, by adding policy enforcement in the form of authentication, access control, and encryption provisions, to create a new development framework, SSSM. This design and implementation adds security provisions while minimizing the impact on developers using the framework. SSSM is evaluated in terms of developer and system resource burden and shows that SSSM does not significantly increase developer burden and preserves the ease-of-use of SSM

How to Improve the Security Skills of Mobile App Developers:An Analysis of Expert Knowledge

Much of the world relies heavily on apps. Increasingly those apps handle sensitive information: controlling our financial transactions, enabling our personal communication and holding intimate details of our lives. So the security of those apps is becoming increasingly vital. Yet research shows that those apps contain frequent security and privacy problems; and that almost all of these issues could have been avoided had the developers had sufficient motivation, support and knowledge. This lack of developer knowledge and support is widely perceived as a major threat. We therefore investigated the skills, approach and motivation required for developers. We conducted a Constructivist Grounded Theory study, involving face-to-face interviews with a dozen experts whose cumulative experience totalled over 100 years of secure app development, to develop theory on secure development techniques. The study identified that the subdiscipline of app development security is still at an early stage, and found surprising discrepancies between current industry understanding and the experts’ recommendations. In particular it found that a secure development process tends not to appeal to app developers; and that the approach of identifying common types of security problems is too limited to give an effective security solution. Instead we identified a set of successful techniques we call ‘Dialectical Security’, where ‘dialectic’ means learning by questioning. These techniques use dialogue with a range of counterparties to achieve app security in an effective and economical way. The security increase comes from continued dialog, not passive learning. The novel contribution of our work is to provide: A grounded theory of secure app development that challenges conventional processes and checklists, and A shift in perspective from process to dialectic. Only by working to develop the Dialectical Security skills of app developers shall we begin to see the kinds of secure apps we need to combat crime and privacy invasions

Hybrid Museum Experiences

"So you’re the one getting this gift? Lucky you! Someone who knows you has visited the museum. They searched out things they thought you would care about, and they took photos and left messages for you.” This is the welcoming message for the Gift app, designed to create a very personal museum visit. Hybrid Museum Experiences use new technologies to augment, expand or alter the physical experience of visiting the museum. They are designed to be experienced in close relation to the physical space and exhibit. In this book we discuss three forms of hybridity in museum experiences: incorporating the digital and the physical, creating social, yet personal and intimate experiences, and exploring ways to balance visitor participation and museum curation. The book reports on a three-year cross-disciplinary research project in which artists, design researchers and museum professionals have collaborated to create technology-mediated experiences that merge with the museum environment