CodeCoT and Beyond: Learning to Program and Test like a Developer

In natural language processing, transformer-based large language models (LLMs) like GPT-x models developed by OpenAI have revolutionized the landscape. Despite their impressive capabilities, these models often encounter challenges when handling tasks that differ from their training data, resulting in compromised performance. To address this, few-shot learning has emerged as a valuable technique, allowing LLMs to adapt with minimal task-specific data. One innovative strategy, known as Chain-of-Thought Prompting (CoT), has been introduced to guide LLMs in revealing cognitive processes during multi-step reasoning. In this paper, we propose Code Chain-of-Thought~(CodeCoT), which consists of two components: the Vanilla CodeCoT and the Self-exam CodeCoT. The latter incorporates self-examination, empowering the model to iteratively generate code, formulate test cases, and refine its outputs. Specifically, the process entails the generation of test examples by the model corresponding to the code it is tasked to implement. If it fails on the test examples, then it regenerates the code based on the erroneous code and associated error types. Through comprehensive experiments, we observed that both techniques significantly enhance code generation accuracy across various LLM variants. Our evaluation results reveal that CodeCoT improves the code generation effectiveness, including an unprecedented pass@1 accuracy of 79.27\% using the Self-exam CodeCoT approach on the gpt-3.5-turbo-0613 model in the HumanEval dataset

Virtual Worlds and the Transformation of the Web to 3D

The notion that the Web will transform into a three-dimensional space and that an avatar will be involved in our travels through this virtual world has been voiced from a range of sources. One such vision is that Second Life or a similar virtual world will grow to become this if a common grid protocol were adopted and anyone who wished could connect a server running a space on a compatible engine. This is, of course, very similar to how the Internet itself grew and Linden Labs has taken some steps in the direction of making this a possibility. An alternate path is offered by a new three-dimensional web browser still in beta called ExitReality. Here an attempt is made to render an existing two-dimensional web site in 3D with some thought given to how a site could be made more three dimensional by its owners. As with virtual worlds, an ExitReality avatar representing the person using the browser is incorporated into the experience and it is possible to encounter others surfing three-dimensionally with the browser as well. This paper explores the feasibility, potential, and challenges to creating a three-dimensional Web. The role that rapidly-expanding social networking on the Internet (Facebook, MySpace, etc.) may play, either in encouraging the development of the three-dimensional Web or modifying the vision by making site-localized virtual rooms like Yoville or Google\u27s Lively the only three dimensionality attached to sites, is explored. Diffusion of innovations theory is applied to address factors that are likely to affect the adoption of a three-dimensional web and which of the possible paths will be most likely to succeed

Aligning the Diversity Requirement in General Education with a broader institutional DEI agenda

In this workshop, we will describe the recent effort currently underway at University of the Pacific to revisit and revise the learning outcomes for courses meeting our Diversity Requirement. We plan to share our process from start to the present, from identifying stakeholders to including student voices, and how we were able to align with university-wide efforts at all levels to arrive where we are today. There will be time to strategize how similar efforts might work at your institution, including how to identify allies, include students, etc. to drive institutional change. Speakers from the University of the Pacific: Qingwen Dong, Professor at University of the Pacific Jeffrey Hole, Association Professor of English & Director of Pacific Humanities Scholars at University of the Pacific ( Angel Zhong, Student Support Services at University of the Pacific Chris Goff, Director of General Education at University of the Pacifi

FMT: Removing Backdoor Feature Maps via Feature Map Testing in Deep Neural Networks

Deep neural networks have been widely used in many critical applications, such as autonomous vehicles and medical diagnosis. However, their security is threatened by backdoor attack, which is achieved by adding artificial patterns to specific training data. Existing defense strategies primarily focus on using reverse engineering to reproduce the backdoor trigger generated by attackers and subsequently repair the DNN model by adding the trigger into inputs and fine-tuning the model with ground-truth labels. However, once the trigger generated by the attackers is complex and invisible, the defender can not successfully reproduce the trigger. Consequently, the DNN model will not be repaired since the trigger is not effectively removed. In this work, we propose Feature Map Testing~(FMT). Different from existing defense strategies, which focus on reproducing backdoor triggers, FMT tries to detect the backdoor feature maps, which are trained to extract backdoor information from the inputs. After detecting these backdoor feature maps, FMT will erase them and then fine-tune the model with a secure subset of training data. Our experiments demonstrate that, compared to existing defense strategies, FMT can effectively reduce the Attack Success Rate (ASR) even against the most complex and invisible attack triggers. Second, unlike conventional defense methods that tend to exhibit low Robust Accuracy (i.e., the model's accuracy on the poisoned data), FMT achieves higher RA, indicating its superiority in maintaining model performance while mitigating the effects of backdoor attacks~(e.g., FMT obtains 87.40\% RA in CIFAR10). Third, compared to existing feature map pruning techniques, FMT can cover more backdoor feature maps~(e.g., FMT removes 83.33\% of backdoor feature maps from the model in the CIFAR10 \& BadNet scenario).Comment: 12 pages, 4 figure

Feature Map Testing for Deep Neural Networks

Due to the widespread application of deep neural networks~(DNNs) in safety-critical tasks, deep learning testing has drawn increasing attention. During the testing process, test cases that have been fuzzed or selected using test metrics are fed into the model to find fault-inducing test units (e.g., neurons and feature maps, activating which will almost certainly result in a model error) and report them to the DNN developer, who subsequently repair them~(e.g., retraining the model with test cases). Current test metrics, however, are primarily concerned with the neurons, which means that test cases that are discovered either by guided fuzzing or selection with these metrics focus on detecting fault-inducing neurons while failing to detect fault-inducing feature maps. In this work, we propose DeepFeature, which tests DNNs from the feature map level. When testing is conducted, DeepFeature will scrutinize every internal feature map in the model and identify vulnerabilities that can be enhanced through repairing to increase the model's overall performance. Exhaustive experiments are conducted to demonstrate that (1) DeepFeature is a strong tool for detecting the model's vulnerable feature maps; (2) DeepFeature's test case selection has a high fault detection rate and can detect more types of faults~(comparing DeepFeature to coverage-guided selection techniques, the fault detection rate is increased by 49.32\%). (3) DeepFeature's fuzzer also outperforms current fuzzing techniques and generates valuable test cases more efficiently.Comment: 12 pages, 5 figures. arXiv admin note: text overlap with arXiv:2307.1101