Enriching Source Code with Contextual Data for Code Completion Models: An Empirical Study

Transformer-based pre-trained models have recently achieved great results in solving many software engineering tasks including automatic code completion which is a staple in a developer's toolkit. While many have striven to improve the code-understanding abilities of such models, the opposite -- making the code easier to understand -- has not been properly investigated. In this study, we aim to answer whether making code easier to understand through using contextual data improves the performance of pre-trained code language models for the task of code completion. We consider type annotations and comments as two common forms of additional contextual information that often help developers understand code better. For the experiments, we study code completion in two granularity levels; token and line completion and take three recent and large-scale language models for source code: UniXcoder, CodeGPT, and InCoder with five evaluation metrics. Finally, we perform the Wilcoxon Signed Rank test to gauge significance and measure the effect size. Contrary to our expectations, all models perform better if type annotations are removed (albeit the effect sizes are small). For comments, we find that the models perform better in the presence of multi-line comments (again with small effect sizes). Based on our observations, we recommend making proper design choices when training, fine-tuning, or simply selecting such models given the intended data and application. Better evaluations and multi-modal techniques can also be further investigated to improve the practicality and accuracy of auto-completions.Comment: 13 pages. To appear in the Proceedings of the 20th International Conference on Mining Software Repositories (MSR 2023

Uncovering Energy-Efficient Practices in Deep Learning Training:Preliminary Steps Towards Green AI

Modern AI practices all strive towards the same goal: better results. In the context of deep learning, the term "results"often refers to the achieved accuracy on a competitive problem set. In this paper, we adopt an idea from the emerging field of Green AI to consider energy consumption as a metric of equal importance to accuracy and to reduce any irrelevant tasks or energy usage. We examine the training stage of the deep learning pipeline from a sustainability perspective, through the study of hyperparameter tuning strategies and the model complexity, two factors vastly impacting the overall pipeline's energy consumption. First, we investigate the effectiveness of grid search, random search and Bayesian optimisation during hyperparameter tuning, and we find that Bayesian optimisation significantly dominates the other strategies. Furthermore, we analyse the architecture of convolutional neural networks with the energy consumption of three prominent layer types: convolutional, linear and ReLU layers. The results show that convolutional layers are the most computationally expensive by a strong margin. Additionally, we observe diminishing returns in accuracy for more energy-hungry models. The overall energy consumption of training can be halved by reducing the network complexity. In conclusion, we highlight innovative and promising energy-efficient practices for training deep learning models. To expand the application of Green AI, we advocate for a shift in the design of deep learning models, by considering the trade-off between energy efficiency and accuracy.</p

Batching for Green AI -- An Exploratory Study on Inference

The batch size is an essential parameter to tune during the development of new neural networks. Amongst other quality indicators, it has a large degree of influence on the model's accuracy, generalisability, training times and parallelisability. This fact is generally known and commonly studied. However, during the application phase of a deep learning model, when the model is utilised by an end-user for inference, we find that there is a disregard for the potential benefits of introducing a batch size. In this study, we examine the effect of input batching on the energy consumption and response times of five fully-trained neural networks for computer vision that were considered state-of-the-art at the time of their publication. The results suggest that batching has a significant effect on both of these metrics. Furthermore, we present a timeline of the energy efficiency and accuracy of neural networks over the past decade. We find that in general, energy consumption rises at a much steeper pace than accuracy and question the necessity of this evolution. Additionally, we highlight one particular network, ShuffleNetV2(2018), that achieved a competitive performance for its time while maintaining a much lower energy consumption. Nevertheless, we highlight that the results are model dependent.Comment: 8 pages, 4 figures, 1 table. Accepted at Euromicro Conference Series on Software Engineering and Advanced Applications (SEAA) 202

Real‐time alerts from AI‐enabled camera traps using the Iridium satellite network: A case‐study in Gabon, Central Africa

Efforts to preserve, protect and restore ecosystems are hindered by long delays between data collection and analysis. Threats to ecosystems can go undetected for years or decades as a result. Real-time data can help solve this issue but significant technical barriers exist. For example, automated camera traps are widely used for ecosystem monitoring but it is challenging to transmit images for real-time analysis where there is no reliable cellular or WiFi connectivity. We modified an off-the-shelf camera trap (Bushnell™) and customised existing open-source hardware to create a ‘smart’ camera trap system. Images captured by the camera trap are instantly labelled by an artificial intelligence model and an ‘alert’ containing the image label and other metadata is then delivered to the end-user within minutes over the Iridium satellite network. We present results from testing in the Netherlands, Europe, and from a pilot test in a closed-canopy forest in Gabon, Central Africa. All reference materials required to build the system are provided in open-source repositories. Results show the system can operate for a minimum of 3 months without intervention when capturing a median of 17.23 images per day. The median time-difference between image capture and receiving an alert was 7.35 min, though some outliers showed delays of 5-days or more when the system was incorrectly positioned and unable to connect to the Iridium network. We anticipate significant developments in this field and hope that the solutions presented here, and the lessons learned, can be used to inform future advances. New artificial intelligence models and the addition of other sensors such as microphones will expand the system's potential for other, real-time use cases including real-time biodiversity monitoring, wild resource management and detecting illegal human activities in protected areas

Real‐time alerts from AI‐enabled camera traps using the Iridium satellite network: A case‐study in Gabon, Central Africa

Efforts to preserve, protect and restore ecosystems are hindered by long delays between data collection and analysis. Threats to ecosystems can go undetected for years or decades as a result. Real‐time data can help solve this issue but significant technical barriers exist. For example, automated camera traps are widely used for ecosystem monitoring but it is challenging to transmit images for real‐time analysis where there is no reliable cellular or WiFi connectivity.We modified an off‐the‐shelf camera trap (Bushnell™) and customised existing open‐source hardware to create a ‘smart’ camera trap system. Images captured by the camera trap are instantly labelled by an artificial intelligence model and an ‘alert’ containing the image label and other metadata is then delivered to the end‐user within minutes over the Iridium satellite network. We present results from testing in the Netherlands, Europe, and from a pilot test in a closed‐canopy forest in Gabon, Central Africa. All reference materials required to build the system are provided in open‐source repositories.Results show the system can operate for a minimum of 3 months without intervention when capturing a median of 17.23 images per day. The median time‐difference between image capture and receiving an alert was 7.35 min, though some outliers showed delays of 5‐days or more when the system was incorrectly positioned and unable to connect to the Iridium network.We anticipate significant developments in this field and hope that the solutions presented here, and the lessons learned, can be used to inform future advances. New artificial intelligence models and the addition of other sensors such as microphones will expand the system's potential for other, real‐time use cases including real‐time biodiversity monitoring, wild resource management and detecting illegal human activities in protected areas

Real-time alerts from AI-enabled camera traps using the Iridium satellite network: a case-study in Gabon, Central Africa

Efforts to preserve, protect, and restore ecosystems are hindered by long delays between data collection and analysis. Threats to ecosystems can go undetected for years or decades as a result. Real-time data can help solve this issue but significant technical barriers exist. For example, automated camera traps are widely used for ecosystem monitoring but it is challenging to transmit images for real-time analysis where there is no reliable cellular or WiFi connectivity. Here, we present our design for a camera trap with integrated artificial intelligence that can send real-time information from anywhere in the world to end-users. We modified an off-the-shelf camera trap (Bushnell) and customised existing open-source hardware to rapidly create a 'smart' camera trap system. Images captured by the camera trap are instantly labelled by an artificial intelligence model and an 'alert' containing the image label and other metadata is then delivered to the end-user within minutes over the Iridium satellite network. We present results from testing in the Netherlands, Europe, and from a pilot test in a closed-canopy forest in Gabon, Central Africa. Results show the system can operate for a minimum of three months without intervention when capturing a median of 17.23 images per day. The median time-difference between image capture and receiving an alert was 7.35 minutes. We show that simple approaches such as excluding 'uncertain' labels and labelling consecutive series of images with the most frequent class (vote counting) can be used to improve accuracy and interpretation of alerts. We anticipate significant developments in this field over the next five years and hope that the solutions presented here, and the lessons learned, can be used to inform future advances. New artificial intelligence models and the addition of other sensors such as microphones will expand the system's potential for other, real-time use cases. Potential applications include, but are not limited to, wildlife tourism, real-time biodiversity monitoring, wild resource management and detecting illegal human activities in protected areas

Enriching Source Code with Contextual Data for Code Completion Models: An Empirical Study

Transformer-based pre-trained models have recently achieved great results in solving many software engineering tasks including automatic code completion which is a staple in a developer’s toolkit. While many have striven to improve the code-understanding abilities of such models, the opposite – making the code easier to understand – has not been properly investigated. In this study, we aim to answer whether making code easier to understand through using contextual data improves the performance of pre-trained code language models for the task of code completion. We consider type annotations and comments as two common forms of additional contextual information that often help developers understand code better. For the experiments, we study code completion in two granularity levels; token and line completion and take three recent and large-scale language models for source code: UniXcoder, CodeGPT, and InCoder with five evaluation metrics. Finally, we perform the Wilcoxon Signed Rank test to gauge significance and measure the effect size. Contrary to our expectations, all models perform better if type annotations are removed (albeit the effect sizes are small). For comments, we find that the models perform better in the presence of multi-line comments (again with small effect sizes). Based on our observations, we recommend making proper design choices when training, fine-tuning, or simply selecting such models given the intended data and application. Better evaluations and multimodal techniques can also be further investigated to improve the practicality and accuracy of auto-completions.Green Open Access added to TU Delft Institutional Repository 'You share, we take care!' - Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.Software EngineeringSoftware Technolog

Three-dimensional information exchange over the semantic web for the domain of architecture, engineering, and construction

3D geometry can be described in many ways, with both a varying syntax and a varying semantics. As a result, several very diverse schemas and file formats can be deployed to describe geometry, depending on the application domain in question. In a multidisciplinary domain such as the domain of architecture, engineering and construction (AEC), this range of specialised schemas makes file format conversions inevitable. The approach adopted by current conversion tools, however, often results in a loss of information, most often due to a ‘mistranslation’ between different syntaxes and/or semantics, leading to errors and limitations in the design conception stage and to inefficiency due to the required remodelling efforts. An approach based on semantic web technology may reduce the loss of information significantly, leading to an improved processing of 3D information and hence to an improved design practice in the AEC domain. This paper documents our investigation of the nature of this 3D information conversion problem and how it may be encompassed using semantic web technology. In a double test case, we show how the specific deployment of semantic rule languages and an appropriate inference engine are to be adopted to improve this 3D information exchange. It shows how semantic web technology allows the coexistence of diverse descriptions of the same 3D information, interlinked through explicit conversion rules. Although only a simple example is used to document the process, and a more in-depth investigation is needed, the initial results indicate the suggested approach to be a useful alternative approach to obtain an improved 3D information exchange