A Comparative Analysis of Distributed Training Strategies for GPT-2

The rapid advancement in Large Language Models has been met with significant challenges in their training processes, primarily due to their considerable computational and memory demands. This research examines parallelization techniques developed to address these challenges, enabling the efficient and scalable training of Large Language Models. A comprehensive analysis of both data and model parallelism strategies, including Fully Sharded Data Parallelism and Distributed Data-Parallel frameworks, is provided to assess methods that facilitate efficient model training. Furthermore, the architectural complexities and training methodologies of the Generative Pre-Trained Transformer-2 model are explored. The application of these strategies is further investigated, which is crucial in managing the substantial computational and memory demands of training sophisticated models. This analysis not only highlights the effectiveness of these parallel training strategies in enhancing training efficiency but also their role in enabling the scalable training of large language models. Drawing on recent research findings, through a comprehensive literature review, this research underscores the critical role of parallelization techniques in addressing the computational challenges of training state-of-the-art Large Language Models, thereby contributing to the advancement of training more sophisticated and capable artificial intelligence systems.Comment: Submitted to the International Journal of Parallel Programming and is currently under revie

Humanity's Last Exam

Benchmarks are important tools for tracking the rapid advancements in large language model (LLM) capabilities. However, benchmarks are not keeping pace in difficulty: LLMs now achieve over 90\% accuracy on popular benchmarks like MMLU, limiting informed measurement of state-of-the-art LLM capabilities. In response, we introduce Humanity's Last Exam (HLE), a multi-modal benchmark at the frontier of human knowledge, designed to be the final closed-ended academic benchmark of its kind with broad subject coverage. HLE consists of 3,000 questions across dozens of subjects, including mathematics, humanities, and the natural sciences. HLE is developed globally by subject-matter experts and consists of multiple-choice and short-answer questions suitable for automated grading. Each question has a known solution that is unambiguous and easily verifiable, but cannot be quickly answered via internet retrieval. State-of-the-art LLMs demonstrate low accuracy and calibration on HLE, highlighting a significant gap between current LLM capabilities and the expert human frontier on closed-ended academic questions. To inform research and policymaking upon a clear understanding of model capabilities, we publicly release HLE at https://lastexam.ai

Enablement of digital twins for railway overhead catenary system

Railway has the potential to become one of the most sustainable mediums for passenger and freight transport. This is possible by continuous updates to the asset management regime supporting Prognostics and Health Management (PHM). Railway tracks and catenaries are linear assets, and their length plays a vital role in maintenance. Railway catenary does not present many failures as compared to the rail track, but the failures that occur do not give enough opportunity for quick recovery. These failures cause extensive time delays disrupting railways operations. Such situations can be handled better by updating the maintenance approach. The domain of maintenance explores possible tools, techniques, and technologies to retain and restore the systems. PHM is dependent on data acquisition and analytics to predict the future state of a system with the least possible divergence. In the case of railway catenary and many other domains, this new technology of data acquisition is Light Detection And Ranging (LiDAR) device-based spatial point cloud collection. Current methods of catenary inspection depend on contact-based methods of inspection of railway catenary and read signals from the pantograph and contact wire while ignoring the rest of the wires and surroundings. Locomotive-mounted LiDAR devices support the collection of spatial data in the form of point-cloud from all the surrounding equipment and environment. This point cloud data holds a large amount of information, waiting for algorithms and technologies to harness it. A Digital Twin (DT) is a virtual representation of a physical system or process, achieved through models and simulations and maintains bidirectional communication for progressive enrichment at both ends. A systems digital twin is exposed to all the same conditions virtually. Such a digital twin can be used to provide prognostics by varying factors such as time, malfunction in components of the system, and conditions in which the system operates. Railways is a multistakeholder domain that depends on many organisations to support smooth function. The development of digital twins depends on the understanding of the system, the availability of sensors to read the state and actuators to affect the system’s state. Enabling a digital twin depends on governance restrictions, business requirements and technological competence. A concrete step towards enablement of the digital twin is designing an architecture to accommodate the technical requirements of content management, processing and infrastructure while addressing railway operations' governance and business aspects.The main objective of this work is to develop and provide architecture and a platform for the enablement of a DT solution based on Artificial Intelligence (AI) and digital technologies aimed at PHM of railway catenary system. The main results of this thesis are i) analysis of content management and processing requirements for railway overhead catenary system ii) methodology for catenary point cloud data processing and information representation iii) architecture and infrastructure requirements for enablement of Digital Twin and iv) roadmap for digital twin enablement for PHM of railway overhead catenary system

A Novel Approach to Developing Digital Twins in Maintenance Utilising Industrial Artificial Intelligence

Industrial assets have become increasingly complex to support the requirements of quality, productivity, and cost-effectiveness. The industrial needs and requirements to effectively and efficiently operate, maintain and manage the complex technical industrial assets have propelled the advancement of technology. Digitalisation has been one of the significant enablers for operating, maintaining, and managing such complex technical assets. The operations of an organisation are significantly influenced by asset management. It is characterised as the means through which an organisation can derive value from its assets to meet its goals. When managing complex technical System-of-Systems, maintenance plays an essential role to ensure that the delivered function of the system fulfils the requirements. An efficient maintenance process helps detect potential problems early, preventing them from becoming significant failures and reducing costly downtime. By keeping assets in optimal condition, organisations can enhance reliability and performance, which is crucial for achieving business and operational objectives, as well as meeting regulatory requirements. Traditional maintenance planning methods are inadequate for linear assets because of their extended lifespan and varying conditions. A more effective approach is needed to address RAMS, criticality, resilience, and sustainability cost-effectively throughout the asset's lifespan. Linear assets refer to infrastructure that spans over large geographical areas, such as high-tension power cables, railway overhead catenary, pipelines, highways, and underground mining drifts. These assets are difficult to maintain and often lack a comprehensive digital footprint due to absence of appropriate sensors and data processing techniques. This research aims to address these challenges by adapting techniques from cyber-physical systems and development of Digital Twins (DT) for linear assets. To manage the inherent complexity System-of-Systems approach has been employed during the development process. The primary focus of this research is on spatial condition monitoring and health management of linear assets through maintenance decisions and decision support tools, with emphasis on railway overhead catenary and underground mining drifts. However, the advancement of Artificial Intelligence (AI) and digital technologies facilitates the creation of solutions that are anticipated to improve business processes, asset management, and the operation and maintenance of industries. Technological advancements, especially AI represented by Digital Twins, have the potential to revolutionise business processes, operational strategies, and maintenance practices, thereby leading to operational excellence. Hence, the research aims to enhance the maintenance of linear assets through the development of Digital Twins (DT) empowered by digital technologies and Artificial Intelligence (AI)

Enablement of digital twins for railway overhead catenary system

Railway has the potential to become one of the most sustainable mediums for passenger and freight transport. This is possible by continuous updates to the asset management regime supporting Prognostics and Health Management (PHM). Railway tracks and catenaries are linear assets, and their length plays a vital role in maintenance. Railway catenary does not present many failures as compared to the rail track, but the failures that occur do not give enough opportunity for quick recovery. These failures cause extensive time delays disrupting railways operations. Such situations can be handled better by updating the maintenance approach. The domain of maintenance explores possible tools, techniques, and technologies to retain and restore the systems. PHM is dependent on data acquisition and analytics to predict the future state of a system with the least possible divergence. In the case of railway catenary and many other domains, this new technology of data acquisition is Light Detection And Ranging (LiDAR) device-based spatial point cloud collection. Current methods of catenary inspection depend on contact-based methods of inspection of railway catenary and read signals from the pantograph and contact wire while ignoring the rest of the wires and surroundings. Locomotive-mounted LiDAR devices support the collection of spatial data in the form of point-cloud from all the surrounding equipment and environment. This point cloud data holds a large amount of information, waiting for algorithms and technologies to harness it. A Digital Twin (DT) is a virtual representation of a physical system or process, achieved through models and simulations and maintains bidirectional communication for progressive enrichment at both ends. A systems digital twin is exposed to all the same conditions virtually. Such a digital twin can be used to provide prognostics by varying factors such as time, malfunction in components of the system, and conditions in which the system operates. Railways is a multistakeholder domain that depends on many organisations to support smooth function. The development of digital twins depends on the understanding of the system, the availability of sensors to read the state and actuators to affect the system’s state. Enabling a digital twin depends on governance restrictions, business requirements and technological competence. A concrete step towards enablement of the digital twin is designing an architecture to accommodate the technical requirements of content management, processing and infrastructure while addressing railway operations' governance and business aspects.The main objective of this work is to develop and provide architecture and a platform for the enablement of a DT solution based on Artificial Intelligence (AI) and digital technologies aimed at PHM of railway catenary system. The main results of this thesis are i) analysis of content management and processing requirements for railway overhead catenary system ii) methodology for catenary point cloud data processing and information representation iii) architecture and infrastructure requirements for enablement of Digital Twin and iv) roadmap for digital twin enablement for PHM of railway overhead catenary system

A Novel Approach to Developing Digital Twins in Maintenance Utilising Industrial Artificial Intelligence

Industrial assets have become increasingly complex to support the requirements of quality, productivity, and cost-effectiveness. The industrial needs and requirements to effectively and efficiently operate, maintain and manage the complex technical industrial assets have propelled the advancement of technology. Digitalisation has been one of the significant enablers for operating, maintaining, and managing such complex technical assets. The operations of an organisation are significantly influenced by asset management. It is characterised as the means through which an organisation can derive value from its assets to meet its goals. When managing complex technical System-of-Systems, maintenance plays an essential role to ensure that the delivered function of the system fulfils the requirements. An efficient maintenance process helps detect potential problems early, preventing them from becoming significant failures and reducing costly downtime. By keeping assets in optimal condition, organisations can enhance reliability and performance, which is crucial for achieving business and operational objectives, as well as meeting regulatory requirements. Traditional maintenance planning methods are inadequate for linear assets because of their extended lifespan and varying conditions. A more effective approach is needed to address RAMS, criticality, resilience, and sustainability cost-effectively throughout the asset's lifespan. Linear assets refer to infrastructure that spans over large geographical areas, such as high-tension power cables, railway overhead catenary, pipelines, highways, and underground mining drifts. These assets are difficult to maintain and often lack a comprehensive digital footprint due to absence of appropriate sensors and data processing techniques. This research aims to address these challenges by adapting techniques from cyber-physical systems and development of Digital Twins (DT) for linear assets. To manage the inherent complexity System-of-Systems approach has been employed during the development process. The primary focus of this research is on spatial condition monitoring and health management of linear assets through maintenance decisions and decision support tools, with emphasis on railway overhead catenary and underground mining drifts. However, the advancement of Artificial Intelligence (AI) and digital technologies facilitates the creation of solutions that are anticipated to improve business processes, asset management, and the operation and maintenance of industries. Technological advancements, especially AI represented by Digital Twins, have the potential to revolutionise business processes, operational strategies, and maintenance practices, thereby leading to operational excellence. Hence, the research aims to enhance the maintenance of linear assets through the development of Digital Twins (DT) empowered by digital technologies and Artificial Intelligence (AI)

LiDAR data processing for railway catenary digital twin enablement

Prognostics and health management enables predictive maintenance through techniques like data analytics. Railway catenary is categorised as a linear asset, where inspection and maintenance present challenges due to large distribution of the asset and limitation of current methods. Digital twin can be used to support system level analytics from design to decommissioning. Development of digital twin for railway catenary requires data analytics as well as strategy for information and knowledge storage. Point cloud data recovered through LiDAR scanning contains spatial information, Point cloud data analytics and representation of extracted information forms the base for development of railway catenary digital twin.</p

Health Monitoring of Ground Support System Through Point-Cloud Processing: Rockbolts Extraction Phase

Safety in underground mining operations relies on understanding the geological and geotechnical properties of the site. The creation of an underground void for mining induces instability in the rock structure, resulting in deformation. The compressive strength of the rocks is maintained by the tensioning of ground support, such as rockbolts. Monitoring and predicting the condition of the mining ground support system is crucial for ensuring the safety of operations. Inspecting the mining tunnels poses challenges due to their large span and ongoing production activities. Light Detection and Ranging (LiDAR) technology can scan physical structures and generate point cloud data, which is valuable for creating applications like topographic mapping and spatial models. Extracting rockbolt information from point cloud data from underground mines can offer comprehensive mine coverage. This information can be utilised to monitor the condition of the rockbolts over time. Extracted rockbolt data can assist in the health monitoring of ground support, indicating deformation due to geostatic pressure. This paper proposes a method for extracting rockbolt spatial information from point cloud datasets collected via LiDAR technology to facilitate Prognostics and Health Management for ground support in underground mining.Funder: Mining Innovation for Ground Support (MIGS);Fulltext license: CC BY;This article has previously appeared as a manuscript in a thesis. </p