SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization

Transfer learning has fundamentally changed the landscape of natural language processing (NLP) research. Many existing state-of-the-art models are first pre-trained on a large text corpus and then fine-tuned on downstream tasks. However, due to limited data resources from downstream tasks and the extremely large capacity of pre-trained models, aggressive fine-tuning often causes the adapted model to overfit the data of downstream tasks and forget the knowledge of the pre-trained model. To address the above issue in a more principled manner, we propose a new computational framework for robust and efficient fine-tuning for pre-trained language models. Specifically, our proposed framework contains two important ingredients: 1. Smoothness-inducing regularization, which effectively manages the capacity of the model; 2. Bregman proximal point optimization, which is a class of trust-region methods and can prevent knowledge forgetting. Our experiments demonstrate that our proposed method achieves the state-of-the-art performance on multiple NLP benchmarks.Comment: The 58th annual meeting of the Association for Computational Linguistics (ACL 2020

Outlook for the coal industry and new coal production technologies

Historically, energy resources have evolved from high carbon to lower carbon fuels (from coal to oil to natural gas), then to non-carbon (hydroelectric, geothermal, wind power and solar). This dynamic process has reflected the evolution of human civilization and industrialization. As one of the most useful and classical energy resources, what is the outlook for the coal industry in the future? What factors will have a great impact on the outlook? Can new technologies in coal production make the coal industry cleaner and more competitive and increase its demand in the world market? How effective are CO2 capture technologies for coal power plants? This editorial work attempts to provide insights into these issues.1. The future outlook for the coal industryLimiting global warming to 2 ◦C versus pre-industrial levels would imply reducing carbon dioxide (CO2) emissions by 80% of the 1990 level by 2050 and a net-zero emission by the end of this century. To make this happen requires a fast transition of traditional fossil fuels to renewable energy such as wind and solar. In this framework, coal demand is expected to decline by about 8% by 2030 compared to the pre-crisis level in 2019. Advanced economies will cut their demand by 45% compared to 2019. China is still the largest consumer and producer, and the coal usage in China is expected to rebound in the near term and achieve its peak around 2025 followed by a gradual decline after 2025. In the Asia Pacific area, India, Indonesia and Southeast Asian countries will increase their coal demand for power and industrial usage in the next decade (IEA, 2020). By 2030 the global coal demand is projected to decrease by about 400 million tonnes of coal equivalent compared to 2019.2. The impact of major factors on the outlookEnvironmental concerns to reduce CO2 emissions from coal-fired plants, declining coal usage in the power sector due to renewables expansion and a cheap natural gas price, and policies that phase out coal to achieve carbon neutralities are the three major factors leading to the coal market downturn. The power sector accounts for nearly 65% of coal demand. By 2030, advanced economies will consume 50% less coal compared to their level in 2019 mainly due to policy-driven requirements. The increasing supply of renewables and natural gas has diversified the power generation sources, and significantly reduced the coal usage in this sector. Coal is the largest source of CO2 emissions and will be responsible for approximately 38% of the global CO2 emissions from 2020 to 2030. The policy-driven changes are significantly affecting coal usage worldwide (IEA, 2020; IMF, 2020).3. Development of new technologies in the coal industryPotential technologies are emerging in the coal industry, such as hydraulic fracturing to improve the coalbed methane production; the CO2 capture, utilization and sequestration (CCUS) to reduce CO2 emissions from coal combustion; the coal-to-liquid and coal-to-gas fuel conversion technologies to improve the fuel efficiency and reduce the CO2 emissions; internet of things, big data analytics, artificial intelligence, and automation to reduce operational costs and improve safety concerns and production efficiency in coal operations; and, underground coal gasification (UCG) to recover unminable coal. Particularly, UCG has been attempted for over a century, and has not yet achieved a commercial-scale development. Successful development and utilization of this technology would make the coal industry more competitive and increase its demand in the world market. A UCG operation consists of a series of injection and production wells drilled into a coal seam; the coal is ignited after certain air and/or oxygen is injected. Chemical reactions convert the coal to syngas by pyrolysis, combustion and gasification reactions in a manner similar to those processes in a surface gasifier. The produced syngas is a mixture of mainly carbon monoxide and hydrogen, which can be used as fuel for power generation and feedstock for various chemical products (i.e., hydrogen and ammonia) (Nourozieh et al., 2010; Seifi et al., 2015). The carbon captured during syngas utilization can be used for enhanced oil recovery.Emissions from syngas combustion are generally cleaner and less greenhouse gas emissions than coal-fired facilities. The UCG process is less costly than conventional surficial coal gasification because no coal mining, processing and transport are required, and no ash and slag removal or disposal is necessary. The environmental impact of UCG is relatively low compared to surficial gasification, as major disturbances in landscape and surface disposal of ash and coal tailings are not required. A properly designed UCG site will recognize and address potential groundwater pollution and subsidence issues; tests related to the cap rock integrity and highly cemented wells should be performed to avoid these issues. UCG can have obvious advantages compared with other in situ coal applications, including mining, coalbed methane exploration and development; the cavities after UCG can be used as CO2 storage (Jiang et al., 2019).4. CO2 capture technologies for coal power plantsFive main types of CO2 capture technologies from flue gas are proven. The average capture efficiency is from 80% to 90%. Cryogenic separation can provide the highest capture efficiency up to 99.99%. The CO2 capture step represents 70-80% of the overall CO2 capture and sequestration costs. Economic analysis shows that US $70-100 are needed to capture one tonne CO2 from flue gas (the average CO2 concentration is about 3-14$300 to $1,500 to capture CO2 directly from air (Brandl et al., 2021). It would be possible to commercialize the capture efficiency beyond 90%. However, operators are not able to make benefits under the current policies because of the high associated  capital and operational costs. The capture approaches include post-combustion, pre-combustion, oxyfuel combustion, chemical looping combustion and from air. The post-combustion technology is a mature technology, and has been widely applied. Future CO2 capture technologies are likely to focus on hybrid capture technologies, such as integrated CO2 capture and conversion. The coal use in the power sector surpassed 10 Gt CO2 emissions globally in 2018, and a successful application of carbon capture technologies can help the world reduce up to 8-10 Gt CO2 emissions annually. CCUS has received much research attention over the past two decades.Cited as: Ma, H., Chen, S., Xue, D., Chen, Y., Chen, Z. Outlook for the coal industry and new coal production technologies. Advances in Geo-Energy Research, 2021, 5(2): 119-120, doi: 10.46690/ager.2021.02.0

Selfishness in device-to-device communication underlaying cellular networks

In a device-to-device (D2D) communication underlaying cellular network, user equipments are required to operate cooperatively and unselfishly to transmit data as relays. However, most users behave in a more or less selfish way, which makes user selfishness a key factor that affects the performance of the whole communication system. We focus on the impact of node selfishness on D2D communications. By separating the user selfishness into two types in accordance with two D2D transmission modes – connected D2D transmission and opportunistic D2D transmission, we propose a time-varying graph model that characterizes the impacts of both individual and social selfishness on D2D communications. Simulation results obtained under the realistic networking settings indicate that the interaction between connected and opportunistic selfishness worsens the impairment caused by individual selfishness, while the harmful interaction caused by social selfishness can be alleviated