Mixture of Tokens: Efficient LLMs through Cross-Example Aggregation

Despite the promise of Mixture of Experts (MoE) models in increasing parameter counts of Transformer models while maintaining training and inference costs, their application carries notable drawbacks. The key strategy of these models is to, for each processed token, activate at most a few experts - subsets of an extensive feed-forward layer. But this approach is not without its challenges. The operation of matching experts and tokens is discrete, which makes MoE models prone to issues like training instability and uneven expert utilization. Existing techniques designed to address these concerns, such as auxiliary losses or balance-aware matching, result either in lower model performance or are more difficult to train. In response to these issues, we propose Mixture of Tokens, a fully-differentiable model that retains the benefits of MoE architectures while avoiding the aforementioned difficulties. Rather than routing tokens to experts, this approach mixes tokens from different examples prior to feeding them to experts, enabling the model to learn from all token-expert combinations. Importantly, this mixing can be disabled to avoid mixing of different sequences during inference. Crucially, this method is fully compatible with both masked and causal Large Language Model training and inference

Thermal Performance Analysis of U-pipe Evacuated Tube Solar Heater System with and without PCM

The study presents a comprehensive thermal performance analysis of a U-pipe Evacuated Tube Solar Collector (U-ETSC) system, integrated with Phase Change Material (PCM), for satisfying water demand in domestic applications. The innovative approach of incorporating PCM aims to enhance the system's efficiency during night or off-sunny hours, addressing a common limitation in solar heater systems. The research primarily focuses on the impact of varying mass flow rates of the heat transfer fluid (HTF) on the thermal dynamics of the system. Experimental results reveal that as the mass flow rates of HTF increase from 0.5 to 1 and then to 1.5 liters per minute, there is a notable change in the phase transition times of the PCM. Specifically, the melting time of the PCM is increased by 13% and 16% for the respective flow rates, suggesting a delayed response in energy absorption. Conversely, the solidification time of the PCM is reduced by 16% and 18% respectively, indicating a faster release of stored thermal energy. This behavior underscores the PCM's significant role in stabilizing the system's thermal output during varying solar intensities. The findings of this study highlight the potential of integrating PCM in U-ETSC systems to achieve a more consistent and reliable supply of hot water for domestic purposes, especially during periods when solar irradiance is low or absent

MoE-Mamba: Efficient Selective State Space Models with Mixture of Experts

State Space Models (SSMs) have become serious contenders in the field of sequential modeling, challenging the dominance of Transformers. At the same time, Mixture of Experts (MoE) has significantly improved Transformer-based Large Language Models, including recent state-of-the-art open models. We propose that to unlock the potential of SSMs for scaling, they should be combined with MoE. We showcase this on Mamba, a recent SSM-based model that achieves remarkable performance. Our model, MoE-Mamba, outperforms both Mamba and baseline Transformer-MoE. In particular, MoE-Mamba reaches the same performance as Mamba in $2.35\times$ fewer training steps while preserving the inference performance gains of Mamba against Transformer

A General Review of the Current Development of Mechanically Agitated Vessels

The mixing process in a mechanically agitated vessel is a widespread phenomenon which plays an important role among industrial processes. In that process, one of the crucial parameters, the mixing efficiency, depends on a large number of geometrical factors, as well as process parameters and complex interactions between the phases which are still not well understood. In the last decade, large progress has been made in optimisation, construction and numerical and experimental analysis of mechanically agitated vessels. In this review, the current state in this field has been presented. It shows that advanced computational fluid dynamic techniques for multiphase flow analysis with reactions and modern experimental techniques can be used with success to analyse in detail mixing features in liquid-liquid, gas-liquid, solid-liquid and in more than two-phase flows. The objective is to show the most important research recently carried out

Thermodynamic analysis of a gas turbine combined cycle integration with a high-temperature nuclear reactor

The recently a large number of various companies from the energy sector have to follow new very restrictive regulations which protect the natural environment against pollution and degradation. At the same time, the efficiency of energy conversion and the cost of energy remain important economic aspects. One of the solutions for today's challenges can be power generation based on Gas Turbine Combined Cycle proposed in this paper which is the most efficient and environmentally friendly cycle. What is more important this solution can be easily integrated with a High-Temperature nuclear Reactor. The proposed in this work systems consist of the reactor combined with a gas turbine as well as steam turbine. The results show that it is possible to obtain for proposed cycle an efficiency higher than 50% which is not only more than could be offered by traditional coal power plant but much more than can be proposed by any other nuclear technology

An analysis of the thermodynamic cycles with high-temperature nuclear reactor for power generation and hydrogen co-production

In the present paper, numerical analysis of the thermodynamic cycle with the high-temperature nuclear gas reactor (HTGR) for electricity and hydrogen production have been done. The analysed system consists of two independent loops. The first loop is for HTGR and consists of a nuclear reactor, heat exchangers, and blower. The second loop (Rankine cycle) consist of up-to four steam turbines, that operate in heat recovery system. The analysis of the system shows that it is possible to achieve significantly higher efficiency than could be offered by traditional nuclear reactor technology (PWR and BWR). It is shown that the thermal efficiency about 52.5% it is possible to achieve when reactor works at standard conditions and steam is superheated up to 530oC. For the cases when the steam has supercritical conditions the value of thermal efficiency is still very high and equal about 50%

Analysis of the anti-icing system used in air handling units with a counterflow heat exchanger

The present paper examines one of the most popular anti-icing solutions on the system performance. Different ratios of flowing fresh air to exhaust air were applied in order to heat the exchanger’s iced surface with warm air. The analysed system demonstrated that the system efficiency significantly dropped down under unfavourable conditions in winter. The results show that this type of solution is insufficient and should not be applied. A long operation with the iced surface can cause irreversible permanent damage to the heat exchanger unit and a serious system failure

An analysis of the thermodynamic cycles with high-temperature nuclear reactor for power generation and hydrogen co-production

In the present paper, numerical analysis of the thermodynamic cycle with the high-temperature nuclear gas reactor (HTGR) for electricity and hydrogen production have been done. The analysed system consists of two independent loops. The first loop is for HTGR and consists of a nuclear reactor, heat exchangers, and blower. The second loop (Rankine cycle) consist of up-to four steam turbines, that operate in heat recovery system. The analysis of the system shows that it is possible to achieve significantly higher efficiency than could be offered by traditional nuclear reactor technology (PWR and BWR). It is shown that the thermal efficiency about 52.5% it is possible to achieve when reactor works at standard conditions and steam is superheated up to 530oC. For the cases when the steam has supercritical conditions the value of thermal efficiency is still very high and equal about 50%

Self-Consumption and Self-Sufficiency Improvement for Photovoltaic System Integrated with Ultra-Supercapacitor

This research study uses a computer simulation based on real input data to examine the impact of a supercapacitor module working as a fast response energy storage unit in renewable energy systems to increase energy self-consumption and self-sufficiency. The evaluated system includes a photovoltaic system with a capacity of 3.0 kWp and between 0 and 5 supercapacitor units with a capacity of 500 F per module. The study was carried out using experimental data for electrical load, solar irradiance, and ambient temperature for the year 2020, with a 1 min temporal resolution. The daily average ambient temperature was 10.7 °C, and the daily average solar irradiance was 3.1 kWh/m2/day. It is assumed that the supercapacitor could only be charged from a photovoltaic system using renewable energy and not from the grid. The simulation results showed that using the supercapacitors to feed the short and large peaks of the electrical load significantly increases energy self-consumption and self-sufficiency. With only five supercapacitor modules, yearly energy self-sufficiency increases from 28.09% to 40.77%