Segmented Recurrent Transformer: An Efficient Sequence-to-Sequence Model

Transformers have shown dominant performance across a range of domains including language and vision. However, their computational cost grows quadratically with the sequence length, making their usage prohibitive for resource-constrained applications. To counter this, our approach is to divide the whole sequence into segments and apply attention to the individual segments. We propose a segmented recurrent transformer (SRformer) that combines segmented (local) attention with recurrent attention. The loss caused by reducing the attention window length is compensated by aggregating information across segments with recurrent attention. SRformer leverages Recurrent Accumulate-and-Fire (RAF) neurons' inherent memory to update the cumulative product of keys and values. The segmented attention and lightweight RAF neurons ensure the efficiency of the proposed transformer. Such an approach leads to models with sequential processing capability at a lower computation/memory cost. We apply the proposed method to T5 and BART transformers. The modified models are tested on summarization datasets including CNN-dailymail, XSUM, ArXiv, and MediaSUM. Notably, using segmented inputs of varied sizes, the proposed model achieves $6-22\%$ higher ROUGE1 scores than a segmented transformer and outperforms other recurrent transformer approaches. Furthermore, compared to full attention, the proposed model reduces the computational complexity of cross attention by around $40\%$.Comment: EMNLP 2023 Finding

Kooperativna evolucija za kvalitetno pružanje usluga u paradigmi Interneta stvari

To facilitate the automation process in the Internet of Things, the research issue of distinguishing prospective services out of many “similar” services, and identifying needed services w.r.t the criteria of Quality of Service (QoS), becomes very important. To address this aim, we propose heuristic optimization, as a robust and efficient approach for solving complex real world problems. Accordingly, this paper devises a cooperative evolution approach for service composition under the restrictions of QoS. A series of effective strategies are presented for this problem, which include an enhanced local best first strategy and a global best strategy that introduces perturbations. Simulation traces collected from real measurements are used for evaluating the proposed algorithms under different service composition scales that indicate that the proposed cooperative evolution approach conducts highly efficient search with stability and rapid convergence. The proposed algorithm also makes a well-designed trade-off between the population diversity and the selection pressure when the service compositions occur on a large scale.Kako bi se automatizirali procesi u internetu stvati, nužno je rezlikovati bitne usluge u moru sličnih kao i identificirati potrebne usluge u pogledu kvalitete usluge (QoS). Kako bi doskočili ovome problemu prdlaže se heuristička optimizacija kao robustan i efikasan način rješavajne kompleksnih problema. Nadalje, u članku je predložen postupak kooperativne evolucije za slaganje usluga uz ograničenja u pogledu kvalutete usluge. Predstavljen je niz efektivnih strategija za spomenuti problem uključujući strategije najboljeg prvog i najboljeg globalnog koje unose perturbacije u polazni problem. Simulacijski rezultati kao i stvarni podatci su korišteni u svrhu evaluacije prodloženog algoritma kako bi se osigurala efikasna pretraga uz stabilnost i brzu konvergenciju. Predloženi algoritam tako.er vodi računa o odnosu izme.u različitosti populacije i selekcijskog pritiska kada je potrebno osigurati slaganje usluga na velikoj skali

To Sense or to Transmit: A Learning-Based Spectrum Management Scheme for Cognitive Radiomesh Networks

Abstract—Wireless mesh networks, composed of interconnected clusters of mesh router (MR) and multiple associated mesh clients (MCs), may use cognitive radio equipped transceivers, allowing them to choose licensed frequencies for high bandwidth communication. However, the protection of the licensed users in these bands is a key constraint. In this paper, we propose a reinforcement learning based approach that allows each mesh cluster to independently decide the operative channel, the durations for spectrum sensing, the time of switching, and the duration for which the data transmission happens. The contributions made in this paper are threefold. First, based on accumulated rewards for a channel mapped to the link transmission delays, and the estimated licensed user activity, the MRs assign a weight to each of the channels, thereby selecting the channel with highest performance for MCs operations. Second, our algorithm allows dynamic selection of the sensing time interval that optimizes the link throughput. Third, by cooperative sharing, we allow the MRs to share their channel table information, thus allowing a more accurate learning model. Simulations results reveal significant improvement over classical schemes which have pre-set sensing and transmission durations in the absence of learning. I

Origin of Ferroelectricity in Orthorhombic LuFeO3_3

We demonstrate that small but finite ferroelectric polarization ($\sim$0.01 $\mu$C/cm$^2$) emerges in orthorhombic LuFeO$_3$ ($Pnma$) at $T_N$ ($\sim$600 K) because of commensurate (k = 0) and collinear magnetic structure. The synchrotron x-ray and neutron diffraction data suggest that the polarization could originate from enhanced bond covalency together with subtle contribution from lattice. The theoretical calculations indicate enhancement of bond covalency as well as the possibility of structural transition to the polar $Pna2_1$ phase below $T_N$. The $Pna2_1$ phase, in fact, is found to be energetically favorable below $T_N$ in orthorhombic LuFeO$_3$ ($albeit$ with very small energy difference) than in isostructural and nonferroelectric LaFeO$_3$ or NdFeO$_3$. Application of electric field induces finite piezostriction in LuFeO$_3$ via electrostriction resulting in clear domain contrast images in piezoresponse force microscopy.Comment: 12 pages, 8 figure

Communication protocols for wireless cognitive radio ad-hoc networks

Cognitive radio (CR) technology allows devices to share the wireless spectrum with other users that have a license for operation in these spectrum bands. This area of research promises to solve the problem of spectrum scarcity in the unlicensed bands, and improve the inefficient spectrum utilization in the bands reserved for the licensed users. However, the opportunistic use of the available spectrum by the CR users must not affect the licensed users. This raises several concerns regarding spectrum sensing, sharing and reliable end-to-end communication in CR networks. This thesis is concerned with the design and implementation of communication protocols for the multi-hop infrastructure-less CR ad-hoc networks (CRAHNs). In addition, it also addresses the critical issue of interference-free spectrum usage in specific ad-hoc architectures, such as, resource-constrained wireless sensor networks and wireless mesh networks that have high traffic volumes. The problems of spectrum management that are unique to CR networks are first identified in this thesis. These issues are then addressed at each layer of the network protocol stack while considering the distributed operation in CRAHNs. At the physical layer an algorithmic suite is proposed that allows the CR devices to detect and adapt to the presence of wireless LANs and commercial microwave ovens. A common control channel is designed that allows sharing of the spectrum information between the CR users, even when the available spectrum varies dynamically. A spectrum sharing scheme for mesh networks is proposed at the link layer that allows cooperative detection of the licensed users and fair utilization of the available spectrum among the mesh devices. The spectrum availability and route formation are then considered jointly at the network layer, so that the licensed users are protected as well as the CRAHN performance is maximized. Finally, we extend the classical TCP at the transport layer to ensure end-to-end reliability in a multi-hop CR environment.Ph.D.Committee Chair: Akyildiz, Ian; Committee Member: Ingram, Mary Ann; Committee Member: Blough, Douglas; Committee Member: Dovrolis, Konstantinos; Committee Member: Li, Y