When Training-Free NAS Meets Vision Transformer: A Neural Tangent Kernel Perspective

This paper investigates the Neural Tangent Kernel (NTK) to search vision transformers without training. In contrast with the previous observation that NTK-based metrics can effectively predict CNNs performance at initialization, we empirically show their inefficacy in the ViT search space. We hypothesize that the fundamental feature learning preference within ViT contributes to the ineffectiveness of applying NTK to NAS for ViT. We both theoretically and empirically validate that NTK essentially estimates the ability of neural networks that learn low-frequency signals, completely ignoring the impact of high-frequency signals in feature learning. To address this limitation, we propose a new method called ViNTK that generalizes the standard NTK to the high-frequency domain by integrating the Fourier features from inputs. Experiments with multiple ViT search spaces on image classification and semantic segmentation tasks show that our method can significantly speed up search costs over prior state-of-the-art NAS for ViT while maintaining similar performance on searched architectures.Comment: ICASSP2024 ora

Monitorización de entornos mediante plataforma de análisis de IoT

Durante los últimos años, el IoT ha llegado a su máximo esplendor, llegando hasta una cantidad de dispositivos nunca antes alcanzado, logrando una cantidad de 11 billones de dispositivos IoT conectados según estudios del instituto de investigación 451 Research. Hoy en día estamos rodeados de estos dispositivos, y esto hace que tengamos la oportunidad de realizar una conexión entre ellos, automatizar sus procesos según nuestras necesidades, monitorizar los datos obtenidos por estos y hacer un análisis de los datos para una toma de decisión más acertada. En este trabajo, decidimos seleccionar el entorno de un patrimonio cultural y/o centro de trabajo, en concreto el de Torre Juana. Para dicho entorno se implementará una plataforma donde se refleje la monitorización de los datos de las diferentes métricas obtenidas por los sensores IoT que se hallan repartidos por el centro. Con el objetivo de convertir los datos de bajo nivel (datos directos del sensor) en información que sea de más utilidad por parte de los propietarios del centro. Para conseguir este objetivo, primeramente tenemos que capturar los datos producidos por los sensores y diseñar un almacén de datos con la estructura adecuada detallando los atributos y las métricas que necesitamos. El siguiente paso son los procesos ETL, este paso es especialmente crítico en los sistemas de IoT, ya que es donde se realiza todo el proceso de la transformación de unos datos crudos de bajo nivel, que resultan inapropiados y de difícil interpretación por parte del usuario final; en información relevante con valores extras añadidos. Finalmente, estos datos son guardados en el almacén de datos para su posterior análisis y visualización. El último paso y no menos importante, es la visualización de los datos, consiste en crear grafos y gráficas llamativas con el objetivo de ayudar a nuestros usuarios finales a entender de una forma más rápida y cómoda los datos transformados del Datawarehouse. El resultado que hemos conseguido ha sido una plataforma con una monitorización del estado del ambiente del centro (temperatura, humedad, luminosidad, etc.); un ahorro energético, se tiene un mejor control sobre los dispositivos y esto conlleva un uso más adecuado de estos recursos; y un mejor cuidado de las plantas del huerto, teniendo en cuenta medidas como por ejemplo la temperatura del suelo, la humedad de la tierra, etc. Este trabajo ha sido apoyado por el proyecto AETHER-UA (PID 2020-112540RB-C43), financiado por el Ministerio de Ciencia e Innovación, y apoyado por el proyecto BALLADEER (PROMETEO/2021/088), financiado por la Conselleria de Innovación, Universidades, Ciencia y Sociedad Digital (Generalitat Valenciana)

Cooperative Coevolution for Non-Separable Large-Scale Black-Box Optimization: Convergence Analyses and Distributed Accelerations

Given the ubiquity of non-separable optimization problems in real worlds, in this paper we analyze and extend the large-scale version of the well-known cooperative coevolution (CC), a divide-and-conquer optimization framework, on non-separable functions. First, we reveal empirical reasons of why decomposition-based methods are preferred or not in practice on some non-separable large-scale problems, which have not been clearly pointed out in many previous CC papers. Then, we formalize CC to a continuous game model via simplification, but without losing its essential property. Different from previous evolutionary game theory for CC, our new model provides a much simpler but useful viewpoint to analyze its convergence, since only the pure Nash equilibrium concept is needed and more general fitness landscapes can be explicitly considered. Based on convergence analyses, we propose a hierarchical decomposition strategy for better generalization, as for any decomposition there is a risk of getting trapped into a suboptimal Nash equilibrium. Finally, we use powerful distributed computing to accelerate it under the multi-level learning framework, which combines the fine-tuning ability from decomposition with the invariance property of CMA-ES. Experiments on a set of high-dimensional functions validate both its search performance and scalability (w.r.t. CPU cores) on a clustering computing platform with 400 CPU cores

Uncertainty-Aware Consistency Regularization for Cross-Domain Semantic Segmentation

Unsupervised domain adaptation (UDA) aims to adapt existing models of the source domain to a new target domain with only unlabeled data. Many adversarial-based UDA methods involve high-instability training and have to carefully tune the optimization procedure. Some non-adversarial UDA methods employ a consistency regularization on the target predictions of a student model and a teacher model under different perturbations, where the teacher shares the same architecture with the student and is updated by the exponential moving average of the student. However, these methods suffer from noticeable negative transfer resulting from either the error-prone discriminator network or the unreasonable teacher model. In this paper, we propose an uncertainty-aware consistency regularization method for cross-domain semantic segmentation. By exploiting the latent uncertainty information of the target samples, more meaningful and reliable knowledge from the teacher model can be transferred to the student model. In addition, we further reveal the reason why the current consistency regularization is often unstable in minimizing the distribution discrepancy. We also show that our method can effectively ease this issue by mining the most reliable and meaningful samples with a dynamic weighting scheme of consistency loss. Experiments demonstrate that the proposed method outperforms the state-of-the-art methods on two domain adaptation benchmarks, $i.e.,$ GTAV $\rightarrow$ Cityscapes and SYNTHIA $\rightarrow$ Cityscapes

Integrated siRNA design based on surveying of features associated with high RNAi effectiveness

BACKGROUND: Short interfering RNAs have allowed the development of clean and easily regulated methods for disruption of gene expression. However, while these methods continue to grow in popularity, designing effective siRNA experiments can be challenging. The various existing siRNA design guidelines suffer from two problems: they differ considerably from each other, and they produce high levels of false-positive predictions when tested on data of independent origins. RESULTS: Using a distinctly large set of siRNA efficacy data assembled from a vast diversity of origins (the siRecords data, containing records of 3,277 siRNA experiments targeting 1,518 genes, derived from 1,417 independent studies), we conducted extensive analyses of all known features that have been implicated in increasing RNAi effectiveness. A number of features having positive impacts on siRNA efficacy were identified. By performing quantitative analyses on cooperative effects among these features, then applying a disjunctive rule merging (DRM) algorithm, we developed a bundle of siRNA design rule sets with the false positive problem well curbed. A comparison with 15 online siRNA design tools indicated that some of the rule sets we developed surpassed all of these design tools commonly used in siRNA design practice in positive predictive values (PPVs). CONCLUSION: The availability of the large and diverse siRNA dataset from siRecords and the approach we describe in this report have allowed the development of highly effective and generally applicable siRNA design rule sets. Together with ever improving RNAi lab techniques, these design rule sets are expected to make siRNAs a more useful tool for molecular genetics, functional genomics, and drug discovery studies

Context-Aware Mixup for Domain Adaptive Semantic Segmentation

Unsupervised domain adaptation (UDA) aims to adapt a model of the labeled source domain to an unlabeled target domain. Existing UDA-based semantic segmentation approaches always reduce the domain shifts in pixel level, feature level, and output level. However, almost all of them largely neglect the contextual dependency, which is generally shared across different domains, leading to less-desired performance. In this paper, we propose a novel Context-Aware Mixup (CAMix) framework for domain adaptive semantic segmentation, which exploits this important clue of context-dependency as explicit prior knowledge in a fully end-to-end trainable manner for enhancing the adaptability toward the target domain. Firstly, we present a contextual mask generation strategy by leveraging the accumulated spatial distributions and prior contextual relationships. The generated contextual mask is critical in this work and will guide the context-aware domain mixup on three different levels. Besides, provided the context knowledge, we introduce a significance-reweighted consistency loss to penalize the inconsistency between the mixed student prediction and the mixed teacher prediction, which alleviates the negative transfer of the adaptation, e.g., early performance degradation. Extensive experiments and analysis demonstrate the effectiveness of our method against the state-of-the-art approaches on widely-used UDA benchmarks.Comment: Accepted to IEEE Transactions on Circuits and Systems for Video Technology (TCSVT

PyPop7: A Pure-Python Library for Population-Based Black-Box Optimization

In this paper, we present a pure-Python open-source library, called PyPop7, for black-box optimization (BBO). It provides a unified and modular interface for more than 60 versions and variants of different black-box optimization algorithms, particularly population-based optimizers, which can be classified into 12 popular families: Evolution Strategies (ES), Natural Evolution Strategies (NES), Estimation of Distribution Algorithms (EDA), Cross-Entropy Method (CEM), Differential Evolution (DE), Particle Swarm Optimizer (PSO), Cooperative Coevolution (CC), Simulated Annealing (SA), Genetic Algorithms (GA), Evolutionary Programming (EP), Pattern Search (PS), and Random Search (RS). It also provides many examples, interesting tutorials, and full-fledged API documentations. Through this new library, we expect to provide a well-designed platform for benchmarking of optimizers and promote their real-world applications, especially for large-scale BBO. Its source code and documentations are available at https://github.com/Evolutionary-Intelligence/pypop and https://pypop.readthedocs.io/en/latest, respectively.Comment: 5 page