Multi-Modal Classifiers for Open-Vocabulary Object Detection

The goal of this paper is open-vocabulary object detection (OVOD) \unicode{x2013} building a model that can detect objects beyond the set of categories seen at training, thus enabling the user to specify categories of interest at inference without the need for model retraining. We adopt a standard two-stage object detector architecture, and explore three ways for specifying novel categories: via language descriptions, via image exemplars, or via a combination of the two. We make three contributions: first, we prompt a large language model (LLM) to generate informative language descriptions for object classes, and construct powerful text-based classifiers; second, we employ a visual aggregator on image exemplars that can ingest any number of images as input, forming vision-based classifiers; and third, we provide a simple method to fuse information from language descriptions and image exemplars, yielding a multi-modal classifier. When evaluating on the challenging LVIS open-vocabulary benchmark we demonstrate that: (i) our text-based classifiers outperform all previous OVOD works; (ii) our vision-based classifiers perform as well as text-based classifiers in prior work; (iii) using multi-modal classifiers perform better than either modality alone; and finally, (iv) our text-based and multi-modal classifiers yield better performance than a fully-supervised detector.Comment: ICML 2023, project page: https://www.robots.ox.ac.uk/vgg/research/mm-ovod

Exploring data-driven building energy-efficient design of envelopes based on their quantified impacts

Building performance design plays a key role in reducing the energy consumption of buildings. However, the widely used simulation-based design is facing several challenges, such as the labor-intensive modeling process and the performance gaps between design stage estimations and operational energy use. For these reasons, artificial intelligent methods are expected by designers to improve the efficiency and reliability of building energy-efficient design. To date, there has not been a practical data-driven design method of envelopes. This study aimed at exploring data-driven building energy-efficient design of envelopes based on their quantified impacts. A feature selection method and a game-theoretic method were applied to quantify the impacts of envelopes on space heating and cooling energy, which were performed on two building datasets, one of which is from the U.S. and the other from China. Random forest classifiers were developed to conduct the study. Based on discovered energy patterns and quantified impacts of envelopes on energy consumption, a rectified linear design method of envelopes was proposed with the idea of improving the performance of high-impact envelopes. Besides, a validation study was conducted on two office buildings in the hot-summer cold-winter region. To design the envelopes of a building, the data-driven analysis was driven by its similar buildings other than the whole dataset. Moreover, a detailed energy simulation was conducted to evaluate the energy performance of different design solutions. The results showed that compared with baseline design solutions, new strategies could save 1.05%–21.2% energy for space heating and cooling for these two case buildings. The proposed method is a general building envelope design approach and allows designers to easily find an energy-efficient configuration of envelopes. This study demonstrated the feasibility and effectiveness of the data-driven energy-efficient design of building envelopes

Passive classification of Wi-Fi enabled devices

We propose a method for classifying Wi-Fi enabled mobile handheld devices (smartphones) and non-handheld devices (laptops) in a completely passive way, that is resorting neither to traffic probes on network edge devices nor to deep packet inspection techniques to read application layer information. Instead, classification is performed starting from probe requests Wi-Fi frames, which can be sniffed with inexpensive commercial hardware. We extract distinctive features from probe request frames (how many probe requests are transmitted by each device, how frequently, etc.) and take a machine learning approach, training four different classifiers to recognize the two types of devices. We compare the performance of the different classifiers and identify a solution based on a Random Decision Forest that correctly classify devices 95% of the times. The classification method is then used as a pre-processing stage to analyze network traffic traces from the wireless network of a university building, with interesting considerations on the way different types of devices uses the network (amount of data exchanged, duration of connections, etc.). The proposed methodology finds application in many scenarios related to Wi-Fi network management/optimization and Wi-Fi based services

A committee machine gas identification system based on dynamically reconfigurable FPGA

This paper proposes a gas identification system based on the committee machine (CM) classifier, which combines various gas identification algorithms, to obtain a unified decision with improved accuracy. The CM combines five different classifiers: K nearest neighbors (KNNs), multilayer perceptron (MLP), radial basis function (RBF), Gaussian mixture model (GMM), and probabilistic principal component analysis (PPCA). Experiments on real sensors' data proved the effectiveness of our system with an improved accuracy over individual classifiers. Due to the computationally intensive nature of CM, its implementation requires significant hardware resources. In order to overcome this problem, we propose a novel time multiplexing hardware implementation using a dynamically reconfigurable field programmable gate array (FPGA) platform. The processing is divided into three stages: sampling and preprocessing, pattern recognition, and decision stage. Dynamically reconfigurable FPGA technique is used to implement the system in a sequential manner, thus using limited hardware resources of the FPGA chip. The system is successfully tested for combustible gas identification application using our in-house tin-oxide gas sensors