Selective Query Processing: a Risk-Sensitive Selection of System Configurations

In information retrieval systems, search parameters are optimized to ensure high effectiveness based on a set of past searches and these optimized parameters are then used as the system configuration for all subsequent queries. A better approach, however, would be to adapt the parameters to fit the query at hand. Selective query expansion is one such an approach, in which the system decides automatically whether or not to expand the query, resulting in two possible system configurations. This approach was extended recently to include many other parameters, leading to many possible system configurations where the system automatically selects the best configuration on a per-query basis. To determine the ideal configurations to use on a per-query basis in real-world systems we developed a method in which a restricted number of possible configurations is pre-selected and then used in a meta-search engine that decides the best search configuration on a per query basis. We define a risk-sensitive approach for configuration pre-selection that considers the risk-reward trade-off between the number of configurations kept, and system effectiveness. For final configuration selection, the decision is based on query feature similarities. We find that a relatively small number of configurations (20) selected by our risk-sensitive model is sufficient to increase effectiveness by about 15% according(P@10, nDCG@10) when compared to traditional grid search using a single configuration and by about 20% when compared to learning to rank documents. Our risk-sensitive approach works for both diversity- and ad hoc-oriented searches. Moreover, the similarity-based selection method outperforms the more sophisticated approaches. Thus, we demonstrate the feasibility of developing per-query information retrieval systems, which will guide future research in this direction.Comment: 30 pages, 5 figures, 8 tables; submitted to TOIS ACM journa

Análise de vídeo sensível

Orientadores: Anderson de Rezende Rocha, Siome Klein GoldensteinTese (doutorado) - Universidade Estadual de Campinas, Instituto de ComputaçãoResumo: Vídeo sensível pode ser definido como qualquer filme capaz de oferecer ameaças à sua audiência. Representantes típicos incluem ¿ mas não estão limitados a ¿ pornografia, violência, abuso infantil, crueldade contra animais, etc. Hoje em dia, com o papel cada vez mais pervasivo dos dados digitais em nossa vidas, a análise de conteúdo sensível representa uma grande preocupação para representantes da lei, empresas, professores, e pais, devido aos potenciais danos que este tipo de conteúdo pode infligir a menores, estudantes, trabalhadores, etc. Não obstante, o emprego de mediadores humanos, para constantemente analisar grandes quantidades de dados sensíveis, muitas vezes leva a ocorrências de estresse e trauma, o que justifica a busca por análises assistidas por computador. Neste trabalho, nós abordamos este problema em duas frentes. Na primeira, almejamos decidir se um fluxo de vídeo apresenta ou não conteúdo sensível, à qual nos referimos como classificação de vídeo sensível. Na segunda, temos como objetivo encontrar os momentos exatos em que um fluxo começa e termina a exibição de conteúdo sensível, em nível de quadros de vídeo, à qual nos referimos como localização de conteúdo sensível. Para ambos os casos, projetamos e desenvolvemos métodos eficazes e eficientes, com baixo consumo de memória, e adequação à implantação em dispositivos móveis. Neste contexto, nós fornecemos quatro principais contribuições. A primeira é uma nova solução baseada em sacolas de palavras visuais, para a classificação eficiente de vídeos sensíveis, apoiada na análise de fenômenos temporais. A segunda é uma nova solução de fusão multimodal em alto nível semântico, para a localização de conteúdo sensível. A terceira, por sua vez, é um novo detector espaço-temporal de pontos de interesse, e descritor de conteúdo de vídeo. Finalmente, a quarta contribuição diz respeito a uma base de vídeos anotados em nível de quadro, que possui 140 horas de conteúdo pornográfico, e que é a primeira da literatura a ser adequada para a localização de pornografia. Um aspecto relevante das três primeiras contribuições é a sua natureza de generalização, no sentido de poderem ser empregadas ¿ sem modificações no passo a passo ¿ para a detecção de tipos diversos de conteúdos sensíveis, tais como os mencionados anteriormente. Para validação, nós escolhemos pornografia e violência ¿ dois dos tipos mais comuns de material impróprio ¿ como representantes de interesse, de conteúdo sensível. Nestes termos, realizamos experimentos de classificação e de localização, e reportamos resultados para ambos os tipos de conteúdo. As soluções propostas apresentam uma acurácia de 93% em classificação de pornografia, e permitem a correta localização de 91% de conteúdo pornográfico em fluxo de vídeo. Os resultados para violência também são interessantes: com as abordagens apresentadas, nós obtivemos o segundo lugar em uma competição internacional de detecção de cenas violentas. Colocando ambas em perspectiva, nós aprendemos que a detecção de pornografia é mais fácil que a de violência, abrindo várias oportunidades de pesquisa para a comunidade científica. A principal razão para tal diferença está relacionada aos níveis distintos de subjetividade que são inerentes a cada conceito. Enquanto pornografia é em geral mais explícita, violência apresenta um espectro mais amplo de possíveis manifestaçõesAbstract: Sensitive video can be defined as any motion picture that may pose threats to its audience. Typical representatives include ¿ but are not limited to ¿ pornography, violence, child abuse, cruelty to animals, etc. Nowadays, with the ever more pervasive role of digital data in our lives, sensitive-content analysis represents a major concern to law enforcers, companies, tutors, and parents, due to the potential harm of such contents over minors, students, workers, etc. Notwithstanding, the employment of human mediators for constantly analyzing huge troves of sensitive data often leads to stress and trauma, justifying the search for computer-aided analysis. In this work, we tackle this problem in two ways. In the first one, we aim at deciding whether or not a video stream presents sensitive content, which we refer to as sensitive-video classification. In the second one, we aim at finding the exact moments a stream starts and ends displaying sensitive content, at frame level, which we refer to as sensitive-content localization. For both cases, we aim at designing and developing effective and efficient methods, with low memory footprint and suitable for deployment on mobile devices. In this vein, we provide four major contributions. The first one is a novel Bag-of-Visual-Words-based pipeline for efficient time-aware sensitive-video classification. The second is a novel high-level multimodal fusion pipeline for sensitive-content localization. The third, in turn, is a novel space-temporal video interest point detector and video content descriptor. Finally, the fourth contribution comprises a frame-level annotated 140-hour pornographic video dataset, which is the first one in the literature that is appropriate for pornography localization. An important aspect of the first three contributions is their generalization nature, in the sense that they can be employed ¿ without step modifications ¿ to the detection of diverse sensitive content types, such as the previously mentioned ones. For validation, we choose pornography and violence ¿ two of the commonest types of inappropriate material ¿ as target representatives of sensitive content. We therefore perform classification and localization experiments, and report results for both types of content. The proposed solutions present an accuracy of 93% in pornography classification, and allow the correct localization of 91% of pornographic content within a video stream. The results for violence are also compelling: with the proposed approaches, we reached second place in an international competition of violent scenes detection. Putting both in perspective, we learned that pornography detection is easier than its violence counterpart, opening several opportunities for additional investigations by the research community. The main reason for such difference is related to the distinct levels of subjectivity that are inherent to each concept. While pornography is usually more explicit, violence presents a broader spectrum of possible manifestationsDoutoradoCiência da ComputaçãoDoutor em Ciência da Computação1572763, 1197473CAPE

Evaluation of information retrieval systems using structural equation modeling

The interpretation of the experimental data collected by testing systems across input datasets and model parameters is of strategic importance for system design and implementation. In particular, finding relationships between variables and detecting the latent variables affecting retrieval performance can provide designers, engineers and experimenters with useful if not necessary information about how a system is performing. This paper discusses the use of Structural Equation Modeling (SEM) in providing an in-depth explanation of evaluation results and an explanation of failures and successes of a system; in particular, we focus on the case of evaluation of Information Retrieval systems

On Experimentation in Software-Intensive Systems

Context: Delivering software that has value to customers is a primary concern of every software company. Prevalent in web-facing companies, controlled experiments are used to validate and deliver value in incremental deployments. At the same that web-facing companies are aiming to automate and reduce the cost of each experiment iteration, embedded systems companies are starting to adopt experimentation practices and leverage their activities on the automation developments made in the online domain. Objective: This thesis has two main objectives. The first objective is to analyze how software companies can run and optimize their systems through automated experiments. This objective is investigated from the perspectives of the software architecture, the algorithms for the experiment execution and the experimentation process. The second objective is to analyze how non web-facing companies can adopt experimentation as part of their development process to validate and deliver value to their customers continuously. This objective is investigated from the perspectives of the software development process and focuses on the experimentation aspects that are distinct from web-facing companies. Method: To achieve these objectives, we conducted research in close collaboration with industry and used a combination of different empirical research methods: case studies, literature reviews, simulations, and empirical evaluations. Results: This thesis provides six main results. First, it proposes an architecture framework for automated experimentation that can be used with different types of experimental designs in both embedded systems and web-facing systems. Second, it proposes a new experimentation process to capture the details of a trustworthy experimentation process that can be used as the basis for an automated experimentation process. Third, it identifies the restrictions and pitfalls of different multi-armed bandit algorithms for automating experiments in industry. This thesis also proposes a set of guidelines to help practitioners select a technique that minimizes the occurrence of these pitfalls. Fourth, it proposes statistical models to analyze optimization algorithms that can be used in automated experimentation. Fifth, it identifies the key challenges faced by embedded systems companies when adopting controlled experimentation, and we propose a set of strategies to address these challenges. Sixth, it identifies experimentation techniques and proposes a new continuous experimentation model for mission-critical and business-to-business. Conclusion: The results presented in this thesis indicate that the trustworthiness in the experimentation process and the selection of algorithms still need to be addressed before automated experimentation can be used at scale in industry. The embedded systems industry faces challenges in adopting experimentation as part of its development process. In part, this is due to the low number of users and devices that can be used in experiments and the diversity of the required experimental designs for each new situation. This limitation increases both the complexity of the experimentation process and the number of techniques used to address this constraint

Ariel - Volume 12(13) Number 1

Executive Editor Gary E. Fishbein Associate Editor Sam Markind Business Manager Rich Davis Sports Editor Dave Cohen Photography Editor Ben Alman Layout Sheila Grossma

Active Learning With Complementary Sampling for Instructing Class-Biased Multi-Label Text Emotion Classification

High-quality corpora have been very scarce for the text emotion research. Existing corpora with multi-label emotion annotations have been either too small or too class-biased to properly support a supervised emotion learning. In this paper, we propose a novel active learning method for efficiently instructing the human annotations for a less-biased and high-quality multi-label emotion corpus. Specifically, to compensate annotation for the minority-class examples, we propose a complementary sampling strategy based on unlabeled resources by measuring a probabilistic distance between the expected emotion label distribution in a temporary corpus and an uniform distribution. Qualitative evaluations are also given to the unlabeled examples, in which we evaluate the model uncertainties for multi-label emotion predictions, their syntactic representativeness for the other unlabeled examples, and their diverseness to the labeled examples, for a high-quality sampling. Through active learning, a supervised emotion classifier gets progressively improved by learning from these new examples. Experiment results suggest that by following these sampling strategies we can develop a corpus of high-quality examples with significantly relieved bias for emotion classes. Compared to the learning procedures based on traditional active learning algorithms, our learning procedure indicates the most efficient learning curve and estimates the best multi-label emotion predictions

Adaptive Operator Mechanism for Genetic Programming

학위논문 (박사)-- 서울대학교 대학원 : 전기·컴퓨터공학부, 2013. 8. Robert Ian McKay.their performances are competitive with systems without an adaptive operator mechanism. However they showed some drawbacks, which we discuss. To overcome them, we suggest three variants on operator selection, which performed somewhat better. We have investigated evaluation of operator impact in adaptive operator mechanism, which measures the impact of operator applications on improvement of solution. Hence the impact guides operator rates, evaluation of operator impact is very important in adaptive operator mechanism. There are two issues in evaluation of operator impact: the resource and the method. Basically all history information of run are able to be used as resources for the operator impact, but fitness value which is directly related with the improvement of solution, is usually used as a resource. By using a variety of problems, we used two kinds of resources: accuracy and structure in this thesis. On the other hand, although we used same resources, the evaluated impacts are different by methods. We suggested several methods of the evaluation of operator impact. Although they require only small change, they have a large effect on performance. Finally, we verified adaptive operator mechanism by applying it to a real-world applicationa modeling of algal blooms in the Nakdong River. The objective of this application is a model that describes and predicts the ecosystem of the Nakdong River. We verified it with two researches: fitting the parameters of an expert-derived model for the Nakdong River with a GA, and modeling by extending the expert-derived model with TAG3P.유전 프로그래밍은 모델 학습에 효과적인 진화 연산 알고리즘이다. 유전 프로그래밍은 다양한 파라미터를 가지고 있는데, 이들 파라미터의 값은 대체로 주어진 문제에 맞춰 사용자가 직접 조정한다. 유전 프로그래밍의 성능은 파라미터의 값에 따라 크게 좌우되기 때문에 파라미터 설정에 대한 연구는 진화 연산에서 많은 주목을 받고 있다. 하지만 아직까지 효과적으로 파라미터를 설정하는 방법에 대한 보편적인 지침이 없으며, 많은 실험을 통한 시행착오를 거치면서 적절한 파라미터 값을 찾는 방법이 일반적으로 쓰이고 있다. 본 논문에서 제시하는 적응 연산자 메커니즘은 여러 파라미터 중 유전 연산자의 적용률을 설정해 주는 방법으로, 학습 중간중간의 상황에 맞춰 연산자 적용률을 자동적으로 조정한다. 본 논문에서는, 기존의 적응 연산자 방법을 다양한 유전 연산자를 가진 문법 기반의 유전 프로그래밍인 TAG3P에 적용하고 새로운 적응 연산자 방법을 개발함으로써, 적응 연산자 메커니즘의 적용 범위를 유전 프로그래밍 영역까지 확장하였다. 기존의 적응 연산자 알고리즘을 TAG3P에 적용시키는 연구는 성공적으로 이루어졌으나 몇 가지 문제점을 드러내었다. 이 문제점은 본문에서 후술한다. 이 문제점을 해결하기 위해 유전자 선택에 대한 새로운 변형 알고리즘을 제시하였고, 이는 기존 알고리즘과 비교하여 더 좋은 성능을 보여주었다. 한편으로 유전 연산자가 해의 향상에 미치는 영향을 측정하는 연산자 영향력 평가에 대한 연구도 진행하였다. 적응 연산자 메커니즘에서는 측정된 영향력을 바탕으로 연산자의 적용률을 변화시키기 때문에 영향력 평가는 적응 연산자 메커니즘에서 매우 중요하다. 이 연구에서는 어떤 정보를 이용하여 영향력을 측정할 것인지, 그리고 어떤 방법을 이용하여 영향력을 측정할 것인지의 두 가지 주요 쟁점을 다룬다. 연산자 영향력 평가에는 학습 과정의 모든 정보가 사용될 수 있으며, 대체로 해의 향상과 직접적인 관련이 있는 적합도를 이용한다. 본 논문에서는 다양한 문제를 이용하여 정확도와 구조에 관련된 두 지표를 영향력 평가에 이용해보았다. 한편으로 같은 정보를 이용하더라도 그것을 활용하는 방법에 따라 측정되는 영향력이 달라지는데, 본 논문에서는 작은 변화를 통해서도 큰 성능 변화를 야기시킬 수 있는 영향력 평가 방법을 몇가지 소개한다. 마지막으로 적응 연산자 메커니즘을 실제 문제에 적용함으로써 유용성을 확인하였다. 이를 위해 사용된 실제 문제는 낙동강의 녹조 현상에 대한 예측으로, 낙동강의 생태 시스템을 묘사하고 예측하는 모델을 개발하는 것을 목적으로 한다. 2가지 연구를 통해 유용성을 확인하였다. 우선 전문가에 의해 만들어진 기본 모델을 바탕으로, 유전 알고리즘을 이용하여 모델의 파라미터를 최적화 하였고, 그리고 TAG3P를 이용하여 기본 모델의 확장하고 이를 통해 새로운 모델을 만들어 보았다.Genetic programming (GP) is an effective evolutionary algorithm for many problems, especially suited to model learning. GP has many parameters, usually defined by the user according to the problem. The performance of GP is sensitive to their values. Parameter setting has been a major focus of study in evolutionary computation. However there is still no general guideline for choosing efficient settings. The usual method for parameter setting is trial and error. The method used in this thesis, adaptive operator mechanism, replaces the user's action in setting rates of application of genetic operators. adaptive operator mechanism autonomously controls the genetic operators during a run. This thesis extends adaptive operator mechanism to genetic programming, applying existing adaptive operator algorithms and developing them for TAG3P, a grammar-guided GP which supports a wide variety of useful genetic operators. Existing adaptive operator selection algorithms are successfully applied to TAG3P1 Introduction 1 1.1 Background and Motivation 1 1.2 Our Approach and Its Contributions 2 1.3 Outline 4 2 Related Works 5 2.1 Evolutionary Algorithms 5 2.1.1 Genetic Algorithm 5 2.1.2 Genetic Programming 8 2.1.3 Tree Adjoining Grammar based Genetic Programming 9 3 Adaptive Mechanism and Adaptive Operator Selection 16 3.1 Adaptive Mechanism 16 3.2 Adaptive Operator Selection 18 3.2.1 Operator Selection 18 3.2.2 Evaluation of Operator Impact 19 3.3 Algorithms of Adaptive Operator Selection 20 3.3.1 Probability Matching 21 3.3.2 Adaptive Pursuit 22 3.3.3 Multi-Armed Bandits 25 4 Preliminary Experiment for Adaptive Operator Mechanism 28 4.1 Test Problems 28 4.2 Experimental Design 30 4.2.1 Search Space 31 4.2.2 General Parameter Settings 32 4.3 Results and Discussion 34 5 Operator Selection 39 5.1 Operator Selection Algorithms for GP 39 5.1.1 Powered Probability Matching 39 5.1.2 Adaptive Probability Matching 41 5.1.3 Recursive Adaptive Pursuit 41 5.2 Experiments and Results 43 5.2.1 Test Problems 43 5.2.2 Experimental Design 44 5.2.3 Results and Discussion 46 6 Evaluation of Operator Impact 56 6.1 Rates for the Amount of Individual Usage 57 6.1.1 Denition of Rates for the Amount of Individual Usage 57 6.1.2 Results and Discussion 58 6.2 Ratio for the Improvement of Fitness 63 6.2.1 Pairs and Group 64 6.2.2 Ratio and Children Fitness 65 6.2.3 Experimental Design 65 6.2.4 Result and Discussion 66 6.3 Ranking Point 73 6.3.1 Denition of Ranking Point 73 6.3.2 Experimental Design 74 6.3.3 Result and Discussion 74 6.4 Pre-Search Structure 76 6.4.1 Denition of Pre-Search Structure 76 6.4.2 Preliminary Experiment for Sampling 78 6.4.3 Experimental Design 82 6.4.4 Result and Discussion 83 7 Application: Nakdong River Modeling 85 7.1 Problem Description 85 7.1.1 Outline 85 7.1.2 Data Description 86 7.1.3 Model Description 88 7.1.4 Methods 93 7.2 Results 97 7.2.1 Parameter Optimization 97 7.2.2 Modeling 101 7.3 Summary 103 8 Conclusion 104 8.1 Summary 104 8.2 Future Works 108Docto