Efficient Processing of k Nearest Neighbor Joins using MapReduce

k nearest neighbor join (kNN join), designed to find k nearest neighbors from a dataset S for every object in another dataset R, is a primitive operation widely adopted by many data mining applications. As a combination of the k nearest neighbor query and the join operation, kNN join is an expensive operation. Given the increasing volume of data, it is difficult to perform a kNN join on a centralized machine efficiently. In this paper, we investigate how to perform kNN join using MapReduce which is a well-accepted framework for data-intensive applications over clusters of computers. In brief, the mappers cluster objects into groups; the reducers perform the kNN join on each group of objects separately. We design an effective mapping mechanism that exploits pruning rules for distance filtering, and hence reduces both the shuffling and computational costs. To reduce the shuffling cost, we propose two approximate algorithms to minimize the number of replicas. Extensive experiments on our in-house cluster demonstrate that our proposed methods are efficient, robust and scalable.Comment: VLDB201

Study on Buyback Contract in Supply Chain With a Loss-Averse Supplier and Multiple Loss-Averse Retailers Under Stockout Loss Situation

According to the prospect theory and the loss-aversion function, this paper developers the buyback contract model in a two-stage supply chain with a loss-averse supplier and multiple loss-averse retailers. Under the stockout loss setting, we analyze the effect of the loss aversion on the behavior from the retailers and the supplier, and then the buyback contract has been shown to be able to coordinate the supply chain. Furthermore, the number of retailers and loss aversion coefficient meet a certain range, there will be a unique optimal buyback price to achieve supply chain coordination

Evaluation of Performance of Bus Lanes on Urban Expressway Using Paramics Micro-simulation Model

AbstractUrban expressway, as the main skeleton of the road network, is the aorta between urban regions and urban external traffic communication, but also bears the commuter channel. It makes a large amount of traffic flow into the expressway, resulting the congestion in the expressway in many big cities, including Beijing. Taking the Beijing southwest third ring expressway for example, a simulation model was built using Paramics. The simulation model was pre-evaluated before and after the bus lanes set, and the model was post-evaluated to verify the validity of the model after the bus lanes were implemented, it has important theoretical and practical value

Enabling Quality Control for Entity Resolution: A Human and Machine Cooperation Framework

Even though many machine algorithms have been proposed for entity resolution, it remains very challenging to find a solution with quality guarantees. In this paper, we propose a novel HUman and Machine cOoperation (HUMO) framework for entity resolution (ER), which divides an ER workload between the machine and the human. HUMO enables a mechanism for quality control that can flexibly enforce both precision and recall levels. We introduce the optimization problem of HUMO, minimizing human cost given a quality requirement, and then present three optimization approaches: a conservative baseline one purely based on the monotonicity assumption of precision, a more aggressive one based on sampling and a hybrid one that can take advantage of the strengths of both previous approaches. Finally, we demonstrate by extensive experiments on real and synthetic datasets that HUMO can achieve high-quality results with reasonable return on investment (ROI) in terms of human cost, and it performs considerably better than the state-of-the-art alternatives in quality control.Comment: 12 pages, 11 figures. Camera-ready version of the paper submitted to ICDE 2018, In Proceedings of the 34th IEEE International Conference on Data Engineering (ICDE 2018