Data Preparation in the Big Data Era

Preparing and cleaning data is notoriously expensive, prone to error, and time consuming: the process accounts for roughly 80% of the total time spent on analysis. As this O’Reilly report points out, enterprises have already invested billions of dollars in big data analytics, so there’s great incentive to modernize methods for cleaning, combining, and transforming data. Author Federico Castanedo, Chief Data Scientist at WiseAthena.com, details best practices for reducing the time it takes to convert raw data into actionable insights. With these tools and techniques in mind, your organization will be well positioned to translate big data into big decisions. • Explore the problems organizations face today with traditional prep and integration • Define the business questions you want to address before selecting, prepping, and analyzing data • Learn new methods for preparing raw data, including date-time and string data • Understand how some cleaning actions (like replacing missing values) affect your analysis • Examine data curation products: modern approaches that scale • Consider your business audience when choosing ways to deliver your analysis Federico Castanedo is the Chief Data Scientist at WiseAthena.com. Involved in projects related to data analysis in academia and industry for more than a decade, he’s published several scientific papers about data fusion techniques, visual sensor networks, and machine learning

Data fusion to improve trajectory tracking in a Cooperative Surveillance Multi-Agent Architecture

13 pages, 12 figures.In this paper we present a Cooperative Surveillance Multi-Agent System (CS-MAS) architecture extended to incorporate dynamic coalition formation. We illustrate specific coalition formation using fusion skills. In this case, the fusion process is divided into two layers: (i) a global layer in the fusion center, which initializes the coalitions and (ii) a local layer within coalitions, where a local fusion agent is dynamically instantiated. There are several types of autonomous agent: surveillance–sensor agents, a fusion center agent, a local fusion agent, interface agents, record agents, planning agents, etc. Autonomous agents differ in their ability to carry out a specific surveillance task. A surveillance–sensor agent controls and manages individual sensors (usually video cameras). It has different capabilities depending on its functional complexity and limitations related to sensor-specific aspects. In the work presented here we add a new autonomous agent, called the local fusion agent, to the CS-MAS architecture, addressing specific problems of on-line sensor alignment, registration, bias removal and data fusion. The local fusion agent is dynamically created by the fusion center agent and involves several surveillance–sensor agents working in a coalition. We show how the inclusion of this new dynamic local fusion agent guarantees that, in a video-surveillance system, objects of interest are successfully tracked across the whole area, assuring continuity and seamless transitions.This work was supported in part by Projects CICYT TIN2008-06742-C02-02/TSI, CICYT TEC2008-06732-C02-02/TEC, SINPROB, CAM MADRINET S-0505 /TIC/0255 and DPS2008-07029-C02-02.Publicad

A multi-agent architecture to support active fusion in a visual sensor network

8 pages, 12 figures.-- Contributed to: Second ACM/IEEE International Conference on Distributed Smart Cameras (ICDSC'2008, Stanford, California, US, Sep 7-11, 2008).One of the main characteristics of a visual sensor network environment is the high amount of data generated. In addition, the application of some process, as for example tracking objects, generate a highly noisy output which may potentially produce an inconsistent system output. By inconsistent output we mean highly differences between tracking information provided by the visual sensors. A visual sensor network, with overlapped field of views, could exploit the redundancy between the field of view of each visual sensor to avoid inconsistencies and obtain more accurate results. In this paper, we present a visual sensor network system with overlapped field of views, modeled as a network of software agents. The communication of each software agent allows the use of feedback information in the visual sensors, called active fusion. Results of the software architecture to support active fusion scheme in an indoor scenario evaluation are presented.This work was supported in part by Projects MADRINET, TEC2005-07186-C03-02, SINPROB, TSI2005-07344-C02-02.Publicad

A multi-agent architecture based on the BDI model for data fusion in visual sensor networks

30 pages, 18 figures.-- Article in press.The newest surveillance applications is attempting more complex tasks such as the analysis of the behavior of individuals and crowds. These complex tasks may use a distributed visual sensor network in order to gain coverage and exploit the inherent redundancy of the overlapped field of views. This article, presents a Multi-agent architecture based on the Belief-Desire-Intention (BDI) model for processing the information and fusing the data in a distributed visual sensor network. Instead of exchanging raw images between the agents involved in the visual network, local signal processing is performed and only the key observed features are shared. After a registration or calibration phase, the proposed architecture performs tracking, data fusion and coordination. Using the proposed Multi-agent architecture, we focus on the means of fusing the estimated positions on the ground plane from different agents which are applied to the same object. This fusion process is used for two different purposes: (1) to obtain a continuity in the tracking along the field of view of the cameras involved in the distributed network, (2) to improve the quality of the tracking by means of data fusion techniques, and by discarding non reliable sensors. Experimental results on two different scenarios show that the designed architecture can successfully track an object even when occlusions or sensor’s errors take place. The sensor’s errors are reduced by exploiting the inherent redundancy of a visual sensor network with overlapped field of views.This work was partially supported by projects CICYT TIN2008-06742-C02-02/TSI, CICYT TEC2008-06732-C02-02/TEC, SINPROB, CAM MADRINET S-0505/TIC/0255 and DPS2008-07029-C02-02.En prens

Analysis of distributed fusion alternatives in coordinated vision agents

6 pages, 10 figures.-- Contributed to: 11th International Conference on Information Fusion (FUSION'2008, Cologne, Germany, Jun 30-Jul 3, 2008).In this paper, we detail some technical alternatives when building a coherent distributed visual sensor network by using the multi-agent paradigm. We argue that the multi-agent paradigm fits well within the visual sensor network architecture and in this paper we specially focus on the problem of distributed data fusion. Three different data fusion coordination schemes are proposed and experimental results of passive fusion are presented and discussed. The main contributions of this paper are twofold, first we propose the use of multi-agent paradigm as the visual sensor architecture and present a real system results. Secondly, the use of feedback information in the visual sensors, called active fusion, is proposed. The experimental results prove that the multi-agent paradigm fits well within the visual sensor network and provide a novel mechanism to develop a real visual sensor network system.This work was partially supported by projects MADRINET, TEC2005-07186-C03-02, SINPROB, TSI2005-07344-C02-02.Publicad

Influence of postoperative complications on long-term survival in liver transplant patients

Background: Liver transplant (LT) is a complex procedure with frequent postoperative complications. In other surgical procedures such as gastrectomy, esophagectomy or resection of liver metastases, these complications are associated with poorer long-term survival. It is possible this happens in LT but there are not enough data to establish this relationship. Aim: To analyze the possible influence of postoperative complications on long-term survival and the ability of the comprehensive complication index (CCI) to predict this. Methods: Retrospective study in a tertiary-level university hospital. The 164 participants were all patients who received a LT from January 2012 to July 2019. The follow-up was done in the hospital until the end of the study or death. Comorbidity and risk after transplantation were calculated using the Charlson and balance of risk (BAR) scores, respectively. Postoperative complications were graded according to the Clavien-Dindo classification and the CCI. To assess the CCI cut-off value with greater prognostic accuracy a receiver operating characteristic (ROC) curve was built, with calculation of the area under the curve (AUC). Overall survival was estimated according to the Kaplan-Meier test and log-rank test. Groups were compared by the Mann-Whitney test. For the multivariable analysis the Cox regression was used. Results: The mean follow-up time of the cohort was 37.76 (SD = 24.5) mo. A ROC curve of CCI with 5-year survival was built. The AUC was 0.826 (0.730-0.922), P 33.5 (33.5 = median CCI value) showed estimated 5-year survival was 57.4 and 45.71 months, respectively (log-rank < 0.0001). Dividing patients according to the mode CCI value (20.9) showed an estimated 5-year survival of 60 mo for a CCI below 20.9 vs 57 mo for a CCI above 20.9 (log-rank = 0.147). The univariate analysis did not show any association between individual complications and long-term survival. A multivariate analysis was carried out to analyse the possible influence of CCI, Charlson comorbidity index, BAR and hepatocellular carcinoma on survival. Only the CCI score showed significant influence on long-term survival. Conclusion: A complicated postoperative period - well-defined by means of the CCI score - can influence not only short-term survival, but also long-term survival