IRC channel data analysis using Apache Solr

Internet Relay Chat (IRC) was one of the first real-time communication protocols over the internet. It was not designed with any form of Authentication, Authorization and Accounting features. This made IRC channels a place to conduct transactions in complete anonymity. On the other hand with the advent of Big Data we are now able to process large quantities of data in a very short period of time. This research presents a method to use Apache Solr, a text indexing server built on top of Lucene to index and search large quantities of IRC data collected over months from public IRC channels. It even presents a highly scalable approach to monitor public IRC channels by creation of IRC Client Bots which are in turn controlled by a robust IRC Parent Bot. The data thus collected is analyzed by Apache Solr and MS SQL servers and the response times are compared. This research concluded that Apache Solr outperforms MS SQL by a very great margin and such an implementation can be used by digital forensic investigators to monitor and search public IRC channels

Network Traffic Analysis Framework For Cyber Threat Detection

The growing sophistication of attacks and newly emerging cyber threats requires advanced cyber threat detection systems. Although there are several cyber threat detection tools in use, cyber threats and data breaches continue to rise. This research is intended to improve the cyber threat detection approach by developing a cyber threat detection framework using two complementary technologies, search engine and machine learning, combining artificial intelligence and classical technologies. In this design science research, several artifacts such as a custom search engine library, a machine learning-based engine and different algorithms have been developed to build a new cyber threat detection framework based on self-learning search and machine learning engines. Apache Lucene.Net search engine library was customized in order to function as a cyber threat detector, and Microsoft ML.NET was used to work with and train the customized search engine. This research proves that a custom search engine can function as a cyber threat detection system. Using both search and machine learning engines in the newly developed framework provides improved cyber threat detection capabilities such as self-learning and predicting attack details. When the two engines run together, the search engine is continuously trained by the machine learning engine and grow smarter to predict yet unknown threats with greater accuracy. While customizing the search engine to function as a cyber threat detector, this research also identified and proved the best algorithms for the search engine based cyber threat detection model. For example, the best scoring algorithm was found to be the Manhattan distance. The validation case study also shows that not every network traffic feature makes an equal contribution to determine the status of the traffic, and thus the variable-dimension Vector Space Model (VSM) achieves better detection accuracy than n-dimensional VSM. Although the use of different technologies and approaches improved detection results, this research is primarily focused on developing techniques rather than building a complete threat detection system. Additional components such as those that can track and investigate the impact of network traffic on the destination devices make the newly developed framework robust enough to build a comprehensive cyber threat detection appliance

Discovery and Push Notification Mechanisms for Mobile Cloud Services

Viimase viie aasta jooksul on mobiilsed seadmed nagu sülearvutid, pihuarvutid, nutitelefonid jmt. tunginud peaaegu kõigisse inimeste igapäevaelu tegevustesse. Samuti on põhjalik teadus- ja arendustegevus mobiilsete tehnoloogiate vallas viinud märkimisväärsete täiustusteni riistvara, tarkvara ja andmeedastuse alal. Tänapäeval on mobiilsed seadmed varustatud sisseehitatud sensorite, kaamera, puutetundliku ekraani, suurema hulga mäluga, kuid ka tõhusamate energiatarbemehhanismidega. Lisaks on iOS ja Android operatsioonisüsteemide väljalaske tõttu suurenenud nii mobiilirakenduste arv kui keerukus, pakkudes arvukamalt kõrgetasemelisi rakendusi. Sarnaselt on toimunud olulised arengud ja standardiseerimisele suunatud jõupingutused veebiteenusete valdkonnas ja elementaarsetele veebiteenuste ligipääsu kasutatakse laialdaselt nutitelefonidest. See on viinud loogilise järgmise sammuna veebiteenuste pakkumiseni nutitelefonidest. Telefonidest veebiteenuste pakkumise kontseptsioon ei ole uus ning seda on põhjalikult uurinud Srirama, kes pakkus välja Mobile Host (Mobiilne Veebiteenuse Pakkuja) kontseptsiooni. Algne realisatsioon kasutas aga aegunud tehnoloogiaid nagu JMEE, PersonalJava, SOAP arhitektuur jne. See töö uuendab Mobile Host'i kasutades uusimaid tehnoloogiad, nagu Android OS ja REST arhitektuur, ning pakub välja teenusemootori, mis põhineb Apache Felix'il - OSGi platvormi realisatsioonil piiratud ressurssidega seadmetele. Hämmastava kiirusega toimunud arengud mobiilsete arvutuste vallas võimaldavad uue põlvkonna veebirakenduste loomist valdkondades nagu keskkonnateadlikkus, sotsiaalvõrgustikud, koostöövahendid, asukohapõhised teenused jne. Sellised rakendused saavad ära kasutada Mobile Host'i võimalusi. Selle tulemusena on klientidel ligipääs väga suurele hulgale teenustele, mistõttu tekib vajadus efektiivse teenuste avastamise mehhanismi järele. See töö pakub välja kataloogipõhise avastusmehhanismi võrgu ülekatte toega suurtele, kõrge liikuvusega võrgustikele. See mehhanism toetub OWL-S'le, mis on ontoloogia veebiteenuseid pakkuvate ressursside avastamiseks, väljakutseks, koostamiseks ja jälgimiseks. Töö kirjeldab ka Srirama välja pakutud algupärast teenuste avastamise mehhanismi, mis toetub peer-to-peer võrkudele ja Apache Lucene võtmesõna otsingumootorile. Uurimuse käigus uuendatakse teenuseotsing kasutama Apache Solr'i, Apache Lucene'i viimast versiooni. Teenuste avastust testiti põhjalikult ja tulemused on töös kokkuvõtvalt välja toodud. Mobiilsete tehnoloogiate vallas uuritakse ka võimalust kasutada pilvetehnolologiat laiendamaks mobiilseadmete salvestusmahtu ja töökoormust edastades pilve andme- ja arvutusmahukad ülesanded. See soodustab keerulisemate ja võimalusrohkemate mobiilirakenduste arendust. Pilve delegeeritavate toimingute aeganõudva iseloomu tõttu aga on vajalik asünkroonne mehhanism teavitamaks kasutajat, millal töömahukad tegevused on lõpetatud. Mobiilsete pilveteenuste pakkujad ja vahevara lahendused võivad kasu saada Mobile Host'ist ja selle asünkroonsete teavituste võimekusest. Uurimus esitleb nelja teavitusmehhanismi: AC2DM, APNS, IBM MQTT ja Mobile Host'i põhine teavitus. Töö võtab kokku kvantitatiivse analüüsi tulemused ja toob välja nelja teavitamise lähenemise tugevused ja nõrkused. Lisaks kirjeldatakse CroudSTag rakenduse realisatsiooni - CroudSTag on mobiilirakendus, mille eesmärgiks on sotsiaalsete gruppide moodustamine kasutades näotuvastustehnoloogiat. CroudSTag-i realisatsioon kasutab mobiilseid pilveteenuseid ja Mobile Host'i, et pakkuda oma funktsionaalsust kasutajale.In the last lustrum the mobile devices such as laptops, PDAs, smart phones, tablets, etc. have pervaded almost all the environments where people perform their day-to-day activities. Further, the extensive Research and Development in mobile technologies has led to significant improvements in hardware, software and transmission. Similarly, there are significant developments and standardization efforts in web services domain and basic web services have been widely accessed from smart phones. This has lead to the logical next step of providing web services from the smart phones. The concept of the web service provisioning from smart phones is not new and has been extensively explored by Srirama who proposed the concept of Mobile Host. However, the original implementation considered aged technologies such as JMEE, PersonalJava, SOAP architecture among others. This work updates the Mobile Host to the latest technologies like Android OS and REST architecture and proposes a service engine based on Apache Felix, and OSGI implementation for resource constraint devices. Moreover, the astonishing speed in developments in mobile computing enable the new generation of applications from domains such as context-awareness, social network, collaborative tools, location based services, etc., which benefit from the Mobile Host service provisioning capabilities. As a result the clients have access to a huge number of services available; therefore, an efficient and effective service discovery mechanism is required. The thesis proposes a directory-based with network overlay support discovery mechanism for large networks with high mobility. The proposed discovery mechanism relies in OWL-S, an ontology for service discovery, invocation, composition, and monitoring of web resources. The work also considers the original service discovery mechanism proposed by Srirama relying in peer-to-peer networks and Apache Lucene, a keyword search engine. The study updates the service search to Apache Solr, the latest development for Apache Lucene. The service discovery was extensively tested and the results are summarized in this work. Mobile technologies are looking into the clouds for extending their capabilities in storage and processing by offloading data and process intensive tasks. This fosters the development of more complex and rich mobile applications. However, due to the time-consuming nature of the tasks delegated to the clouds, an asynchronous mechanism is necessary for notifying the user when the intensive tasks are completed. Mobile cloud service providers and Middleware solutions might benefit from Mobile Host and its asynchronous notification capabilities. The study presents four push notification mechanisms being AC2DM, APNS, IBM MQTT and Mobile Host based push notification. The work summarizes the results of a quantitative analysis and highlights the strengths and weakness of the four notifications approaches. In addition, it explains CroudSTag realization, a mobile application that aims the social group formation by means of facial recognition that relies in mobile cloud services and Mobile Host to provide its functionality to the user

Hardware Acceleration for Unstructured Big Data and Natural Language Processing.

The confluence of the rapid growth in electronic data in recent years, and the renewed interest in domain-specific hardware accelerators presents exciting technical opportunities. Traditional scale-out solutions for processing the vast amounts of text data have been shown to be energy- and cost-inefficient. In contrast, custom hardware accelerators can provide higher throughputs, lower latencies, and significant energy savings. In this thesis, I present a set of hardware accelerators for unstructured big-data processing and natural language processing. The first accelerator, called HAWK, aims to speed up the processing of ad hoc queries against large in-memory logs. HAWK is motivated by the observation that traditional software-based tools for processing large text corpora use memory bandwidth inefficiently due to software overheads, and, thus, fall far short of peak scan rates possible on modern memory systems. HAWK is designed to process data at a constant rate of 32 GB/s—faster than most extant memory systems. I demonstrate that HAWK outperforms state-of-the-art software solutions for text processing, almost by an order of magnitude in many cases. HAWK occupies an area of 45 sq-mm in its pareto-optimal configuration and consumes 22 W of power, well within the area and power envelopes of modern CPU chips. The second accelerator I propose aims to speed up similarity measurement calculations for semantic search in the natural language processing space. By leveraging the latency hiding concepts of multi-threading and simple scheduling mechanisms, my design maximizes functional unit utilization. This similarity measurement accelerator provides speedups of 36x-42x over optimized software running on server-class cores, while requiring 56x-58x lower energy, and only 1.3% of the area.PhDComputer Science and EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/116712/1/prateekt_1.pd

MUSTI: Dynamic Prevention of Invalid Object Initialization Attacks

Invalid object initialization vulnerabilities have been identified since the 1990’s by a research group at Princeton University. These vulnerabilities are critical since they can be used to totally compromise the security of a Java virtual machine.Recently, such a vulnerability identified as CVE-2017-3289 has been found again in the bytecode verifier of the JVM and affects more than 40 versions of the JVM. In this paper, we present a runtime solution called MUSTIto detect and prevent attacks leveraging this kind of critical vulnerabilities. We optimize MUSTI to have a runtime overhead below 0.5% and a memory overhead below 0.42%. Compared to state-of-the-art, MUSTI is completely automated and does not require to manually annotate the code

DSpace Manual: Software version 1.5

DSpace is an open source software platform that enables organizations to: - Capture and describe digital material using a submission workflow module, or a variety of programmatic ingest options - Distribute an organization's digital assets over the web through a search and retrieval system - Preserve digital assets over the long term This system documentation includes a functional overview of the system, which is a good introduction to the capabilities of the system, and should be readable by nontechnical personnel. Everyone should read this section first because it introduces some terminology used throughout the rest of the documentation. For people actually running a DSpace service, there is an installation guide, and sections on configuration and the directory structure. Note that as of DSpace 1.2, the administration user interface guide is now on-line help available from within the DSpace system. Finally, for those interested in the details of how DSpace works, and those potentially interested in modifying the code for their own purposes, there is a detailed architecture and design section