A study of different web-crawler behaviour

The article deals with a study of web-crawler behaviour on different websites. A classification of web-robots, information gathering tools and their detection methods are provided. Well-known scrapers and their behaviour are analyzed on the base of large web-server log set. Experimental results demonstrate that web-robot can be distinguished from human by feature analysis. The results of the research can be used as a basis for comprehensive intrusion detection and prevention system development

Denial of Service in Web-Domains: Building Defenses Against Next-Generation Attack Behavior

The existing state-of-the-art in the field of application layer Distributed Denial of Service (DDoS) protection is generally designed, and thus effective, only for static web domains. To the best of our knowledge, our work is the first that studies the problem of application layer DDoS defense in web domains of dynamic content and organization, and for next-generation bot behaviour. In the first part of this thesis, we focus on the following research tasks: 1) we identify the main weaknesses of the existing application-layer anti-DDoS solutions as proposed in research literature and in the industry, 2) we obtain a comprehensive picture of the current-day as well as the next-generation application-layer attack behaviour and 3) we propose novel techniques, based on a multidisciplinary approach that combines offline machine learning algorithms and statistical analysis, for detection of suspicious web visitors in static web domains. Then, in the second part of the thesis, we propose and evaluate a novel anti-DDoS system that detects a broad range of application-layer DDoS attacks, both in static and dynamic web domains, through the use of advanced techniques of data mining. The key advantage of our system relative to other systems that resort to the use of challenge-response tests (such as CAPTCHAs) in combating malicious bots is that our system minimizes the number of these tests that are presented to valid human visitors while succeeding in preventing most malicious attackers from accessing the web site. The results of the experimental evaluation of the proposed system demonstrate effective detection of current and future variants of application layer DDoS attacks

Detecting Web Robots Using Resource Request Patterns

A significant proportion of Web traffic is now attributed to Web robots, and this proportion is likely to grow over time. These robots may threaten the security, privacy, functionality, and performance of a Web server due to their unregulated crawling behavior. Therefore, to assess their impact, it must be possible to accurately detect Web robot requests. Contemporary detection approaches, however, may cease to be effective as the behavior of both robots and humans evolves. In this paper, we present a novel detection approach that is based on the contrasts in the resource request patterns of robots and humans. The proposed scheme, which relies on an invariant behavioral difference between humans and robots, builds on the lessons from contemporary approaches. We demonstrate that the proposed approach can accurately detect Web robots and argue that it is expected to remain effective even as they continue their rapid evolution