Optimization strategies for large-scale distributed computing and data management in the presence of security and other requirements

The growth of e-science and grid computing has presented new opportunities to researchers working on distributed, collaborative projects, or for whom access to large computational and data-storage resources is essential. There is, however, much work to be done to improve the experience for these users, in particular by mitigating the effects that failing to consider fully the requirements of their processing tasks (or jobs) can have on the likelihood of success. Any improvement in this regard would be significant, not only because of the beneficial effect it would have on useful resource utilization, but also due to the improved perception of grid computing it would engender among its users. Allocating jobs to resources is never easy, and this is particularly true when the resources are of heterogeneous construction, geographically distant, and owned and operated by a variety of institutions. Attempts to allocate jobs intelligently must consider not only the strict hardware requirements of the job and whether these match a particular candidate resource, but must also consider requirements related to the security infrastructures in place (including, for example, the rights of the user who wishes to run the job, the licences associated with the application to be run and the data it will use and produce), how the job might affect other users of the resource, and indeed how the behaviour of other users of the resource might affect the job. This thesis examines the data, security and legal requirements of typical collaborative research projects, and discusses how these requirements suggest particular constraints that will govern any jobs which need to be executed. It reviews some of the technologies which are common in this field, and describes current state-of-the-art technology in this area. This thesis then presents a framework for describing requirements, along with an algorithmic method for allocating jobs to resources. Case studies and an analysis of performance are presented using these algorithms, which show how they build upon, and improve, the state-of-the-art in this domain