Exploring the Fairness and Resource Distribution in an Apache Mesos Environment

Apache Mesos, a cluster-wide resource manager, is widely deployed in massive scale at several Clouds and Data Centers. Mesos aims to provide high cluster utilization via fine grained resource co-scheduling and resource fairness among multiple users through Dominant Resource Fairness (DRF) based allocation. DRF takes into account different resource types (CPU, Memory, Disk I/O) requested by each application and determines the share of each cluster resource that could be allocated to the applications. Mesos has adopted a two-level scheduling policy: (1) DRF to allocate resources to competing frameworks and (2) task level scheduling by each framework for the resources allocated during the previous step. We have conducted experiments in a local Mesos cluster when used with frameworks such as Apache Aurora, Marathon, and our own framework Scylla, to study resource fairness and cluster utilization. Experimental results show how informed decision regarding second level scheduling policy of frameworks and attributes like offer holding period, offer refusal cycle and task arrival rate can reduce unfair resource distribution. Bin-Packing scheduling policy on Scylla with Marathon can reduce unfair allocation from 38\% to 3\%. By reducing unused free resources in offers we bring down the unfairness from to 90\% to 28\%. We also show the effect of task arrival rate to reduce the unfairness from 23\% to 7\%

Non-Cooperative Scheduling of Multiple Bag-of-Task Applications

Multiple applications that execute concurrently on heterogeneous platforms compete for CPU and network resources. In this paper we analyze the behavior of $K$ non-cooperative schedulers using the optimal strategy that maximize their efficiency while fairness is ensured at a system level ignoring applications characteristics. We limit our study to simple single-level master-worker platforms and to the case where each scheduler is in charge of a single application consisting of a large number of independent tasks. The tasks of a given application all have the same computation and communication requirements, but these requirements can vary from one application to another. In this context, we assume that each scheduler aims at maximizing its throughput. We give closed-form formula of the equilibrium reached by such a system and study its performance. We characterize the situations where this Nash equilibrium is optimal (in the Pareto sense) and show that even though no catastrophic situation (Braess-like paradox) can occur, such an equilibrium can be arbitrarily bad for any classical performance measure

Increasing Block Tariffs in the Water Sector: A Semi-Welfarist Approach

We analyze the properties of progressive water tariffs that are often applied in the sector in the form of discretely increasing block tariffs (IBT). We are particularly interested in water tarification in a poverty context where a subsistence level of water has to be allocated to each household. Our approach is "semi-welfarist" to the extent that we analyze second-best pricing schemes that may be applied in practice due to "fairness" or other, non-welfarist considerations. In our theoretical model we compare a modified Coase-tariff and a progressively increasing block tariff with respect to water consumption, water expenses and utility levels. When we impose cost coverage on the water utility, there are clearly adverse effects on the "almost poor" by introducing a progressive tariff. This result is supported with a numerical application using real data from Bangladesh: progressive tariffs may fail to achieve "fair" cross-subsidization of low-income groups.water, tarification, prices, fairness, distribution, institutions

Disentangling and Operationalizing AI Fairness at LinkedIn

Operationalizing AI fairness at LinkedIn's scale is challenging not only because there are multiple mutually incompatible definitions of fairness but also because determining what is fair depends on the specifics and context of the product where AI is deployed. Moreover, AI practitioners need clarity on what fairness expectations need to be addressed at the AI level. In this paper, we present the evolving AI fairness framework used at LinkedIn to address these three challenges. The framework disentangles AI fairness by separating out equal treatment and equitable product expectations. Rather than imposing a trade-off between these two commonly opposing interpretations of fairness, the framework provides clear guidelines for operationalizing equal AI treatment complemented with a product equity strategy. This paper focuses on the equal AI treatment component of LinkedIn's AI fairness framework, shares the principles that support it, and illustrates their application through a case study. We hope this paper will encourage other big tech companies to join us in sharing their approach to operationalizing AI fairness at scale, so that together we can keep advancing this constantly evolving field

Procedural fairness requirements in decision-making: Legal issues and challenges for government secondary school principals in New South Wales

The application of the rules of procedural fairness, which is an element of administrative law, is an area of law that has not been previously examined in the context of government (public) secondary school principals in New South Wales (‘NSW’). Using a basic qualitative case study design, this study sought to discover the processes that these principals undertook in applying the rules of procedural fairness when managing student discipline, special education and industrial relations. The study examined to what extent New South Wales government (public) secondary school principals were equipped to make decisions that are consistent with the administrative law principles of procedural fairness. NSW Department of Education secondary school principals, in-house legal officers and external lawyers were interviewed to ascertain how school principals undertook the complex and challenging task of decision-making in accordance with the rules of procedural fairness given they receive no formal training. The study provides findings in terms of four broad themes which are developed from case law, literature and the study (procedural fairness in policy and procedures; student wellbeing and procedural fairness; industrial relations and procedural fairness; and legal training in procedural fairness) where the rules of procedural fairness dictate the process a government secondary school principal ought to undertake. The study found that NSW government secondary school principals did undertake the application of the rules of procedural fairness to an appropriate standard; however, the ways in which the participants undertook informal learning at the deputy principal level could be an area for improvement by the NSW Department of Education prior to individuals being appointed to principalship to reduce any actual or perceived risk