A Prototype Toolkit For Evaluating Indoor Environmental Quality In Commercial Buildings

Measurement of building environmental parameters is often complex, expensive, and not easily proceduralized in a manner that covers all commercial buildings. Evaluating building indoor environmental quality performance is therefore not standard practice. This project developed a prototype toolkit that addressed existing barriers to widespread indoor environmental quality performance evaluation. A toolkit with both hardware and software elements was designed for practitioners around the indoor environmental quality requirements of the American Society of Heating, Refrigeration and Air Conditioning Engineers / Chartered Institution of Building Services / United States Green Building Council Performance Measurement Protocols. This unique toolkit was built on a wireless mesh network with a web-based data collection, analysis, and reporting application. The toolkit provided a fast, robust deployment of sensors, real-time data analysis, Performance Measurement Protocol-based analysis methods and a scorecard and report generation tools. A web-enabled Geographic Information System-based metadata collection system also reduced field-study deployment time. The toolkit was evaluated through three case studies, which were discussed in this report

What's the Difference? How Foundation Trustees View Evaluation

Trustee Evaluation ToolkitTrustees care deeply about impact. Understanding results is part of their fiduciary duty. As foundations strive to improve performance, advance accountability and share knowledge, their desire for evaluation -- reliable data on organizational effectiveness -- grows. Based on discussions with trustees, we've heard that current evaluation approaches don't always generate useful information. In too many cases, foundation evaluation practices don't align with trustee needs. Trustees across the United States believe there are ways to improve how we determine the effectiveness of social investments. FSG Social Impact Advisors, with funding from the James Irvine Foundation, interviewed dozens of foundation trustees, CEOs and evaluation experts to uncover critical issues and exciting ideas related to evaluation. This "toolkit" shares highlights from these interviews, and explores innovative new approaches

DiPerF: an automated DIstributed PERformance testing Framework

We present DiPerF, a distributed performance testing framework, aimed at simplifying and automating service performance evaluation. DiPerF coordinates a pool of machines that test a target service, collects and aggregates performance metrics, and generates performance statistics. The aggregate data collected provide information on service throughput, on service "fairness" when serving multiple clients concurrently, and on the impact of network latency on service performance. Furthermore, using this data, it is possible to build predictive models that estimate a service performance given the service load. We have tested DiPerF on 100+ machines on two testbeds, Grid3 and PlanetLab, and explored the performance of job submission services (pre WS GRAM and WS GRAM) included with Globus Toolkit 3.2.Comment: 8 pages, 8 figures, will appear in IEEE/ACM Grid2004, November 200

MultiMedEval: A Benchmark and a Toolkit for Evaluating Medical Vision-Language Models

We introduce MultiMedEval, an open-source toolkit for fair and reproducible evaluation of large, medical vision-language models (VLM). MultiMedEval comprehensively assesses the models' performance on a broad array of six multi-modal tasks, conducted over 23 datasets, and spanning over 11 medical domains. The chosen tasks and performance metrics are based on their widespread adoption in the community and their diversity, ensuring a thorough evaluation of the model's overall generalizability. We open-source a Python toolkit (github.com/corentin-ryr/MultiMedEval) with a simple interface and setup process, enabling the evaluation of any VLM in just a few lines of code. Our goal is to simplify the intricate landscape of VLM evaluation, thus promoting fair and uniform benchmarking of future models.Comment: Under review at MIDL 202

BGC-val: a model- and grid-independent Python toolkit to evaluate marine biogeochemical models

The biogeochemical evaluation toolkit, BGC-val, is a model- and grid-independent Python toolkit that has been built to evaluate marine biogeochemical models using a simple interface. Here, we present the ideas that motivated the development of the BGC-val software framework, introduce the code structure, and show some applications of the toolkit using model results from the Fifth Climate Model Intercomparison Project (CMIP5). A brief outline of how to access and install the repository is presented in Appendix A, but the specific details on how to use the toolkit are kept in the code repository. The key ideas that directed the toolkit design were model and grid independence, front-loading analysis functions and regional masking, interruptibility, and ease of use. We present each of these goals, why they were important, and what we did to address them. We also present an outline of the code structure of the toolkit illustrated with example plots produced by the toolkit. After describing BGC-val, we use the toolkit to investigate the performance of the marine physical and biogeochemical quantities of the CMIP5 models and highlight some predictions about the future state of the marine ecosystem under a business-as-usual CO2 concentration scenario (RCP8.5)