This paper describes the evaluation of DR-Advisor algorithms on \u27\u27The Great Energy Predictor Shootout - The First Building Data Analysis and Prediction Competition\u27\u27 held in 1993-94 by ASHRAE

Behl, Madhur

Mangharam, Rahul

English

Kosmopolis

University of Pennsylvania
ScholarlyCommons
Real-Time and Embedded Systems Lab (mLAB) School of Engineering and Applied Science
7-24-2014
Evaluation of DR-Advisor on the ASHRAE Great
Energy Predictor Shootout Challenge
Madhur Behl
University of Pennsylvania, mbehl@seas.upenn.edu
Rahul Mangharam
University of Pennsylvania, rahulm@seas.upenn.edu
Follow this and additional works at: http://repository.upenn.edu/mlab_papers
Part of the Computer Engineering Commons, and the Electrical and Computer Engineering
Commons
This paper is posted at ScholarlyCommons. http://repository.upenn.edu/mlab_papers/75
For more information, please contact libraryrepository@pobox.upenn.edu.
Recommended Citation
Madhur Behl and Rahul Mangharam, "Evaluation of DR-Advisor on the ASHRAE Great Energy Predictor Shootout Challenge", . July
2014.
Evaluation of DR-Advisor on the ASHRAE Great Energy Predictor
Shootout Challenge
Abstract
This paper describes the evaluation of DR-Advisor algorithms on ''The Great Energy Predictor Shootout - The
First Building Data Analysis and Prediction Competition'' held in 1993-94 by ASHRAE.
Disciplines
Computer Engineering | Electrical and Computer Engineering
This technical report is available at ScholarlyCommons: http://repository.upenn.edu/mlab_papers/75
Evaluation of DR-Advisor on the ASHRAE Great
Energy Predictor Shootout Challenge
Madhur Behl
Dept. of Electrical and Systems Engineering
University of Pennsylvania
{mbehl}@seas.upenn.edu
1. About the Competition
This paper describes the methods and results in making predictions for test
problem A given by ”The Great Energy Predictor Shootout - The First Building
Data Analysis and Prediction Competition” held in 1993-94 by ASHRAE.
Two distinct data sets were provided for prediction (training and testing).
Contestants were given these two sets of independent variables with the cor-
responding values of dependent variables, e.g., energy usage. The accuracy of
predictions of the dependent variables from values of independent variables from
the test data set was the criteria for judging this competition.
The following criteria was used by the organizers for assessing the respective
accuracies of the entries when analyzing the testing set:
CoefficientofVariation, CV
CV =
√
{∑nt=1(ŷt − y)2} /n
ȳ
The idea was to look for simpler models that may not have such a strong
physical basis, yet that perform well at prediction. The competition attracted
150 entrants, who attempted to predict the unseen power loads from weather
and solar radiation data using a variety of approaches. The winner of the
competition was an entry from David Mackay [1]. Mackays algorithm was based
on Bayesian modeling using neural networks, with an ”Automatic Relevance
Determination” (ARD) prior to help select the relevant variables from the large
number of possible inputs. Although this algorithm won the competition by
some margin, a large fraction of the other highly ranked algorithms were also
based on some form of neural network.
2. Data Description
The training data was a time record of hourly chilled water, hot water and
whole building electricity usage for a four-month period in an institutional build-
July 24, 2015
0 100 200 300 400 500 600 700 800 900 1,000 1,100 1,200 1,300 1,400 1,500 1,600 1,700 1,800 1,900 2,000 2,100 2,200 2,300 2,400 2,500 2,600 2,700 2,800 2,900 3,000
0
500
1,000
1,500
2,000
W
/m
2
Incident Solar Irradiation
0 100 200 300 400 500 600 700 800 900 1,000 1,100 1,200 1,300 1,400 1,500 1,600 1,700 1,800 1,900 2,000 2,100 2,200 2,300 2,400 2,500 2,600 2,700 2,800 2,900 3,000
0
1
2
3
·10−2
R
at
io
Humidity
0 100 200 300 400 500 600 700 800 900 1,000 1,100 1,200 1,300 1,400 1,500 1,600 1,700 1,800 1,900 2,000 2,100 2,200 2,300 2,400 2,500 2,600 2,700 2,800 2,900 3,000
0
5
10
15
20
TimeSteps
m
p
h
Wind
0 100 200 300 400 500 600 700 800 900 1,000 1,100 1,200 1,300 1,400 1,500 1,600 1,700 1,800 1,900 2,000 2,100 2,200 2,300 2,400 2,500 2,600 2,700 2,800 2,900 3,000
0
20
40
60
80
100
◦F
Temperature
Figure 1: Plot fo the training data for the ASHRAE Great Energy Predictor Shootout Chal-
lenge.
ing. Weather data and a time stamp were also included. The hourly values of
usage of these three energy forms was to be predicted for the two following
months. The testing set consisted of the two months following the four-month
training period. The training data had approximately 3000 samples taken hourly
during Sep - Dec 1989. The following information was provided for each time
step:
1. Outside temperature (◦F)
2. Wind speed (mph)
3. Humidity ratio (water/dry air)
4. Solar flux (W/m2)
5. Hour of Day
6. Whole building electricity, WBE (kWh/hr)
7. Whole building chilled water, CHW (millions of Btu/hr)
8. Whole building hot water, HW (millions of Btu/hr)
The corresponding date is also provided. The training data features or the
independent variables are shown in Figure 1. One of the dependent or response
variable (WBE) is plotted in Figure 2.
In addition to the variables which are provided, the date for each sample
point allows us to define proxy or context variables of our own. We define three
such proxy variables:
2
         
TimeSteps
0 500 1000 1500 2000 2500 3000
k
W
h
0
200
400
600
800
1000
1200
Whole Building Energy (Training Data)
Veterans Day
Extended
Weekend
due to
Thanksgiving
Christmas Week
M T W T F
SS
Figure 2: Plot of the whole building electricity (WBE) training data. It can be seen that
special days and weekends can be easily identified in the data.
1. Day of Week: This is a number which takes values from 1 to 7 based on
what day of the week it is. The intuition behind introducing this variable
is that any repeated building use patters fro a specific day will be captured
by this variable.
2. IsWeekend: This is a boolean variable which takes the value TRUE for
Saturdays and Sundays and FALSE otherwise.
3. IsHoliday: Sometimes the building might follow a modified schedule on
a weekday due to a holiday or a special day. By referring to the 1989
calender or the training data duration, we are able to identify days in the
training and testing data which are special days or holidays.
The motivation of introducing the proxy variables can be understood from
Figure 2. One can note how weekday consumption patters appear similar, except
on certain holidays where the power consumption is irregular or seems to follow
a weekend schedule (e.g., Thanksgiving). Likewise, the building has an irregular
power consumption profile during the entire Christmas week, probably because
of a vacation schedule.
3. Results
We run the random forests method on the training data to learn three dif-
ferent models, one each for predicting the whole building energy WBE, chilled
water consumption, CHW and hot water consumption HW. The test data con-
sists of 1282 samples of weather information and date from the first two months
of 1990. Our objective is to predict the WBE, CHW and HW for these two
months.
3
0 10 20 30 40 50 60 70 80 90 100 110 120 130 140 150 160 170 180 190 200 210 220 230 240 250 260 270 280 290 300
60
65
70
75
80
85
90
95
100
105
110
115
120
125
130
135
140
Number of trees
R
M
S
E
(k
W
h
/h
r)
Random Forests Resubstitution Error
Figure 3: Resubstitution error is shown for the number of trees in the random forest method.
Table 1: Shootout Competition Results
ASHRAE Team ID WBE CV CHW CV HW CV Average CV
9 10.36 13.02 15.24 12.87
Random Forests 11.72 14.88 28.13 18.24
6 11.78 12.97 30.63 18.46
3 12.79 12.78 30.98 18.85
2 11.89 13.69 31.65 19.08
7 13.81 13.63 30.57 19.34
0 50 100 150 200 250 300 350 400 450 500 550 600 650 700 750 800 850 900 950 1,000 1,050 1,100 1,150 1,200 1,250 1,300
0
100
200
300
400
500
600
700
800
900
1,000
TimeSteps
k
W
h
Whole building electricity, WBE (Testing Data)
Ground Truth
Prediction
Figure 4: Comparison between predicted and ground truth values for WBE for the testing
data.
4
The comparison between the predicted WBE values and the ground truth are
shown in Figure 4. The coefficient of variation is 11.72%. The re-substitution
error (on the training data) is plotted for a different number of trees used by
the random forest in Figure 3. In the actual competition, the winners were
selected based on the accuracy of all predictions as measured by the coefficient
of variation statistic CV. The smaller the value of CV, the better the prediction
accuracy. ASHRAE released the results of the competition for the top 19 entries
which they received. In Table 1, we list the performance of the top 5 winners
of the competition and compare our results with them.
It can be seen from table 1, that the random forest method ranks 2nd in
terms of WBE CV and the overall average CV. The other algorithms in the ta-
ble are (9) Mackays Bayesian Non-Linear Modeling, (6) Ohlssons Feedforward
Multi-layer Perceptron, (2) Feustons Neural Network with Pre and Post Process-
ing. Algorithm (9) (Mackay) generates the best results overall, beating many
other neural network implementations in the competition, which suggests that
it is some particular feature of this implementation, e.g., input preprocessing,
network architecture, or training approach, that is important.
However, the results we obtain clearly demonstrate that random forests can
generate predictive performance that is comparable with the ASHRAE Shootout
winners.
References
[1] David JC MacKay et al. Bayesian nonlinear modeling for the prediction
competition. ASHRAE transactions, 100(2):1053–1062, 1994.
5


Evaluation of DR-Advisor on the ASHRAE Great Energy Predictor Shootout Challenge

ScholarlyCommons@Penn

University of PennsylvaniaScholarlyCommonsReal-Time and Embedded Systems Lab (mLAB) School of Engineering and Applied Science7-24-2014Evaluation of DR-Advisor on the ASHRAE GreatEnergy Predictor Shootout ChallengeMadhur BehlUniversity of Pennsylvania, mbehl@seas.upenn.eduRahul MangharamUniversity of Pennsylvania, rahulm@seas.upenn.eduFollow this and additional works at: http://repository.upenn.edu/mlab_papersPart of the Computer Engineering Commons, and the Electrical and Computer EngineeringCommonsThis paper is posted at ScholarlyCommons. http://repository.upenn.edu/mlab_papers/75For more information, please contact libraryrepository@pobox.upenn.edu.Recommended CitationMadhur Behl and Rahul Mangharam, "Evaluation of DR-Advisor on the ASHRAE Great Energy Predictor Shootout Challenge", . July2014.Evaluation of DR-Advisor on the ASHRAE Great Energy PredictorShootout ChallengeAbstractThis paper describes the evaluation of DR-Advisor algorithms on ''The Great Energy Predictor Shootout - TheFirst Building Data Analysis and Prediction Competition'' held in 1993-94 by ASHRAE.DisciplinesComputer Engineering | Electrical and Computer EngineeringThis technical report is available at ScholarlyCommons: http://repository.upenn.edu/mlab_papers/75Evaluation of DR-Advisor on the ASHRAE GreatEnergy Predictor Shootout ChallengeMadhur BehlDept. of Electrical and Systems EngineeringUniversity of Pennsylvania{mbehl}@seas.upenn.edu1. About the CompetitionThis paper describes the methods and results in making predictions for testproblem A given by ”The Great Energy Predictor Shootout - The First BuildingData Analysis and Prediction Competition” held in 1993-94 by ASHRAE.Two distinct data sets were provided for prediction (training and testing).Contestants were given these two sets of independent variables with the cor-responding values of dependent variables, e.g., energy usage. The accuracy ofpredictions of the dependent variables from values of independent variables fromthe test data set was the criteria for judging this competition.The following criteria was used by the organizers for assessing the respectiveaccuracies of the entries when analyzing the testing set:CoefficientofVariation, CVCV =√{∑nt=1(yˆt − y)2} /ny¯The idea was to look for simpler models that may not have such a strongphysical basis, yet that perform well at prediction. The competition attracted150 entrants, who attempted to predict the unseen power loads from weatherand solar radiation data using a variety of approaches. The winner of thecompetition was an entry from David Mackay [1]. Mackays algorithm was basedon Bayesian modeling using neural networks, with an ”Automatic RelevanceDetermination” (ARD) prior to help select the relevant variables from the largenumber of possible inputs. Although this algorithm won the competition bysome margin, a large fraction of the other highly ranked algorithms were alsobased on some form of neural network.2. Data DescriptionThe training data was a time record of hourly chilled water, hot water andwhole building electricity usage for a four-month period in an institutional build-July 24, 20150 100 200 300 400 500 600 700 800 900 1,000 1,100 1,200 1,300 1,400 1,500 1,600 1,700 1,800 1,900 2,000 2,100 2,200 2,300 2,400 2,500 2,600 2,700 2,800 2,900 3,00005001,0001,5002,000W/m2Incident Solar Irradiation0 100 200 300 400 500 600 700 800 900 1,000 1,100 1,200 1,300 1,400 1,500 1,600 1,700 1,800 1,900 2,000 2,100 2,200 2,300 2,400 2,500 2,600 2,700 2,800 2,900 3,0000123·10−2RatioHumidity0 100 200 300 400 500 600 700 800 900 1,000 1,100 1,200 1,300 1,400 1,500 1,600 1,700 1,800 1,900 2,000 2,100 2,200 2,300 2,400 2,500 2,600 2,700 2,800 2,900 3,00005101520TimeStepsmphWind0 100 200 300 400 500 600 700 800 900 1,000 1,100 1,200 1,300 1,400 1,500 1,600 1,700 1,800 1,900 2,000 2,100 2,200 2,300 2,400 2,500 2,600 2,700 2,800 2,900 3,000020406080100◦FTemperatureFigure 1: Plot fo the training data for the ASHRAE Great Energy Predictor Shootout Chal-lenge.ing. Weather data and a time stamp were also included. The hourly values ofusage of these three energy forms was to be predicted for the two followingmonths. The testing set consisted of the two months following the four-monthtraining period. The training data had approximately 3000 samples taken hourlyduring Sep - Dec 1989. The following information was provided for each timestep:1. Outside temperature (◦F)2. Wind speed (mph)3. Humidity ratio (water/dry air)4. Solar flux (W/m2)5. Hour of Day6. Whole building electricity, WBE (kWh/hr)7. Whole building chilled water, CHW (millions of Btu/hr)8. Whole building hot water, HW (millions of Btu/hr)The corresponding date is also provided. The training data features or theindependent variables are shown in Figure 1. One of the dependent or responsevariable (WBE) is plotted in Figure 2.In addition to the variables which are provided, the date for each samplepoint allows us to define proxy or context variables of our own. We define threesuch proxy variables:2         TimeSteps0 500 1000 1500 2000 2500 3000kWh020040060080010001200Whole Building Energy (Training Data)Veterans DayExtendedWeekenddue toThanksgivingChristmas WeekM T W T FSSFigure 2: Plot of the whole building electricity (WBE) training data. It can be seen thatspecial days and weekends can be easily identified in the data.1. Day of Week: This is a number which takes values from 1 to 7 based onwhat day of the week it is. The intuition behind introducing this variableis that any repeated building use patters fro a specific day will be capturedby this variable.2. IsWeekend: This is a boolean variable which takes the value TRUE forSaturdays and Sundays and FALSE otherwise.3. IsHoliday: Sometimes the building might follow a modified schedule ona weekday due to a holiday or a special day. By referring to the 1989calender or the training data duration, we are able to identify days in thetraining and testing data which are special days or holidays.The motivation of introducing the proxy variables can be understood fromFigure 2. One can note how weekday consumption patters appear similar, excepton certain holidays where the power consumption is irregular or seems to followa weekend schedule (e.g., Thanksgiving). Likewise, the building has an irregularpower consumption profile during the entire Christmas week, probably becauseof a vacation schedule.3. ResultsWe run the random forests method on the training data to learn three dif-ferent models, one each for predicting the whole building energy WBE, chilledwater consumption, CHW and hot water consumption HW. The test data con-sists of 1282 samples of weather information and date from the first two monthsof 1990. Our objective is to predict the WBE, CHW and HW for these twomonths.30 10 20 30 40 50 60 70 80 90 100 110 120 130 140 150 160 170 180 190 200 210 220 230 240 250 260 270 280 290 3006065707580859095100105110115120125130135140Number of treesRMSE(kWh/hr)Random Forests Resubstitution ErrorFigure 3: Resubstitution error is shown for the number of trees in the random forest method.Table 1: Shootout Competition ResultsASHRAE Team ID WBE CV CHW CV HW CV Average CV9 10.36 13.02 15.24 12.87Random Forests 11.72 14.88 28.13 18.246 11.78 12.97 30.63 18.463 12.79 12.78 30.98 18.852 11.89 13.69 31.65 19.087 13.81 13.63 30.57 19.340 50 100 150 200 250 300 350 400 450 500 550 600 650 700 750 800 850 900 950 1,000 1,050 1,100 1,150 1,200 1,250 1,30001002003004005006007008009001,000TimeStepskWhWhole building electricity, WBE (Testing Data)Ground TruthPredictionFigure 4: Comparison between predicted and ground truth values for WBE for the testingdata.4The comparison between the predicted WBE values and the ground truth areshown in Figure 4. The coefficient of variation is 11.72%. The re-substitutionerror (on the training data) is plotted for a different number of trees used bythe random forest in Figure 3. In the actual competition, the winners wereselected based on the accuracy of all predictions as measured by the coefficientof variation statistic CV. The smaller the value of CV, the better the predictionaccuracy. ASHRAE released the results of the competition for the top 19 entrieswhich they received. In Table 1, we list the performance of the top 5 winnersof the competition and compare our results with them.It can be seen from table 1, that the random forest method ranks 2nd interms of WBE CV and the overall average CV. The other algorithms in the ta-ble are (9) Mackays Bayesian Non-Linear Modeling, (6) Ohlssons FeedforwardMulti-layer Perceptron, (2) Feustons Neural Network with Pre and Post Process-ing. Algorithm (9) (Mackay) generates the best results overall, beating manyother neural network implementations in the competition, which suggests thatit is some particular feature of this implementation, e.g., input preprocessing,network architecture, or training approach, that is important.However, the results we obtain clearly demonstrate that random forests cangenerate predictive performance that is comparable with the ASHRAE Shootoutwinners.References[1] David JC MacKay et al. Bayesian nonlinear modeling for the predictioncompetition. ASHRAE transactions, 100(2):1053–1062, 1994.5

https://repository.upenn.edu/cgi/viewcontent.cgi?article=1093&amp;context=mlab_papers

Evaluation of DR-Advisor on the ASHRAE Great Energy Predictor Shootout Challenge

Abstract

Similar works

Full text

Available Versions

Kosmopolis

ScholarlyCommons@Penn