Research

# A decision support system for benchmarking the energy and waste performance of schools in Toronto

Julian Scott Yeomans

Author Affiliations

Operations Management & Information Systems Area, Schulich School of Business, York University, 4700 Keele Street, Toronto, Ontario, M3J 1P3, Canada

Environmental Systems Research 2012, 1:5  doi:10.1186/2193-2697-1-5

 Received: 16 May 2012 Accepted: 14 August 2012 Published: 14 August 2012

This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

### Abstract

#### Background

The Toronto District School Board (TDSB) oversees the largest school district in Canada and has been spent more than one third of its annual maintenance budget on energy and waste. This has directed attention toward system-wide reductions to both energy consumption patterns and waste generation rates. In this paper, a decision support system (DSS) that can process unit-incompatible measures is used for rating, ranking, and benchmarking the schools within the TDSB.

#### Results

The DSS permits the ranking of any set of schools by contextually evaluating their relative attractiveness to other identified school groupings. Consequently, the DSS was used to explicitly rank each school’s performance within the district and to determine realistic energy improvement targets. Achieving these benchmarks would reduce system-wide energy costs by twenty-five percent.

#### Conclusions

The TDSB study demonstrates that this DSS provides an extremely useful approach for evaluating, benchmarking and ranking the relative energy and waste performance within the school system, and the potential to extend its much broader applicability into other applications clearly warrants additional exploration.

##### Keywords:
Benchmarking Performance; Data Envelopment Analysis; Decision Support Systems; Energy & Waste

### Background

Representing more than 600 schools, the Toronto District School Board (TDSB) oversees the largest education constituency within Canada. In the 2003 fiscal year, the TDSB allocated $48 million of its annual budget to the energy and waste requirements of the school system, but within three years, found that the energy and waste expenditures had escalated to more than$69 million (Christie [2003,2007]; Christie & Coppinger [2006]). This rapid forty percent increase in spending during a period of public economic retrenchment necessitated stringent attention toward system-wide reductions to both energy consumption patterns and waste generation rates (Christie [2003,2007]; Christie & Coppinger [2006]).

While the need to decrease energy consumption and minimize waste was universally acknowledged, to achieve long-term success in these efforts, the TDSB resolutely believed that any successful systematic reduction efforts would need to adequately address three critical questions: (i) what initiatives would achieve the most effective results; (ii) how could these “most effective reduction initiatives” actually be identified; and, (iii) how effective would these initiatives prove to be once implemented (Christie [2003]) Additional “essential components” to effectively implement any proposed reduction schemes necessitated that improvement initiatives had to include: (i) the establishment of realistic benchmarks for each school to strive toward; (ii) the setting of achievable annual performance targets for each school; (iii) a commitment to rational and objective management of the system data; and, (iv) an assurance of transparency and neutrality in policy-setting and decision-making (Christie [2003,2007]; Christie & Coppinger [2006]). Unfortunately, it proves to be an extremely difficult process to evaluate system-wide performance and to establish benchmarks when there are multiple incommensurate criteria measurements present (Camp [1995]), which was the case within the budgetary-constrained and politically-charged environment of the TDSB (Christie [2003,2007]; Christie & Coppinger [2006]).

Decision support systems (DSS) are intelligent information systems based on decision models that can be used to extract large quantities of data from databases, to provide interfaces and methods for effectively processing it, and for deriving meaningful decisions of managerial/economic significance from it. DSS have been used to analyze a wide variety of performance information and to provide a readily-accessible medium for distributing any knowledge generated to a wide variety of stakeholders (Lin et al. [2008]). In this paper, a DSS that can simultaneously combine unit-incompatible energy and waste performance measures is introduced for evaluating the system-wide energy and waste performance of the schools within the TDSB. This DSS incorporates several data envelopment analysis (DEA) modules that had been developed in Yeomans ([2004]) and threads these modules together using commonly available software (Albright [2010]; Seref et al. [2007]; Zhu [2003]). The underlying DEA methodology has been shown to hold advantages over many other multi-criteria methods by providing an objective decision-making tool that does not require variables to have the same scale or conversion weights applied to them (Cook & Zhu [2008]; Zhu [2003]), while permitting a simultaneous combination of both quantitative and qualitative measures (Cook et al. [1996]).

The DSS developed can be used to rate and rank each school according to its energy and waste performance relative to the other schools in the system by recursively partitioning the schools into sub-groups of relatively superior and inferior performers. In addition, the DSS can establish realistically achievable improvement targets for each school by benchmarking their performance against the operations of schools in higher efficiency categories. This relative performance comparison between the schools is important since it ensures that the underlying analysis involves the use of energy and waste values actually occurring at peer institutions within the TDSB and not through “externally-generated”, potentially unrepresentative values.

The DSS developed directly addresses the TDSB’s three requisite “critical questions” while simultaneously addressing most of the key issues in the “essential components” identified above. Furthermore, the most practical contribution from this approach is that, since DEA can be readily implemented using common spreadsheet software linked together by relatively straightforward VBA programming (Albright [2010]; Seref et al. [2007]; Zhu [2003]), this entire methodology can be implemented on virtually any computer. Consequently, practitioners in any organization could easily modify and extend this methodology to match their own very specific multi-criteria applications and circumstances. An illustrative example of an application of the DSS is provided through an analysis of a subset of the high schools in the TDSB.

### Results and Discussion

The primary purpose in the performance evaluations of most organizational systems is to appraise the current operations of individual entities and to benchmark these against peer entities to identify best practices. While individual performance measure, or “gap”, analysis has often provided the fundamental basis for performance evaluation and benchmarking (Zhu [2003]), it remains a difficult task to satisfactorily combine multiple disparate, unit-incompatible, single-criteria measurements into an overall conclusion (Camp [1995]). Since a complex entity’s actual performance generally represents multifaceted phenomena, the use of single measures in gap analysis explicitly ignores all interactions, substitutions, and tradeoffs between the various performance measures. Therefore, it is a rare occurrence when a one-measure-at-a-time gap-analysis can suffice for the purposes of an effective performance evaluation of organizational systems (Camp [1995]; Zhu [2003]).

Clearly it is difficult to evaluate an organization’s performance or to establish benchmarks when there are multiple measurements present (Camp [1995]). If the specific algebraic functional relationship between performance measures is known, then established multi-criteria techniques can be used to estimate best-practice levels of performance. These algebraic functional forms cannot be specified without a priori information on the corresponding tradeoffs and, unfortunately, such information is generally unavailable in most practical situations. However, when the best practices of similar types of operations can be identified empirically at a specific point in time, it becomes possible to empirically estimate the resulting best-practice level of performance, or efficient frontier, using these observations (Zhu [2003]).

DEA has proved to be an effective empirical tool for identifying multi-criteria efficient frontiers, for subsequently evaluating relative performance efficiencies, and for implicitly estimating the tradeoffs inherent within the empirically designed frontier. DEA’s empirical orientation and absence of a priori assumptions establish it as the ideal analytical instrument for application to a wide variety of practical situations, since its underlying theoretical basis is consistent with the practice of rating entities by concurrently examining the relative efficiencies of their multiple performance measures. Furthermore, DEA allows performance comparisons to be made between numerous entities that have been evaluated by multiple, unit-incomparable measures without employing any a priori weightings or conversion factors typically required in other methods. Since the process is adaptable and invariant to data type, DEA permits the inclusion and comparison of non-numeric environmental-type variables that might prove incomparable using many other techniques (Zhu [2003]). By incorporating DEA into its DSS, the focus of performance evaluation for the TDSB application could shift from a characterization of each school’s energy and waste usage in terms of single measures to evaluating performance from a mathematically rational, multidimensional system perspective (Linton et al. [2007]; Yeomans [2004]). Consequently, inherent energy and waste relationships and their role in performance ranking can be explicitly brought into the analysis in a rational, transparent, and neutral fashion.

#### An overview of the energy and waste DSS for the TDSB

The DSS, itself, contains a series of specific DEA modules (more fully explained in subsequent sections) for conducting the performance evaluation of the TDSB. The first module recursively partitions the selected schools into sub-groups of relatively superior and inferior performers according to their energy and waste performance. The second module then determines realistically achievable energy and waste improvement targets for each school by benchmarking them against the system-wide operations for schools in selected higher performance categories. If desired, the DSS can also execute a third module to establish an explicit rank ordering of each school relative to any desired set of schools within the TDSB.

The set of DEA modules was created using readily available spreadsheet optimization software linked together by a combination of straightforward programming. In order to make the entire analysis process readily accessible to the various system users, the DSS was implemented using Microsoft Access and Microsoft Excel together with VBA (contained in all Microsoft Office products) as the programming language. By using these specific computer packages, the entire set of modules for evaluating the performance of the TDSB was created using software residing on most current personal computers. In the following sections, each of the individual DEA modules used in the DSS is explained in detail and illustrated using data from 65 high schools in the TDSB.

#### Rank ordering module

While the stratification module partitions the schools into distinct levels of energy and waste performers, it does not rank order the standings of any of the schools within each grouping. If only a small number of schools were under consideration in an analysis or if a large number of groupings each containing only a very small number of schools had been produced, then the stratification, itself, might be sufficient for actually ranking the specific schools. For the general case, however, the stratification might not prove restrictive enough to permit sufficient degrees of preference discrimination. If this situation proves to be the case, then a contextual attractiveness concept can be incorporated into the DSS to permit an explicit ranking of the schools (Simonson & Tversky [1992]; Tversky & Simonson [1993]). Obtaining relative attractiveness scores for the schools requires that relative performance be defined with respect to some particular evaluation context and the L partitions from the stratification module can be used to supply these contexts.

Define to be the d-degree, d = 1,…, L - l0, contextual attractiveness of DMUq = (xq, yq) from some specific level , l0 ∈ {1,…, L – 1}. can be calculated by solving the model:

(6)

subject to:

(7)

(8)

(9)

(10)

DMUq is viewed as a more attractive option than another DMU if it possesses a larger value for its contextual attractiveness measure . Hence, it is possible to rank each school within each stratum using a direct sorting of these contextual attractiveness scores. Since all schools within a contextual grouping are considered better performers than any school in a lower contextual group, when the rankings within each grouping are subsequently concatenated, a complete, rank ordering of the entire set of selected schools will be produced, de facto.

Using the next-lower stratification grouping as the appropriate context, the attractiveness scores , for each high school were calculated. Since any school is considered more efficient than the other schools within its grouping if it possesses a larger contextual attractiveness score, it now becomes possible to rank order the schools within each stratum by sorting these scores. Concatenating these individually sorted groupings produces a comprehensive rank ordering from 1 to 65 based upon the energy and waste usage in all of the schools. The results of this contextual scoring system, the subsequent sorting within each partition, and the overall efficiency ranking of each high school within the TDSB are shown in Table 3.

Table 3. Overall rankings of high schools based upon contextual attractiveness scores calculated relative to the next lower partitioning group

If a large number of schools existed within any particular level after the stratification stage, then a situation might occur in which two or more of the schools each received exactly the same attractiveness measure. This tied ranking problem can be alleviated by incorporating a lexicographic ranking modification into the procedure described above and could be accomplished in the following manner. In order to reduce the likelihood of tied rankings, the lexicographic procedure would be used to calculate attractiveness measures for each school by using every level lower than it as the contextual basis. Hence, each school in the highest level would receive L-1 separate contextual attractiveness scores, each school in the second highest level would receive L-2 separate attractiveness scores, and so on. A lexicographical rank ordering of schools within each specific grouping would then be performed with greater emphasis given to attractiveness scores calculated from closer contextual groupings (i.e. through an alphabetical or dictionary style of sorting). This multi-scoring, lexicographic rank ordering process would produce additional discriminating powers by reducing the likelihood of ties occurring between schools within any specific grouping. However, since no tied-scores occurred in the contextual attractiveness calculations already performed, this lexicographic ranking produces the same rank ordering as Table 3.

### Conclusions

In this paper, a DEA-based DSS for analyzing, rating, ranking, and benchmarking the multi-criteria energy and waste system of the schools in the TDSB has been studied. Several benefits of this DSS were demonstrated through an illustrative investigation of 65 Toronto high schools. The DSS stratified the schools into similarly-efficient groupings based upon their energy and waste usage. The DSS then generated realistic energy and waste improvement targets for any relatively inefficient school against another grouping of benchmarked schools. Amongst other findings, it was shown that achieving these target reductions would produce system-wide energy cost savings of twenty-five percent. The DSS also permits the ranking of any set of schools by contextually evaluating their relative attractiveness to other identified school groupings. The findings with respect to the high schools have been extended in an analysis of all 600 schools within the TDSB. Based upon the TDSB study, DEA has shown itself to be an extremely useful approach for evaluating, benchmarking and ranking the relative energy and waste performance within the school system, and the potential for its much broader applicability to other applications clearly warrants additional exploration.

### Methods

The mathematical models of each of the DEA modules were created as separate computer models within Microsoft Excel worksheets (Albright [2010]; Seref et al. [2007]; Zhu [2003]). The optimization of the spreadsheets of these DEA models was performed using the standard, built-in Excel Solver function. The data for all 600 schools in the TDSB can be found in Christie ([2003]) and the data for the 65 schools used in the analysis appears in Table 1. All data was stored in a Microsoft Access database. The programming language used to link together the various components was VBA which is contained in all Microsoft Office products. Hence, the mathematical and computer-based DSS for evaluating the performance of the TDSB was created using software residing on essentially all personal computers.

### Abbreviations

DSS, Decision support system; DEA, Data envelopment analysis; TDSB, Toronto district school board; DMU, Decision making unit; VBA, Visual basic for applications.

### Competing interests

The author declares that there are no competing interests.

### Author’s contributions

JSY is the sole author of this paper, and read and approved the final manuscript.

### Acknowledgments

This work was supported in part by grant OGP0155871 from the Natural Sciences and Engineering Research Council. The author would like to thank Richard Christie of the Toronto District School Board for providing access to the energy and waste data of the various schools.

### References

• Albright SC (2010) VBA for Modelers: Developing Decision Support Systems. Cengage Learning, Mason, OH.

• Camp RC (1995) Business Process Benchmarking, Finding and Implementing Best Practices. ASQC Quality Press, Milwaukee, WI.

• Christie R (2003) The Case for an Annual Environment Report at the Toronto District. Department of Environmental Education, School Services, Toronto District School Board. , .

• Christie R (2007) Environmental Report. Department of Ecological Literacy and Sustainable Development, Toronto District School Board.

• Christie R, Coppinger F (2006) Toronto District School Board Energy Conservation Report. Department of Ecological Literacy and Sustainable Development, Toronto District School Board.

• Cook W, Zhu J (2008) Data Envelopment Analysis: Modeling Operational Processes and Measuring Productivity. CreateSpace, Charleston, SC.

• Cook W, Kress M, Seiford L (1996) Data Envelopment Analysis in the Presence of both Quantitative and Qualitative Factors. J Oper Res Soc 47:945-953

• Lin QG, Huang GH, Bass B, Chen B, Zhang BY, Zhang XD (2008) CCEM: A City-cluster Energy Systems Planning Model. Energy Sources, Part A: Recovery, Utilization, and Environmental Effects 31:1-14 Publisher Full Text

• Linton J, Morabito J, Yeomans JS (2007) An Extension to a DEA Support System Used for Assessing R&D Projects. R&D Management 37:29-36 PubMed Abstract

• Seref MMH, Ahuja RA, Winston WL (2007) Developing Decision Spreadsheet-Based Decision Support Systems. Dynamic Ideas, Boston.

• Simonson I, Tversky A (1992) Choice in Context: Tradeoff Contrast and Extremeness Aversion. J Mark Res 29:281-295 Publisher Full Text

• Tversky A, Simonson I (1993) Context-Dependent Preferences. Manag Sci 39:1179-1189 Publisher Full Text

• Yeomans JS (2004) Rating and Evaluating the Combined Financial and Environmental Performance of Companies in the Metals and Mining Sector. Journal of Environmental Informatics 2004(3):95-105

• Zhu J (2003) Quantitative Models for Performance Evaluation and Benchmarking. Kluwer Academic Publishers, Norwell, MA.