The Toronto District School Board (TDSB) oversees the largest school district in Canada and has been spent more than one third of its annual maintenance budget on energy and waste. This has directed attention toward system-wide reductions to both energy consumption patterns and waste generation rates. In this paper, a decision support system (DSS) that can process unit-incompatible measures is used for rating, ranking, and benchmarking the schools within the TDSB.
The DSS permits the ranking of any set of schools by contextually evaluating their relative attractiveness to other identified school groupings. Consequently, the DSS was used to explicitly rank each school’s performance within the district and to determine realistic energy improvement targets. Achieving these benchmarks would reduce system-wide energy costs by twenty-five percent.
The TDSB study demonstrates that this DSS provides an extremely useful approach for evaluating, benchmarking and ranking the relative energy and waste performance within the school system, and the potential to extend its much broader applicability into other applications clearly warrants additional exploration.
Keywords:Benchmarking Performance; Data Envelopment Analysis; Decision Support Systems; Energy & Waste
Representing more than 600 schools, the Toronto District School Board (TDSB) oversees the largest education constituency within Canada. In the 2003 fiscal year, the TDSB allocated $48 million of its annual budget to the energy and waste requirements of the school system, but within three years, found that the energy and waste expenditures had escalated to more than $69 million (Christie [2003,2007]; Christie & Coppinger ). This rapid forty percent increase in spending during a period of public economic retrenchment necessitated stringent attention toward system-wide reductions to both energy consumption patterns and waste generation rates (Christie [2003,2007]; Christie & Coppinger ).
While the need to decrease energy consumption and minimize waste was universally acknowledged, to achieve long-term success in these efforts, the TDSB resolutely believed that any successful systematic reduction efforts would need to adequately address three critical questions: (i) what initiatives would achieve the most effective results; (ii) how could these “most effective reduction initiatives” actually be identified; and, (iii) how effective would these initiatives prove to be once implemented (Christie ) Additional “essential components” to effectively implement any proposed reduction schemes necessitated that improvement initiatives had to include: (i) the establishment of realistic benchmarks for each school to strive toward; (ii) the setting of achievable annual performance targets for each school; (iii) a commitment to rational and objective management of the system data; and, (iv) an assurance of transparency and neutrality in policy-setting and decision-making (Christie [2003,2007]; Christie & Coppinger ). Unfortunately, it proves to be an extremely difficult process to evaluate system-wide performance and to establish benchmarks when there are multiple incommensurate criteria measurements present (Camp ), which was the case within the budgetary-constrained and politically-charged environment of the TDSB (Christie [2003,2007]; Christie & Coppinger ).
Decision support systems (DSS) are intelligent information systems based on decision models that can be used to extract large quantities of data from databases, to provide interfaces and methods for effectively processing it, and for deriving meaningful decisions of managerial/economic significance from it. DSS have been used to analyze a wide variety of performance information and to provide a readily-accessible medium for distributing any knowledge generated to a wide variety of stakeholders (Lin et al. ). In this paper, a DSS that can simultaneously combine unit-incompatible energy and waste performance measures is introduced for evaluating the system-wide energy and waste performance of the schools within the TDSB. This DSS incorporates several data envelopment analysis (DEA) modules that had been developed in Yeomans () and threads these modules together using commonly available software (Albright ; Seref et al. ; Zhu ). The underlying DEA methodology has been shown to hold advantages over many other multi-criteria methods by providing an objective decision-making tool that does not require variables to have the same scale or conversion weights applied to them (Cook & Zhu ; Zhu ), while permitting a simultaneous combination of both quantitative and qualitative measures (Cook et al. ).
The DSS developed can be used to rate and rank each school according to its energy and waste performance relative to the other schools in the system by recursively partitioning the schools into sub-groups of relatively superior and inferior performers. In addition, the DSS can establish realistically achievable improvement targets for each school by benchmarking their performance against the operations of schools in higher efficiency categories. This relative performance comparison between the schools is important since it ensures that the underlying analysis involves the use of energy and waste values actually occurring at peer institutions within the TDSB and not through “externally-generated”, potentially unrepresentative values.
The DSS developed directly addresses the TDSB’s three requisite “critical questions” while simultaneously addressing most of the key issues in the “essential components” identified above. Furthermore, the most practical contribution from this approach is that, since DEA can be readily implemented using common spreadsheet software linked together by relatively straightforward VBA programming (Albright ; Seref et al. ; Zhu ), this entire methodology can be implemented on virtually any computer. Consequently, practitioners in any organization could easily modify and extend this methodology to match their own very specific multi-criteria applications and circumstances. An illustrative example of an application of the DSS is provided through an analysis of a subset of the high schools in the TDSB.
Results and Discussion
The primary purpose in the performance evaluations of most organizational systems is to appraise the current operations of individual entities and to benchmark these against peer entities to identify best practices. While individual performance measure, or “gap”, analysis has often provided the fundamental basis for performance evaluation and benchmarking (Zhu ), it remains a difficult task to satisfactorily combine multiple disparate, unit-incompatible, single-criteria measurements into an overall conclusion (Camp ). Since a complex entity’s actual performance generally represents multifaceted phenomena, the use of single measures in gap analysis explicitly ignores all interactions, substitutions, and tradeoffs between the various performance measures. Therefore, it is a rare occurrence when a one-measure-at-a-time gap-analysis can suffice for the purposes of an effective performance evaluation of organizational systems (Camp ; Zhu ).
Clearly it is difficult to evaluate an organization’s performance or to establish benchmarks when there are multiple measurements present (Camp ). If the specific algebraic functional relationship between performance measures is known, then established multi-criteria techniques can be used to estimate best-practice levels of performance. These algebraic functional forms cannot be specified without a priori information on the corresponding tradeoffs and, unfortunately, such information is generally unavailable in most practical situations. However, when the best practices of similar types of operations can be identified empirically at a specific point in time, it becomes possible to empirically estimate the resulting best-practice level of performance, or efficient frontier, using these observations (Zhu ).
DEA has proved to be an effective empirical tool for identifying multi-criteria efficient frontiers, for subsequently evaluating relative performance efficiencies, and for implicitly estimating the tradeoffs inherent within the empirically designed frontier. DEA’s empirical orientation and absence of a priori assumptions establish it as the ideal analytical instrument for application to a wide variety of practical situations, since its underlying theoretical basis is consistent with the practice of rating entities by concurrently examining the relative efficiencies of their multiple performance measures. Furthermore, DEA allows performance comparisons to be made between numerous entities that have been evaluated by multiple, unit-incomparable measures without employing any a priori weightings or conversion factors typically required in other methods. Since the process is adaptable and invariant to data type, DEA permits the inclusion and comparison of non-numeric environmental-type variables that might prove incomparable using many other techniques (Zhu ). By incorporating DEA into its DSS, the focus of performance evaluation for the TDSB application could shift from a characterization of each school’s energy and waste usage in terms of single measures to evaluating performance from a mathematically rational, multidimensional system perspective (Linton et al. ; Yeomans ). Consequently, inherent energy and waste relationships and their role in performance ranking can be explicitly brought into the analysis in a rational, transparent, and neutral fashion.
An overview of the energy and waste DSS for the TDSB
The DSS, itself, contains a series of specific DEA modules (more fully explained in subsequent sections) for conducting the performance evaluation of the TDSB. The first module recursively partitions the selected schools into sub-groups of relatively superior and inferior performers according to their energy and waste performance. The second module then determines realistically achievable energy and waste improvement targets for each school by benchmarking them against the system-wide operations for schools in selected higher performance categories. If desired, the DSS can also execute a third module to establish an explicit rank ordering of each school relative to any desired set of schools within the TDSB.
The set of DEA modules was created using readily available spreadsheet optimization software linked together by a combination of straightforward programming. In order to make the entire analysis process readily accessible to the various system users, the DSS was implemented using Microsoft Access and Microsoft Excel together with VBA (contained in all Microsoft Office products) as the programming language. By using these specific computer packages, the entire set of modules for evaluating the performance of the TDSB was created using software residing on most current personal computers. In the following sections, each of the individual DEA modules used in the DSS is explained in detail and illustrated using data from 65 high schools in the TDSB.
DEA relative performance rating module
The first module in the DSS partitions any selected group of schools into relatively superior and inferior performers based upon their energy and waste performance. In DEA terminology a decision-making unit (DMU) designates the specific entity being studied (Zhu ). For the example, the set of DMUs consists of the 65 high schools in the TDSB. Each DMU possesses a set of inputs and outputs that represent its multiple measures of performance and, for the high schools, these inputs and outputs consist of various observed energy and waste measures. While a considerable number of different combinations of energy and waste performance indicators could have been selected from the available data, for illustrative purposes the example in this paper contains only two inputs and four outputs. The inputs considered were (i) school enrolments measured in terms of the number of students and (ii) school sizes measured in square metres (sqm), while the set of outputs consisted of (i) total energy consumption measured in gigajoules (GJ), (ii) total energy costs in dollars ($), (iii) total waste in kilograms (Kg), and (iv) waste diversion percentage (%).
One analytical requirement for DEA is that all inputs have to be measured in units where “less is better”, while all outputs have to be expressed in units in which “more is better”. Hence, prior to implementing any of the procedures, the DSS transforms all selected inputs and outputs into a format consistent with this analytical requirements. Table 1 provides a complete list of the transformed inputs and outputs for the 65 schools.
Table 1. Each high school’s energy and waste input/output measurements transformed into appropriate format for DEA usage
The analytical approach of DEA evaluates the data by “enveloping” the entities being studied based upon the values of the performance measures. The underlying concept of DEA requires an evaluation of each DMU through a projection onto an empirically constructed, multi-dimensional efficient frontier. The enveloping process determines the efficiency of DMUs by: (i) creating an m + s dimensional surface, or “efficient frontier”, of the efficient DMUs (where m represents the number of inputs and s represents the number of outputs); (ii) assigning an efficiency score of θ = 1 to any DMU on the efficient frontier; (iii) determining the distance from the frontier for all inefficient DMUs; and, (iv) calculating the value of θ for inefficient DMUs as its proportional, multi-dimensional distance from the efficient frontier. The efficiency score of any inefficient DMU is always a value of θ < 1. The enveloping constructs an efficient frontier of the best-practice entities and also shows how any inefficient DMU can be improved by providing the amounts and directions for improvement to its specific measures (Zhu ).
In assessing the high schools, the overall goal is to identify the system’s best performers by contrasting each school’s observed performance metrics relative to those of every other school considered. However, DEA only determines whether a school is efficient (θ = 1) or inefficient (θ < 1). The magnitude of θ cannot be used to establish relative degrees of inefficiency between non-efficient schools. Because the relative performance of any school can be contrasted only to an identified best-practice frontier, actual measures of relative inefficiency would change only when the best-practice frontier is altered (that is, when one or more of the efficient schools is removed). However, a recursive enveloping module can be created that stratifies the schools into numerous levels, or groupings, of relative best-practice frontiers, rather than the single DEA frontier.
The stratifying module of the DSS proceeds by removing all of those schools placed onto the original best-practice frontier and then using the original DEA enveloping methodology to form a new second-level best-practice frontier from the set of remaining schools. That is, once a first efficient frontier is calculated, all of the associated efficient schools are removed from further consideration and a new efficient frontier based only upon the remaining, initially inefficient schools is calculated. The schools on this second-level efficiency frontier are subsequently removed, permitting a third-level frontier to be constructed, then a fourth-level frontier, and so on, until no schools remain. The final result from this stratification is the creation of a series of efficient frontiers. The recursive procedure stratifies the original set of schools into L groupings of school efficiencies, with the specific value for L algorithmically determined, a posteriori, by an “empty-set” stopping rule.
When the stratification module was applied to the high school data, the 65 schools were partitioned into L = 12 distinct groups with 2, 2, 3, 6, 9, 10, 10, 5, 7, 5, 3, and 3 schools assigned to each respective efficiency stratum (see Table 2). This stratification effectively partitions the high schools into distinct groupings of comparably-performing schools based upon the multi-criteria measurements of their energy and waste performance.
Table 2. Changes required to each high school’s current energy and waste measures to advance into the next higher efficiency group
Benchmark module for generating realistic energy and waste performance targets
Benchmarking is widely used for the identification and adoption of best practices and as a means for improving performance and increasing productivity. Benchmarking can be thought of as the process of defining valid measures of performance comparison among peer schools, using these to determine the relative standing of the peer schools, and ultimately in establishing standards of excellence for performance improvement. The TDSB had stated in their “essential component” requirements that, in order to improve their system-wide energy and waste usage, it was crucial for them to be able to determine attainable performance benchmarks and to establish achievable annual performance targets for each school (Christie ). Clearly the satisfaction of these components would necessitate the creation of multi-criteria benchmark targets to improve each school’s performance and it would be imperative that the various competing stakeholders within the system felt that these targets had been set fairly, objectively, and transparently (Christie [2003,2007]). As observed in the previous section, the stratification module partitioned the 65 high schools into 12 sets of comparably performing groups in which all schools within a higher grouping were better performers than any school in each of the lower groupings. In this sense, the stratification module could be viewed as a type of benchmarking, since the module creates various different performance groupings of peer-efficient schools to which any underperforming schools could be benchmarked.
However, by using the stratification module’s output, it becomes possible to establish performance goals for less efficient schools by benchmarking their current energy and waste usage against the more efficient operations of the schools in any of the higher performing strata. For an inefficient DMU, it is possible to calculate the changes needed to its inputs/outputs for it to become efficient relative to the DMUs on the efficient frontier. These efficiency targets would be the specific values that the inputs and outputs of the inefficient DMU would need to attain in order to move onto the efficiency frontier. For the benchmarking and target setting required in the TDSB case, the set of DMUs in the following procedure would consist of one specific, inefficient DMU under evaluation (i.e. DMU0) and all of the DMUs in the next higher contextual grouping. Hence, assume that there exists a set of n DMUs, with each DMUj, j = 1,…, n, consisting of m input measures xij, i = 1,…, m, and s output measures yrj, r = 1,…, s. Suppose that DMU0 is being evaluated with xi0 and yr0 representing its ith input and rth output measures. Let , i = 1,…, m, and , r = 1,…, s, represent the ith input and rth output slack variables, respectively, and let ε be some non-Archimedean scalar. Then by solving the optimization model:
efficiency targets , i = 1,…, m, and , r = 1,…, s, can be calculated for each of the inputs and outputs of DMU0. The presence of the non-Archimedean ε in the objective function effectively allows the minimization over θ to pre-empt the optimization involving the slacks, and . This creates a two-stage optimization process with the maximal reduction of the inputs being achieved in the first stage via the optimal θ*, followed by the movement onto the efficient frontier achieved in the second stage via the subsequent optimization of the slack variables. This approach to multi-criteria benchmarking proves particularly suitable to practical situations in which no objective or pre-existing engineered standards are available to define efficient or effective performance.
For illustrative purposes, an incremental goal for each school to progress only into the next highest level of efficiency was set. While a longer-term perspective might seek to advance all schools into the very highest level of performers, these intermediate targets establish more attainable improvements that each inefficient school would need to undertake in order to move into the next higher level of efficiency. Hence, the benchmarking module was used to calculate the specific efficiency targets required for each school to proceed into the next higher category of energy and waste performance and Table 2 shows the specific improvements required in each measure in order to reach the calculated targets. The specific changes shown in Table 2 represent: (i) annual reductions in energy use; (ii) annual reductions in energy expenses; (iii) annual reductions in the quantity of solid wastes generated; and, (iv) annual increases in the percentage diversion of solid wastes. While the target-setting module actually produces multi-criteria measures for improvement, if the changes in Table 2 were to be achieved, then one significant subset of these improvements would be the 25 % percent reduction in annual energy costs. This reduction, alone, represents a direct annual cost savings of $3.7 million from the current energy budget of $14.9 million allocated to the TDSB’s high school. Furthermore, if the established targets were achieved, the table shows potential reductions of over 300,000 GJ of energy and of 44,000 Kg of waste. It should be noted that an analogous percentage cost, energy and waste improvements would be obtained from the targets set for the entire set of the more than 600 schools of the TDSB.
Rank ordering module
While the stratification module partitions the schools into distinct levels of energy and waste performers, it does not rank order the standings of any of the schools within each grouping. If only a small number of schools were under consideration in an analysis or if a large number of groupings each containing only a very small number of schools had been produced, then the stratification, itself, might be sufficient for actually ranking the specific schools. For the general case, however, the stratification might not prove restrictive enough to permit sufficient degrees of preference discrimination. If this situation proves to be the case, then a contextual attractiveness concept can be incorporated into the DSS to permit an explicit ranking of the schools (Simonson & Tversky ; Tversky & Simonson ). Obtaining relative attractiveness scores for the schools requires that relative performance be defined with respect to some particular evaluation context and the L partitions from the stratification module can be used to supply these contexts.
DMUq is viewed as a more attractive option than another DMU if it possesses a larger value for its contextual attractiveness measure . Hence, it is possible to rank each school within each stratum using a direct sorting of these contextual attractiveness scores. Since all schools within a contextual grouping are considered better performers than any school in a lower contextual group, when the rankings within each grouping are subsequently concatenated, a complete, rank ordering of the entire set of selected schools will be produced, de facto.
Using the next-lower stratification grouping as the appropriate context, the attractiveness scores , for each high school were calculated. Since any school is considered more efficient than the other schools within its grouping if it possesses a larger contextual attractiveness score, it now becomes possible to rank order the schools within each stratum by sorting these scores. Concatenating these individually sorted groupings produces a comprehensive rank ordering from 1 to 65 based upon the energy and waste usage in all of the schools. The results of this contextual scoring system, the subsequent sorting within each partition, and the overall efficiency ranking of each high school within the TDSB are shown in Table 3.
Table 3. Overall rankings of high schools based upon contextual attractiveness scores calculated relative to the next lower partitioning group
If a large number of schools existed within any particular level after the stratification stage, then a situation might occur in which two or more of the schools each received exactly the same attractiveness measure. This tied ranking problem can be alleviated by incorporating a lexicographic ranking modification into the procedure described above and could be accomplished in the following manner. In order to reduce the likelihood of tied rankings, the lexicographic procedure would be used to calculate attractiveness measures for each school by using every level lower than it as the contextual basis. Hence, each school in the highest level would receive L-1 separate contextual attractiveness scores, each school in the second highest level would receive L-2 separate attractiveness scores, and so on. A lexicographical rank ordering of schools within each specific grouping would then be performed with greater emphasis given to attractiveness scores calculated from closer contextual groupings (i.e. through an alphabetical or dictionary style of sorting). This multi-scoring, lexicographic rank ordering process would produce additional discriminating powers by reducing the likelihood of ties occurring between schools within any specific grouping. However, since no tied-scores occurred in the contextual attractiveness calculations already performed, this lexicographic ranking produces the same rank ordering as Table 3.
In this paper, a DEA-based DSS for analyzing, rating, ranking, and benchmarking the multi-criteria energy and waste system of the schools in the TDSB has been studied. Several benefits of this DSS were demonstrated through an illustrative investigation of 65 Toronto high schools. The DSS stratified the schools into similarly-efficient groupings based upon their energy and waste usage. The DSS then generated realistic energy and waste improvement targets for any relatively inefficient school against another grouping of benchmarked schools. Amongst other findings, it was shown that achieving these target reductions would produce system-wide energy cost savings of twenty-five percent. The DSS also permits the ranking of any set of schools by contextually evaluating their relative attractiveness to other identified school groupings. The findings with respect to the high schools have been extended in an analysis of all 600 schools within the TDSB. Based upon the TDSB study, DEA has shown itself to be an extremely useful approach for evaluating, benchmarking and ranking the relative energy and waste performance within the school system, and the potential for its much broader applicability to other applications clearly warrants additional exploration.
The mathematical models of each of the DEA modules were created as separate computer models within Microsoft Excel worksheets (Albright ; Seref et al. ; Zhu ). The optimization of the spreadsheets of these DEA models was performed using the standard, built-in Excel Solver function. The data for all 600 schools in the TDSB can be found in Christie () and the data for the 65 schools used in the analysis appears in Table 1. All data was stored in a Microsoft Access database. The programming language used to link together the various components was VBA which is contained in all Microsoft Office products. Hence, the mathematical and computer-based DSS for evaluating the performance of the TDSB was created using software residing on essentially all personal computers.
DSS, Decision support system; DEA, Data envelopment analysis; TDSB, Toronto district school board; DMU, Decision making unit; VBA, Visual basic for applications.
The author declares that there are no competing interests.
JSY is the sole author of this paper, and read and approved the final manuscript.
This work was supported in part by grant OGP0155871 from the Natural Sciences and Engineering Research Council. The author would like to thank Richard Christie of the Toronto District School Board for providing access to the energy and waste data of the various schools.
Lin QG, Huang GH, Bass B, Chen B, Zhang BY, Zhang XD (2008) CCEM: A City-cluster Energy Systems Planning Model. Energy Sources, Part A: Recovery, Utilization, and Environmental Effects 31:1-14 Publisher Full Text
Linton J, Morabito J, Yeomans JS (2007) An Extension to a DEA Support System Used for Assessing R&D Projects. R&D Management 37:29-36 PubMed Abstract
Simonson I, Tversky A (1992) Choice in Context: Tradeoff Contrast and Extremeness Aversion. J Mark Res 29:281-295 Publisher Full Text
Tversky A, Simonson I (1993) Context-Dependent Preferences. Manag Sci 39:1179-1189 Publisher Full Text