Hagan School of Business

Analyzing the Benchmark Data

Once the processes to be benchmarked have been identified, the benchmarking partners are chosen, and the data collected, the next step is to determine the current competitive gap. There are three types of performance gaps: negative, parity, and positive, and are listed in Table 1 (Camp, 1989). When there is a negative gap, this means that the benchmarking partner(s) have superior operational performance numbers in their processes. Examples of such numbers might include the unit cost of service being provided, the level of customer satisfaction, the financial ratio, and other such metrics we have seen in higher education benchmarking projects. If the gap is negative, a significant effort will now be required to change the internal practices and process methods at the home institution to meet or exceed the external findings. The goal is to explain why the differences exist and determine what the specific contributing factors or enablers are. When the comparative analysis finds that the operations are at parity or have little difference, further analysis should be done to find the reason. If the operations of both organizations are indeed using the most efficient method, then no changes are necessary. However, parity is no reason to not continue to look elsewhere for best practices. A positive performance gap means that the internal practices are superior to the other institutions being benchmarked. This is not to be unexpected, especially in benchmarking studies that are broad-based, and part of a consortium or association-sponsored project such as NACUBO. Experiencing a positive gap can help a company maintain best practices and justify the continued search for ways to close the other negative gaps which are found.

Table 1 - Types of Performance Gaps (Camp, 1989)
 

Type	Description	Consequence
Negative	External practices are superior	Benchmark based on external findings
Parity	No significant practice difference	Further analysis justified
Positive	Internal practices are superior	Benchmark based on internal findings

In the NACUBO Benchmarking project, participants can receive a detailed gap analysis, which compares their own institution’s performance with the mean of all study participants and cohort groups (Table 2). The hypothetical analysis of an admissions office shows that some processes such as departmental cost per inquiry, applicant, and matriculant are at parity (for the private research cohort), and have unfavorable gaps for other cohorts. Descriptive comments are also offered which analyze the performance. Although benchmarking data such as this is useful, the overall goal of learning best practices will need to be completed with a site visit to the high performing institution(s).

Table 2 - Gap analysis for cost benchmarks - Hypothetical University’s Admission Office (NACUBO, 1995) (used with permission).
 

Results for	Departmental cost per inquiry	Departmental cost per applicant	Departmental cost per matriculant	Cost of processing a student application
Hypothetical Univ.	$56.20	$341.33	$540.05	$80.25
All Survey Participants Mean: Semi-interquartile Range: Assessment:	$32.34 $12.84 to $26.27 Significantly higher: unfavorable gap	$170.44 $75.98 to $195.38 Significantly higher; unfavorable gap	$494.63 $193.21 to $533.30 Significantly higher; unfavorable gap	$24.42 $9.43 to $32.54 Significantly higher; unfavorable gap
Public Research Cohort Mean: Semi-interquartile Range: Assessment:	$23.80 $12.84 to $26.27 Significantly higher: unfavorable gap	$109.30 $81.38 to $115.04 Significantly higher; unfavorable gap	$324.08 $221.39 to $361.28 Significantly higher: unfavorable gap	$27.51 $9.43 to $32.54 Significantly higher: unfavorable gap
Private Research Cohort Mean: Semi-interquartile Range: Assessment:	$32.23 $23.13 to $60.24 No significant difference: neutral gap	$263.27 $187.99 to $354.65 No significant difference; neutral gap	$1,231.32 $459.18 to $1,209 No significant difference; neutral gap	$36.30 $27.25 to $39.26 Significantly higher; unfavorable gap
Comments on Hypothetical University’s Performance	More costly than other public research universities, but in line with the overall set of institutions and the private research cohort	Higher cost than other public research institutions and overall, but once again, Hypothetical approximates the average value for the private research cohort.	More costly than both public research and overall, but on par with private research cohort.	Among the highest, least favorable values among all survey participants; significantly higher than all cohorts.

The traditional view of college and university operational costs has been functional and organizational, and funding tends to be budgeted on the inputs of each department (Massey & Myerson, 1994). The assumption in this strategy is that the more money that is put into a department, division or institution, the better the quality that will result. Therefore, financial benchmarking is typically seen as input driven, and few measures of service output are usually available. Most process benchmarking efforts do begin to address the need for outputs, which can help colleges and universities reshape their cost structure. Many of these are the more "grass roots" efforts that are conducted by individual units within institutions. The outputs can be measured using the different types of benchmarking we have discussed. Massey and Meyerson (1994) offer an example of output benchmarking, which examines the gift processing performance (Table 3). On average it takes 42 days to acknowledge a donor’s gift to the university at cost of $19 per transaction. If the university processes 48,000 transactions per year, and could meet the Best-in-Class gap of $10, then it could improve the responsiveness by 35 days and reduce the annual aggregate transaction costs by $480,000. This could also yield additional future donations because the "customer" donors would be much better served.

Table 3 - Gift Processing - Performance Assessment (Massey & Myerson, 1994)
 

Gift Acknowledgment Process	Current perfor- mance	Customer perception	Industry bench-marking	Best-in-class bench-marking	Customer gap	Industry gap	Best-in-class gap
Response	42 days	14 days	17 days	7 days	28 days	25 days	35 days
Cost per Acknowledgment	$19	N/A	$11	$9	N/A	$8	$10

By analyzing the benchmark data across the different benchmarking types such as internal, competitive, industry, and generic/best-in-class, it is easy to see where the home organization truly stands in its performance against others. Benchmarking enables the practitioner to go beyond a "gut feel" that the process can be improved. It provides the data or "pegs" of where the performance level can and should advance, both in an industry and externally. Data can be even more compelling if it is compared and analyzed using the different benchmarking types, and/or if the data is analyzed graphically. Camp (1989) and Dale (1995), offer suggestion on how benchmarking results can be graphically analyzed to reveal the performance gaps. If the data being benchmarked is a process, then the vertical Y-axis value units will often show an increase in better quality, speed, or efficiency. If the data is for the cost of a process, then the line chart will usually attempt to show a decrease in costs relative to the benchmarking partners.

The performance gaps between the home institution and the competitors are easy to see on such a chart.

When analyzing the data, one must be aware of the context in which the data was gathered. This is one reason that many benchmarking projects rely on cohort or peer groups of institutions that may be influenced more or less equally by the environmental or external factors. Gregory Watson (1993) recommends asking a series of questions to clarify the results, including the following:

• What is the extent of time over which the process was measured?
• Was there any change in the measurement system, such as the measurement tools used?
• How much of the measurement system relies on human observations or correlation, versus true data analysis?
• How are these correlations substantiated?
• How often or at what intervals were the measures taken?
• What is the estimated margin for error?
• Are the measures verifiable, auditable, and repeatable? (Watson, 1993, p. 49)

The top four causes of benchmarking project failure were poor planning, no-top management support, no process-owner involved, and insufficient benchmarking skills. All four of these problems can be addressed by organizations that plan to conduct a project, and have been discussed earlier in the review of the benchmarking literature. Proper project planning, support from the president’s office and the "grass-roots" units where the efforts should take place, and proper employee training are important for all organizational improvements.

Camp, R. C. (1989). Benchmarking: The Search for Industry Best Practices That Lead to Superior Performance. Milwaukee: ASQC Quality Press.

Dale, B. (1995, October 30 & 31, 1995). Practical Benchmarking for Colleges and Universities. Paper presented at the AAHE Workshop, Key Biscayne, Florida.

Massey, W. F., & Myerson, J. (Eds.). (1994). Measuring Institutional Performance in Higher Education. Princeton, New Jersey: Peterson's Guides.

NACUBO. (1995). Benchmarking Prospectus . Washington, DC: National Association of College and University Business Officers.

Watson, G. H. (1993). Strategic Benchmarking: How To Rate Your Company's Performance Against the World's Best. New York: John Wiley and Sons.