Overall, supervisors in North Carolina state government tend to give their employees high ratings. Every year, about one-third of state employees receive an "Outstanding" and over half receive a "Very Good" rating. If you discovered that a company you did business with, or that you owned shares of stock in, considered 85% of its employees as exceptional performers, what would you think?
People question the accuracy of the ratings when so many employees are rated so highly.
Why Is Rating Inflation a Problem?
Why does rating accuracy matter? There are a number of reasons why accuracy is important:
- Supervisors should "level" with employees regarding their performance. The feedback given to employees, including their annual appraisal rating, should be candid. ("Candid," incidentally, does not mean brutal. Supervisors are encouraged to emphasize the positive when providing feedback to employees, but not to sweep performance shortfalls under the rug!)
- Ratings must be accurate if an organization wishes to reward those employees who make truly significant contributions. When many employees receive the same high rating, those who are truly outstanding become disenchanted and cynical. They may leave the organization, or slacken their work effort, neither of which is a good thing.
- Prospective employees who are highly motivated, achievement oriented, and keen to make a contribution and be recognized accordingly, are not attracted to organizations where top performers are treated the same as mediocre or even poor performers.
- Performance ratings are used to validate the accuracy of hiring decisions. If the ratings are not accurate, the hiring process will never be able to distinguish applicants who are most likely to succeed from those who will become so-so performers.
- The Legislature will always be reluctant to fund the Comprehensive Compensation System if 85% of employees are rated as having exceeded expectations ("Very Good" or "Outstanding"). The numbers are just not believable.
How Can You Tell If Ratings Are Accurate?
If rating accuracy is the key, how do you know if ratings are really accurate? There is no definitive answer to this question, but good assumptions can be made based on:
- Looking at your distribution — A distribution that is too skewed toward positive ratings is not realistic. However, an agency would not be expected to have a perfectly "normal" distribution (one shaped like a bell curve). If your agency is doing a decent job of hiring, training and managing performance, you would naturally expect the distribution to be positively skewed, but not in an exaggerated way.
- Comparing your distribution to other organizations' distributions — Within North Carolina state government, the shape of performance rating distributions varies widely. How does your agency's distribution compare to others'?
- Recognizing that it is usually a small percentage of employees who most critically contribute to the organization's success — Research on a variety of occupations, from jet fighter pilots to computer programmers, has found that a handful of employees typically make things happen, carry the load, get the work done and make the lion's share of contributions. The majority of employees are role players whose contributions are helpful, but not dominating in the same way as the "stars."
- Comparing appraisal ratings to objective performance criteria — Whenever objective individual measures are available for a large number of employees, compare these objective measures against their appraisal ratings. Better yet, why add a layer of administrative activity on top of the measures? Make the objective individual measures the ratings. See the section on Reinventing Performance Management for suggestions on how to do this.
Why Do Ratings Get Inflated?
Ratings are intended to accurately reflect performance. So how do they become inflated?
- The criteria on which employees get rated can be fairly subjective. For example, criteria such as "Ensures all visitors sign in and are routed to the appropriate office" and "Applies analytical skills to the solution of work-related problems" allow considerable room for interpretation. The vagueness of such criteria leads supervisors to assign lenient ratings.
- Even objective ratings based on employee output can become inflated if easy goals are set or the system is gamed.
- Supervisors may wish to avoid ruffling feathers and upsetting people, which they reason would be the effect if they were to give lower, more realistic ratings.
- Performance appraisal is a political process and supervisors often have motives other than accuracy in assigning ratings.
- When there is no monetary reward directly tied to the appraisal rating, giving high ratings is a way of delivering a psychological reward to employees.
How Rating Inflation Can Be Tamed
Are there practical ways to "tame" rating inflation? Yes. We offer, below, a recommended approach and a number of additional or supplementary techniques. The recommended approach consists of three essential ingredients:
- Set Ratings Guidelines — Define what the ratings distribution should look like. Set a limit on how many employees can receive the highest ratings.
- Establish Well-Defined Work Plans — At the front end, make sure supervisors and managers do a good job of planning performance with their employees. Set clear performance standards or goals.
- Require Justification for Out-of-Range Distributions — At the back end, hold managers accountable for accurate ratings — their own, and the ratings given by the supervisors who report to them. If they give out too many high ratings, ask them to justify those ratings with data.
More detail about each of these steps.
Step 1 — Set Ratings Guidelines
There must be guidelines that define what the desired distribution of ratings looks like. Without such guidelines, there would be no way to know whether an agency's distribution of ratings was inflated or not. The state's performance management policy provides such guidelines: No more than 60% of employees shall be rated better than "Good."
Unfortunately, there's more to Step 1 than merely setting the guidelines. The guidelines must be communicated to everyone at the beginning of the performance cycle, have the active support of management, and apply across the board to everyone.
Step 2 — Establish Well-Defined Work Plans
Supervisors and managers should establish work plans for the people who report to them that clearly define their performance expectations. Expectations set at the front end that are measurable and challenging will be more likely to result in accurate ratings at the back end of the process.
Expectations that are more loosely set are more prone to inflation because we tend to be more lenient in the absence of clear standards. Having clear and challenging standards to begin with forces us to take a more disciplined approach when evaluating the results, and makes it easier to assign a lower rating if that is the way the results turned out.
For assistance in establishing work plans, see the Performance Planning, Streamlined Performance Management, or Results-Focused Performance Management sections.
Step 3 — Require Justification for Out-of-Range Distributions
Any supervisor or manager with more than ten direct and indirect reports, whose ratings distribution exceeds the level established in the guidelines, must present the justification for the inflated ratings to a three-member panel consisting of members of the next two levels of management.
Making this presentation puts supervisors' or managers' performance management practices under the spotlight, the sort of exposure most would rather avoid. It would be easier for them to make sure performance management is carried out in a disciplined way in the first place, from the performance planning stage to the assignment of ratings. On the other hand, if a group receives high ratings and these ratings are a realistic reflection of employee performance, then the supervisor or manager would presumably have the data to support the ratings.
Note that following the distribution guidelines does not apply unless there are at least ten employees in the supervisor's or manager's line of direct and indirect reports. This stipulation deals with the problem of imposing "quotas" on supervisors with small numbers of direct reports.
For example, if a supervisor has two direct reports, it is unrealistic to expect their ratings to satisfy the distribution guidelines. The supervisor would be restricted to rating only one of the employees above "Good" in any given year.
Instead of imposing this artificial restriction on the supervisor, you would look at the supervisor's manager. Suppose this manager has three direct reports and a total of nine indirect reports. With a total of 12 employees in his or her line of reports, the guidelines would apply. It would then be this manager's responsibility to make sure that no more than seven (60% of 12) of these individuals receive ratings above "Good," unless the ratings were soundly justified.
If the distribution exceeds the guidelines, the manager would investigate to see if the ratings are indeed accurate. If the manager finds the ratings to be inaccurate, or inflated, the manager would see that adjustments were made in the ratings to make them accurate. If the manager determines the ratings are accurate as submitted, then he or she would collect the data necessary to justify exceeding the guidelines and prepare to meet with a panel of higher-level managers.
If it is necessary to hold a session to justify a ratings distribution, it would be helpful to have a structure or an agenda to follow. The Agenda for Ratings Review Session may be used for this purpose. HR staff may use the agenda to plan and oversee the sessions, managers who are justifying their ratings may use it to prepare for the sessions, and managers serving on the review panel may use it to facilitate the session.
This approach to taming rating inflation actually gets applied slightly differently, depending on the type of performance management system your agency uses:
- Responsibilities and Behaviors — Most agencies' performance management processes use responsibilities and behaviors as the basis for setting expectations and doing appraisals. This kind of system is ultimately pretty subjective, and thus prone to rating inflation. The method described above would address inflation as it occurs in responsibility—and-behavior-based systems. In these systems, it is easier to change a proposed rating because the rating is not based on comparing verifiable data against a standard but on comparing observations against loosely defined, descriptive criteria.
- Results — Results- and metrics-based performance management systems are a different matter. When all 12 of your employees exceed their goals, it does not make sense that you should have to go back and tell five of them that their results did not really exceed the goal. Their results are their results. Supervisors should have no problem justifying high ratings when they occur in a results-based system. It is a matter of presenting performance results to the review panel, explaining standards and moving on. The "fix" would therefore not be to try to finesse employees' ratings to fit the guidelines, but rather to reset their goals for the following year — in other words, raise the bar. In other words, although the ratings distribution guidelines are exceeded in the current year, the goals are adjusted upward and thus next year's ratings are more likely to be within guidelines.
A Few Supplementary Techniques
Here are the supplementary techniques. They are not mutually exclusive; you may use one or more of them in combination with the above recommended method to achieve control over rating inflation. If you find success with methods not included in this list, let us know and we will add to this set of tools.
- Next-level review of draft appraisals — Policy requires that the next-level manager review all appraisals completed by the supervisors who report to them before the supervisors share the appraisals with their employees. However, if managers review appraisals on a piecemeal basis, reviewing them as they are submitted over a period of several days, it can thwart the intended effect of the process. The next-level manager should not approve any appraisals until all have been received and reviewed. This way the manager will know what the full distribution looks like before judging whether any one rating is inflated.
- Justification for "Outstanding" ratings — If supervisors rate their employees "Outstanding," they must provide additional documentation. The agency can require this extra level of documentation in its policy. This requirement would call for data-based documentation and specific examples of accomplishments rather than generalizations or descriptions of effort.
- Normative rating procedure for "checklist" positions — In the checklist approach to performance management, employees are evaluated on a common set of expectations. If a supervisor manages several employees who are all doing the same job, their appraisals can be done side-by-side, which makes it easier for supervisors to be consistent in assigning ratings, which in turn may encourage less inflation.
- Calibrating the ratings — This is a procedure that brings supervisors in synch with each other in using a common mental "yardstick" for evaluating their employees' performance. It involves having supervisors meet with their next-level manager and review each of their employees' performance and ratings. If any ratings are "out of whack" in relation to actual performance, these discrepancies come to light in the calibration sessions and supervisors can make the necessary rating adjustments.
- Publish ratings distributions — Within your agency, publish each manager's distribution of ratings. The social pressure associated with being an "easy grader" may influence a manager to take a more disciplined approach to performance management within his or her group. If you use this approach, you should establish a standard for how large a manager's group must be in order for its ratings distribution to be published. If there are too few ratings making up the distribution, say 25 or fewer, employees may become concerned that others are able to guess what rating they as individuals received. (Recall that completed appraisals are confidential documents.)
- "Best practice" awards — Take a positive approach! Set up criteria for a "best practices in people management" award. As criteria, include some measures of how effectively supervisors and managers carry out performance management, such as clarity of work plans, quality of written appraisals, rating adjustments required, work unit productivity, etc. Give an annual award to divisions that meet all the criteria and the supervisor who stands out as best performance manager.
