ANALYSIS OF BENCHMARKS™

Summary

Benchmarks™ is a multirater assessment instrument designed by the Center for Creative Leadership. Most seminar participants (3,311) participated in 360° feedback. Identical surveys were completed by the participant and the observers. The observers were the participant’s supervisor and (up to) five peer and five subordinate employees. Once completed, each survey was submitted to the Graduate School, USDA, for analysis. The results measure self perception in relation to observer perception of 16 skills and perspectives important for success, which were:

• balance between personal life and work
• confronting problem employees
• decisiveness
• leading employees
• setting a developmental climate
• work team orientation
• hiring talented staff
• compassion and sensitivity
• self-awareness
• acting with flexibility
• being a quick study
• putting people at ease
• resourcefulness
• straightforwardness and composure
• hiring a talented staff
• building and mending relationships

The Graduate School, USDA, provided 2,816 individual feedback reports to participants and several composite group profiles to the National Park Service. The analysis provided insights into the leadership skills and perspectives of NPS supervisors and managers as compared to a norm group. The analysis also identified developmental needs.

Overall, NPS supervisors and managers were in line with the norm group and did not score abnormally low on any of the 16 skills and perspectives. The norm group database consists of more than 80,000 supervisors and managers in the public sector. NPS supervisors and managers rated themselves lowest compared to the norm group on:

• balance between personal life and work
• confronting problem employees
• decisiveness

They rate themselves highest compared to the norm group on acting with flexibility.

Their observers rated the supervisors and managers lowest compared to the norm group on decisiveness and highest compared to the norm group on hiring talented staff.

Overall, the NPS supervisors and managers rate themselves higher in the following skills and perspectives than their observers rate them:

• decisiveness
• leading employees
• setting a developmental climate
• work team orientation
• hiring talented staff
• compassion and sensitivity
• self-awareness
• acting with flexibility

This illuminates some collective blind spots and areas in which to focus training and development.

NPS supervisors and managers do not think they do well in confronting problem employees and decisiveness, and neither do their observers. Observers rated all groups lowest on these two qualities. The supervisors and managers think that they have a better work team orientation than their raters think they have. The difference in scores was greater for this quality than for any of the other skills and perspectives.

This data indicates that high priority in training and development should be on:

• work team orientation
• decisiveness
• confronting problem employees

Overall, in all groups profiled, observers rate NPS supervisors and managers higher than the supervisors and managers rate themselves on:

• being a quick study
• balance between personal life and work
• putting people at ease

NPS supervisors and managers underrate themselves on being a quick study more than on any other quality.

Overall, the male and female group profiles appeared to be very similar. There are larger differences among the other groups (random population, managers [superintendents], executives [SES], males, females) profiled than between the male and female groups. Observers rated females higher than males on almost four times as many qualities as the qualities for which they rate males higher than females.

All five groups (random population, managers [superintendents], and executives [SES]) profiled agree on which eight of the 16 skills and perspectives (qualities) are most important for success:

• acting with flexibility
• resourcefulness
• leading employees
• work team orientation
• straightforwardness and composure
• setting a developmental climate
• building and mending relationships
• hiring a talented staff

Decisiveness tied for number eight for the SES group.

NPS employees and the norm group had almost identical lists of the eight most important qualities. Building and mending relationships was replaced by decisiveness on the norm group list.

Of the eight considered most important in the National Park Service, observers found that we do not do well on five:

• acting with flexibility
• leading employees• work team orientation
• setting a developmental climate• building and mending relationshipsThis may be a further indication of training and development needs.

Benchmarks™ Composite Group Profiles

Five composite group profiles were run by the Center for Creative Leadership on NPS Benchmarks™ data. The profiles are:

1. NPS males, December 5, 1996. This profile includes the 285 males in the December 4, 1996, group.
2. NPS females, December 5, 1996. This profile includes the 115 females in the December 4, 1996, group.
3. NPS random population, September 26, 1997. This profile was run on data from 187 randomly selected NPS participants.
4. NPS managers, September 26, 1997. This profile was run on data from 190 superintendents.
5. NPS senior executives, September 26, 1997. This profile was run on data from 40 SES-level participants.

An NPS random population profile was also run in 1996 as a combination of the male and female groups. It is not included in this analysis because the size of the Center for Creative Leadership’s norm groups database was vastly increased in 1997. By using the 1997 random population profile, NPS participants can be compared to a larger, more current norm group. The 1996 male and female profiles are included in this analysis because of the high level of interest that seminar facilitators and participants expressed in obtaining such a profile. Composite group profiles provided to regions are not included in this analysis.

The group profiles were analyzed to gain insights into the leadership skills and perspectives of NPS supervisors and managers and to identify developmental needs. The analysis, which has been reviewed by the Center for Creative Leadership, follows:

Comparison to the Norm Groups

Overall, the NPS groups did not score abnormally low on any of the 16 skills and perspectives compared to the norm group, which consists of more than 80,000 supervisors and managers in the public sector.

The NPS random population group self (S) scores were lowest compared to the norm group for balance between personal life and work, confronting problem employees, and decisiveness, and they were highest compared to the norm group for acting with flexibility. Their all observer (O) scores were lowest compared to the norm group for decisiveness and highest compared to the norm group for hiring talented staff.The NPS managers (superintendents) group self (S) scores were lowest compared to the norm group for confronting problem employees and were highest compared to the norm group for hiring talented staff.

Their all observer (O) scores were lowest compared to the norm group for decisiveness and highest compared to the norm group for hiring talented staff.The NPS senior executives (SES) group self (S) scores were lowest compared to the norm group for balance between personal life and work and were highest compared to the norm group for hiring talented staff. Their all observer (O) scores were all midrange and higher, with the highest compared to the norm group for hiring talented staff.

Comparison of Mean Scores

Overall, the groups’ self scores and all observer scores are higher in the following order: SES, superintendents, and random population. That is, the SES group rated themselves higher than the superintendents; and the superintendents rate themselves higher than the random population of supervisors and managers. The SES all observer group rated the SES higher than the superintendent’s all observer group rated the superintendents, and so forth.

A comparison of whether the self (S) or all observer (O) scores were higher was made from the mean scores on the Benchmarks™ scales pages of the three groups profiled in September 1997 and from the male and female profiles in December 1996. The results are shown in table 1, where > stands for greater than,* indicates the largest S>O for each group, ** indicates the largest O>S for each group, and the numbers are the difference between S and O (S mean score minus O mean score).

Observations from table 1 follow. “Qualities” refer to the 16 Benchmarks™ skills and perspectives.

Observer ratings for all groups are higher than the self ratings for the following qualities:

• being a quick study
• balance between personal life and work
• putting people at ease

NPS supervisors and managers underrate themselves on being a quick study more than on any other quality. The mean for this quality was highest as rated by all observers for three of the groups profiled and second highest for the other two groups. The mean self ratings were generally low for all five groups.

Self ratings for all groups are higher than the all observer ratings for the following qualities:

• decisiveness
• leading employees
• setting a developmental climate
• work team orientation
• hiring talented staff
• compassion and sensitivity
• self-awareness
• acting with flexibility

Self scores on work team orientation were consistently higher than all observer scores for all groups profiled. This indicates that on the average, NPS supervisors and managers think they have a better work team orientation than their raters think they have. Furthermore, the numerical difference between the self mean and the all observer mean was greater for this quality than for any of the other 15 qualities. This data indicates that high priority in training and development should be given to work team orientation.

Managers (superintendents) rated themselves much higher on hiring talented staff than their all observer group rated them. Hiring talented staff tied for the largest difference of S>O for this group.

To the extent that a better self perception than observer perception is an indication of training and development needs (which is logical according to the Center for Creative Leadership), the three groups profiled in September 1997 need training and development in the order of importance shown on table 2:

Table 3 shows a ranking of supervisors and managers self scores for each group. The rankings are based on the mean scores from the Benchmarks™ scales page of the three group profiles in September 1997 and from the male and female profiles in December 1996. The lower the ranking number, the higher the mean score. For example, a 1 for hiring talented staff indicates that the group self rating was higher for that quality than for any of the other 15 qualities; a 16 for confronting problem employees indicates that the group self rating was lower for that quality than for any of the other 15 qualities.

Table 3 enables an easy comparison of how each of the five groups profiled perceive themselves, on the average, compared to the other groups. The largest differences in rankings are noted below:

Managers (superintendents) and executives (SES) rate themselves relatively higher on resourcefulness than do the other groups.

All groups rate themselves relatively low on decisiveness, with executives (SES) rating themselves relatively lowest of the five groups. All groups rate themselves relatively low on building and mending relationships. All groups rate themselves very low on confronting problem employees.

Managers (superintendents) and executives (SES) rate themselves relatively low on compassion and sensitivity. Executives (SES) rate themselves relatively high on self-awareness. Random population and males rate themselves relatively high on balance between personal life and work.

Females rate themselves relatively high on putting people at ease, and relatively low on work team orientation, being a quick study, and hiring talented staff.

According to the Center for Creative Leadership, low mean self rating is a less valuable indication of training and development needs than is other data. However, to the extent that lower self perception is an indication of training and development need, the three groups profiled in September 1997 need training and development in the order shown on table 4 (8 lowest-ranked qualities by self).

Table 5 shows a ranking of all observer scores for each group. The rankings are based on the mean scores from the Benchmarks™ scales page of the three group profiles in September 1997 and from the male and female profiles in December 1996. The lower the ranking number, the higher the mean score. For example, a 1 for balance between personal life and work indicates that the all observer rating was higher for that quality than for any of the other 15 qualities; a 16 for confronting problem employees indicates that the all observer rating was lower for that quality than for any of the other 15 qualities.

Table 5 enables an easy comparison of how all observers perceive each of the five groups profiled, on the average, compared to the other groups. There is more consistency in rankings across the groups by all observers ratings than by self ratings. The largest differences in rankings are noted below:

Observers rated all groups lowest on decisiveness and confronting problem employees. This conforms closely to the self perceptions.

Observers rated managers (superintendents), executives (SES), and males relatively higher on work team orientation than they rated the other groups.

Observers rated random population and females relatively higher on compassion and sensitivity.

Observers rated managers (superintendents) and executives (SES) relatively lower on balance between personal life and work.

To the extent that lower observer perception is an indication of training and development need, the three groups profiled in September 1997 need training and development in the order shown on table 6 (8 lowest-ranked qualities by all observers):

(1 is highest ranked quality, 16 the lowest)

Table 7 shows the information from tables 3 and 5 on a single table, enabling an easier comparison of the rankings of supervisors and managers self (S) scores and their all observer (O) scores.

*1 is highest ranked quality, 16 is lowest.

Analysis of table 7 enables comparisons between the supervisors/managers self perceptions and those of their raters. Those qualities for which there is marked agreement or disagreement stand out.

There is marked agreement across all groups on doing whatever it takes and straightforwardness and composure. Both self and all observers rank these qualities high.

There is marked agreement across all groups on decisiveness and confronting problem employees. Our supervisors/managers do not think they do well in confronting problem employees and neither do their raters. It is at the bottom of the rankings for all groups for both self and all observers, except self ratings for executives (SES), in which case it is next to last.

There is marked disagreement on being a quick study. Other disagreements are apparent upon closer examination, especially at the group level rather than across all groups.

The December 1996 male and female group profiles and the preceding analyses provide insights into gender differences regarding the 16 Benchmarks™ skills and perspectives. Overall, the male and female profiles appear to be quite close. There are larger differences between the three other groups profiled in September 1997. The most notable differences between males and females were:

Males rate themselves higher than females in:

• balance between personal life and work
• hiring talented staff
• work team orientation
• confronting problem employees

Females rate themselves higher than males in:

• straightforwardness and composure
• putting people at ease
• being a quick study
• doing whatever it takes
• acting with flexibility
• compassion and sensitivity

Observers rate males higher than females in:

• balance between personal life and work
• work team orientation*
• hiring a talented staff*

Observers rate females higher than males in:

• compassion and sensitivity
• straightforwardness and composure• putting people at ease
• being a quick study
• doing whatever it takes
• resourcefulness
• acting with flexibility*
• leading employees*
• confronting problem employees
• setting a developmental climate*
• self-awareness*

* While observers rate males higher than females on work team orientation and hiring a talented staff, they rate each group low on those qualities; and while observers rate females higher than males in setting a developmental climate, compassion and sensitivity, leading employees, acting with flexibility, and self-awareness, they rate each group low on those qualities.

Overall, the observers rated females higher than males.

Males see themselves as being much worse at putting people at ease than do females, but observers rate both groups high in this quality. Males rate themselves considerably lower than females on being a quick study, but their observers rate them high, and they rate females even higher.

Females see themselves as having a poorer balance between personal life and work than do males. Observers perceive both groups as having a better balance than they perceive in themselves.

Table 8 shows the importance for success rankings in order from 1-16 for each group. 1 is the quality that each group considers to be the most important for success, and 16 is the quality that each group considers to be the least important for success. They are derived from the self rankings on the importance for success rankings page of the three group profiles in September 1997 and from the male and female profiles in December 1996.

*1 is highest ranked quality, 16 lowest.

All five groups agree on which 8 of the 16 skills and perspectives are most important for success. They are, in the order ranked by all observers:

• acting with flexibility
• resourcefulness
• leading employees
• work team orientation
• straightforwardness and composure
• setting a developmental climate• building and mending relationships
• hiring talented staff

Decisiveness tied for number 8 for the executive (SES) group.

The all observer groups for each of the five groups also agreed with the self groups and with each other on the top 8, except that the all observer group for the executives included decisiveness (hiring talented staff and work team orientation tied for 8-9).

NPS employees and the norm group had almost identical lists of the eight most important qualities. The norm group included decisiveness rather than building and mending relationships.Of the eight believed most important to the National Park Service, all observer groups find that employees do least well on five. They are:

• acting with flexibility
• leading employees
• work team orientation
• setting a developmental climate
• building and mending relationships

Those five are possible training and development needs for NPS supervisors and managers since they are considered important to the organization and since, on the average, supervisors and managers were rated low on them by their observers. The question is whether training should be focused on what the current culture considers important or whether other considerations should determine the best direction for training and development. It is evident from the rest of the analysis that training and development is needed in the above five qualities.

BENCHMARKS™ IMPACT STUDY

A graduate student at Johns Hopkins University, Rainette Bannon, conducted an independent Benchmarks™ impact study on leadership seminar participants. She found that the following developmental goals were selected by the most participants (in order):

• work team orientation
• setting a developmental climate
• building and mending relationships
• leading employees
• decisiveness
• confronting problem employees
• compassion and sensitivity

Of the eight believed to be most important in the National Park Service, and of those we do least well in according to our all observer group, acting with flexibility was not selected as a developmental goal (see table 8).