Report of Ad Hoc PSI Evaluation Committee
April 3, 1998
Mary Wolfinbarger, Chair, Marketing
Mary Anwar, Educational Psychology
Dot Goldish, Chemistry and Biochemistry
Lloyd Hile, Engineering
David Viera, Film and Electronic Arts

Summary

CSULB Unit III employees were mailed a survey in February 1998. Attitudinal items on the questionnaire were largely based on four focus groups conducted with faculty in October and November 1997 (one of these groups consisted of PSI college committee representatives). Three hundred and fifty of approximately 1500 responded for a 23% response rate. The sample is quite representative across colleges and across rank of T/TT faculty. However, lecturers are under-represented in this sample. Following are the major conclusions based on the survey data.

• CLA and HHS lead the respondents with 30 and 21 percent respectively. Over 80% responding are T/TT faculty, and almost 70% of those are full professors. Lecturers are 14% of the respondents.
• Almost 60% of respondents had received at least one PSI. Over 70% of respondents are in favor of abolishing the PSI. Receiving PSIs results in respondents being on average more positive about the process, especially if they have received PSIs in two or more years. Nevertheless, those receiving PSIs in just one year are still, on average, negative about the process.
• Faculty do not have confidence that what ought to be rewarded is being rewarded. Respondents believe that PSIs should reward teaching, research and service (in that order). Respondents most strongly believe the most important area to reward is teaching, but that teaching is least rewarded of the three areas.
• Respondents believe that PSI awards are more likely to reward political insiders than teaching, research or service. Additionally, over half of those responding to the question about misrepresentations on PSI application believe they "know" about misrepresentations.
• Moreover, respondents are not in favor of PSI applicants having to be meritorious in only one area (as opposed to the current two area requirement).
• Of the twelve potential changes listed (excepting abolishing the PSI), only 4 received support from more than half of the faculty. These four are (1) standardizing requirements for reporting student evaluations (2) establishing time lines that allow more time for college committees to consider PSI awards (3) requiring PSI recipients to skip a year before applying again and (4) integrating the PSI process with RTP. These last two items, while supported by more than half the faculty, are strongly opposed by many respondents.
• While lecturers feel relatively disadvantaged by the PSI process, they do not believe that moving the process to the University level is the solution.
• Faculty appear to believe that the college level is the best place to evaluate the PSIs. There is little support for the idea of moving the process up to the university level or down to the department level.
• Faculty believe that across-the-board raises should be given priority over merit pay. Nevertheless, in aggregate, only a slight majority of faculty believe that most of their colleagues are meritorious. There is a great deal of variance in opinion concerning giving of multiple steps; nevertheless, over half of the faculty believe that multiple steps should not be given.
• More than half of respondents believe that the PSI is of sufficient monetary value to be worth the effort of applying. Nevertheless, faculty would prefer the PSI to be worth more. Respondents from CBA, HHS and Engineering are most likely to prefer increasing the value of the PSI. Moreover, associate professors, assistant professors and lecturers are all more likely to prefer increasing the value of the PSIs than are full professors.
• Respondents have a tendency to believe that their membership group (associate, assistant, full, lecturer) is relatively handicapped in getting a PSI. This tendency is somewhat less pronounced for full professors, but full professors in our sample are more likely to have received a PSI than associate professors or assistant professors.
• The number of assistant professors responding is small (7% of the sample) but the results suggest that perhaps assistant professors are more positive about the prospects of the PSI for building a merit-based CSU.
• CBA is the only college whose respondents are on average positive (albeit slightly) about the PSI process. Respondents from COTA, on the other hand, are most negative.
• CBA, CLA and Engineering faculty scored highest on perceived political "insiderism." However, CBA respondents are the only ones who tend to even slightly believe that PSIs reward teaching (m =4.3). Alternatively, CBA respondents are least likely of all the colleges to believe research is rewarded by the PSI. Only CLA respondents report that service is rewarded by PSIs.
• Those who have served on PSI committees (n=26) believe that their committees are doing a good job. They are less likely than the rest of the sample to believe the process should be integrated with RTP, or that the process should be moved to the University level for either T/TT faculty or lecturers. The "servers" are also more likely to believe that faculty members receiving PSIs should be required to skip a year or more before applying again.
• A fuller discussion of results follows. The questionnaire, focus group report, and frequency distributions for all survey items appear in the appendix to this report.
  
  TOP

In addition, the following important issues and perceptions were expressed by participants in the focus groups.

• Applicants were sometimes "fuzzy" on dates and actual work done.
• Decision makers reported that they were concerned that information on applications could not be verified.
• Decision makers reported that it was difficult to compare research in unrelated fields.
• Decision makers reported that it was difficult to compare research and university service based on quality of work.
• It was sometimes perceived as difficult, if not impossible, to rank large numbers of candidates.

• The PSI document lacks specificity on how teaching should be reported.
• More comparative data on teaching/student evaluation is needed.
• Teaching was measured solely by "flawed" student evaluations.
• Candidates could not be differentiated on teaching; most reported better than average evaluations.
• Some qualified faculty do not apply.
• There are no clear criteria for assigning multiple steps.
• The decision to "spread wealth" varied by college.

• Either research or "political insiderism" were perceived as having the most weight in PSI decision making.
• Some disciplines inherently have less research opportunities; faculty from these disciplines felt relatively disadvantaged in the PSI process.
• Departments with greater representation on committees sometimes received more awards.
• Many feel that the PSI process is demoralizing and divisive for many faculty.
• Many feel that the PSI process is perceived as somewhat burdensome on applicants and committee members.
• Lecturers were often believed to be disadvantaged by the process; this perception was shared by lecturers and T/TT faculty.
• Many believed that lecturers should be considered separately.
• Some participants perceived that monetary value insignificant therefore not worth the effort.
• Multiple steps were viewed negatively by many participants.

A more detailed focus group summary appears in the appendix of this report along with frequency distributions for survey items.

TOP

In spring 1997, FPPC commissioned an ad hoc committee to evaluate faculty responses to the Performance Salary Step Increase (shortened to either PSSI or PSI) program which begun on campus in 1995-1996. The FPPC has been responsible for drafting policy statements concerning the implementation of the PSI, and desired to better understand faculty perceptions and attitudes in order to improve PSI implementation. Our ad hoc committee was charged with collecting information rather than making suggestions. Thus, we have focused on faithfully representing faculty opinions as much as possible in designing this research.

In order to achieve this objective, two research methodologies were employed: focus groups and surveys. The focus groups were conducted to better understand what issues Unit III employees felt we should investigate. Also, the focus group results provide a context in which the more quantitative results can be interpreted. Last, more complex issues can be treated in a more detailed fashion in the semi-structured environment provided by focus groups. On the other hand, the survey provided us access to a much wider Unit III employee population, and resulted in a more generalizable measure of the degree to which faculty hold various opinions. The survey was designed to be short in order to maximize our response rate. Readers should be advised that the focus of this study is faculty perceptions of the process, and it is sometimes difficult to ascertain to what degree these perceptions are reflective of PSI realities.

It was our objective to reach as many faculty as possible in a relatively short time frame; thus, participants who wanted to participate in the focus groups and could not fit in with our schedule were offered the opportunity for individual interviews and/or the opportunity to email us at pssi@csulb.edu. Two faculty members took the committee up on the interview offer.

TOP
Data Collection and Characteristics of Survey Sample

In February 1998, about 1500 questionnaires were mailed to Unit III employees on campus; 350 were returned for a response rate of 23%. The response rate compares quite favorably with the "normal" rate of return reported for marketing research surveys, which is 5-6%.

Over 4/5 of the respondents are full time faculty, and of those faculty almost 70% are full professors, and almost 25% are associate professors. Only 7% identified themselves as being assistant professors. Lecturers make up 14% of the sample. Close to 60% of those responding had received a PSI in at least one of the three years the program has been running.

The numbers for the entire population to which this study would like to generalize, Unit III employees, are provided by the Academic Affairs. The 350 Unit III employees responding to this survey are reasonably representative of their colleges (see table below).

*Faculty early retirement and pre-retirements are counted as full positions regardless of semester of service and faculty with joint positions have been counted once.

*Faculty early retirement and pre-retirements are counted as full positions regardless of semester of service and faculty with joint positions have been counted once.

Nevertheless, and not surprisingly, T/TT faculty are over-represented in this sample (82% of this sample vs. 55% of Unit III employees). Lecturers are not only under-represented in this survey; Gary Reichard, V.P. of Academic Affairs, has indicated that they are under-represented in applying for PSIs. The breakdown of full, associates and assistants in our sample, however, is very close to that of the University at large (see table above).

Sixty percent of our sample received at least one PSI during the three years the process has been in place, while the comparative number from the University is 33%. As a percentage of T/TT faculty, however, the number of people who have received at least one PSI is 59%, a number closer to that indicated by our sample.

In summary, across college and rank of T/TT faculty, the sample is quite representative. However, the T/TT faculty are over-represented in our study, and related to this, those receiving PSIs are over-represented. Because those receiving the award are slightly more positive, the aggregate averages reported for this study may be slightly biased in a positive direction. Nevertheless, a sufficient sub-sample of lecturers (N=49) was obtained in this study, allowing us to estimate and compare lecturer opinions and perceptions and to compare them with those of T/TT faculty.

TOP

The attitudinal items in the survey were based largely on four focus groups conducted with faculty on this campus in October and November of 1997. One of the focus groups was made up exclusively of members of college PSI committees. The survey is 1 1/2 pages and contained three sections: (1) attitudes concerning the PSI (2) potential changes to the PSI policy and (3) categorical items including college, status, and whether or not recipients had applied for and received PSIs. This report is divided into 6 sections:

• Section I: General attitudes
• Section II: What PSIs should and do reward
• Section III: Spreading the Wealth
• Section IV: Monetary Value of PSIs
• Section V: Who Gets PSIs
• Section VI: Potential changes to the PSI Process

All analyses of mean differences are ANOVAs (Analysis of Variance) except where indicated otherwise. The term "statistically significant" in this report refers to a result which is statistically reliable at a level of p<.05 (the probability of the particular finding based on the sample arising from the actual population data by coincidence is 5% or less). All significant p-values in this report indicate that two or more categories have means which are reliably different from each other. On the other hand, the parenthetical notation "(p=n.s.)" indicates that there are no statistically reliable category differences between reported means for the item being analyzed.

TOP

Table 1: General Reactions to the PSI (on a 7 point scale with 7=strongly agree)

Perhaps not surprisingly given the campus buzz concerning PSIs and faculty responses in our focus groups, general opinions about the PSI are negative. In looking over the distribution of responses, slightly less than 30% disagree (choosing 1, 2, or 3 on the 7-item scale) that the PSI should be abolished. Thus, less than 30% of respondents are positive about the PSI process.

TOP
Receiving PSIs improves average responses to the general attitudinal items. In fact, the more years respondents report that they received the PSI, the more likely they are to be positive about the PSI. On a composite attitudinal measurement including the first 6 items above (with some items reverse scored so that the direction of the items would be consistent), those receiving the PSIs more often are much more positive (see means in Table 2 below).

The same pattern is repeated for question 34 alone (not included in the 6 item composite measure).

TOP
Only one item approaches a significant difference in general attitudes by level of respondent (full, associate, assistant): assistant professors (n=17 on this question) are more likely to believe that the PSI could be instrumental in building a more meritorious CSU (full professors=3.2, associate professors=3.3, assistants=4.6, p=.06). The small number of assistant professors responding resulted in difficulty obtaining significant differences. However, concerning abolishing the PSI, assistant professors average 4.1 against 5.0 and 5.1 for associate and full professors. Moreover, a similar pattern emerges for assistant professors on each of these 7 general items. Full time faculty do not differ significantly from lecturers in their responses to abolishing the PSI (question 34), with full time faculty averaging 5.1, and lecturers averaging 5.0 .

Sample size lessens the ability to obtain statistically significant differences between the colleges (there were only 20 -30 responses for each of the smaller 4 colleges). However, the pattern of results for the general PSI items indicate that CBA is slightly less negative than the other colleges.

Overall....................................... 291................................ 5.0............................... 2.5
**p=.07

TOP

Respondents believe that excellent teaching, research and service (in that order of importance on average) should be rewarded by the PSI. Teaching is so important to respondents that both the median and the mode are 7 ("strongly agree")! The gap between what should be and what is rewarded is 2.6 for teaching, 1.7 for research and .9 for service. Thus, respondents believe that none of the three areas are rewarded by the PSI process as much as they should be. Instead, respondents tend to believe that "political insiders" are rewarded.

CBA is the only college who scored greater than 4 on the items concerning the PSI rewarding teaching (m =4.3. p<.10). Those colleges believing that teaching is least likely to be rewarded are HHS and Engineering (m =2.8 and 2.9 respectively). The only colleges believing research is rewarded (means over 4) are CNSM at 4.8 and Education at 4.3. CNSM is most likely to believe research is rewarded (m =4.8) while CBA is least likely to believe research is rewarded by PSIs (m =3.0). CLA is the only college believing that service is rewarded (m =4.6). Respondents from all colleges feel political insiders are rewarded. However, CBA, CLA and Engineering scored highest on political "insiderism," with each of the three colleges being rated above "5" on average (m s=5.2, 5.2 and 5.8 respectively). Assistant professors believe most strongly that research should be, but is not rewarded by the PSI. Perhaps reflecting their newness to the university, assistant professors are also least likely to "know" that there were misrepresentations on PSI applications.

TOP

Faculty are very positive about the idea of giving across the board raises priority over funding for PSIs. Nevertheless, faculty only mildly believe that their colleagues are uniformly meritorious or that multiple steps should be restricted (although "7" was the modal response on the later item, indicating that a large group of respondents felt very strongly that there should be no multiple steps). 

Although the differences are not statistically significant, CNSM is the college most likely to believe that their faculty are mostly meritorious (m =4.8) while CBA is least likely to believe this (m =3.7) . This result is particularly interesting considering that CBA is the college that has had the most applicants for dollars available, and the most spreading of awards (rarely giving multiple steps). Those receiving PSIs, and particularly those receiving PSIs in multiple years, are less likely to believe that most of their colleagues are meritorious.

While respondents do not necessarily feel like the size of the PSI is too small to be worth the trouble of applying, more than half of those responding think PSIs should be worth more salary dollars.

Respondents from CBA, HHS and Engineering are most likely to believe the PSIs should be worth more dollars (m s=5.5, 5.3 and 5.2 respectively), with Education being least likely to prefer increasing the size of the PSI (m =3.9) (p=.05). Those receiving PSIs are also more likely to believe they should be worth more salary dollars than those not receiving PSIs (m =4.4 for non-recipients, 5.0 for recipients, p<.05). Though the differences are not statistically significant, associate professors, assistant professors and lecturers, on average, appear to prefer the PSIs to be worth more money (m =4.5 full professors, 4.9 associate professors, 5.2 assistant professors, 5.1 for lecturers). Interestingly, those serving on PSI committees are less likely to prefer that the awards be worth more (m =4.0 vs. 4.8 for those not serving, p<.05).
TOP

In aggregate, the respondents believe that meritorious lecturers are somewhat disadvantaged in the PSI process (median=5). However, there was some consensus that full professors are not disadvantaged, and if our sample is an indication, full professors were somewhat more likely than associate professors and assistant professors to have received a PSI (68% of full professors vs. 56% of associate professors and 48% of assistant professors reported receiving at least one PSI, p=.10).

The differences between colleges on these two items are not statistically significant; however, COTA is the only college averaging over 4 on the item concerning full professors being less likely to receive PSIs. Thus COTA's respondents are on average most likely to agree that full professors were less likely to receive PSIs (m =4.6). CBA and Education respondents report the lowest means on the full professor item (3.3 and 3.4). Moreover, assistant professors and associate professors are much less likely than full professors to believe that full professors are disadvantaged in the process (m =1.8 for assistant professors, 2.7 for associate professors and 4.4 for full professors, p<.05). Those serving on PSI committees are less likely to believe that full professors are disadvantaged than those not serving (m =2.9 for committee members, m =3.8 for non-members, p<.05).

Lecturers are more likely than tenured and tenure-track faculty to believe they are disadvantaged in the process (T/TT faculty, m =4.4, n=192, lecturers, m =5.8, n=40, p<.05). What is perhaps most striking about the results on these two items is that respondents from all groups believe on average that their membership group is relatively disadvantaged in the PSI process!

TOP

Only 4 of the possible 12 changes receive support from more than half of our respondents. The most widely supported changes involve standardizing requirements for reporting student evaluations and establishing time lines that would allow sufficient time to consider applications. While there is a great diversity of opinion, more than half of respondents also agree that (1) PSI recipients should be required to wait a year or more before applying for a PSI again and (2) the PSI process should be integrated with the RTP processes. There appears to be no support for moving the PSI consideration process away from the college level to either the department or University level.

Only one item among the 12 averaged significantly different between colleges: CLA and Engineering are the schools most concerned about establishing time lines (both schools averaged over 5 on this item). Interestingly, those serving on committees are less likely than those not serving to desire longer time lines (m =4.0 for servers, m =4.8 for non-servers, p<.05).

Lecturers, associate professors and assistant professors are more likely than full professors to desire the PSI process be integrated with RTP (full professors=3.7, lecturers=4.4, associate professors=4.6, assistant professors=5.6, ps<.05). On the other hand, full professors are the only group even slightly positive about the idea of restricting PSI committee membership to full professors (full=4.1, associate professors=2.8, assistant professors=2.6, lecturers=1.9, ps<.05). Surprisingly, lecturers are not significantly more likely to feel they should be evaluated at the University level (2.7 T/TT faculty, 3.0 for lecturers, p=n.s.). While none of the groups are on average positive about moving PSI evaluation to the department, full professors and assistant professors are more positive about departmental evaluation than are associate professors (3.7 for full professors and 3.9 for assistant professors vs. 2.9 for associate professors, p<.05).

TOP

Those who had served on PSI committees differ significantly in some of their opinions regarding changes as compared to those who had not (although there were only 26 servers in our sample). The items for which there are suggestive or significant results follow.

Those serving on committees are (1) less likely to desire more time to consider applications (2) more likely to desire recipients to skip a year or more before applying (3) less likely to believe the process should be integrated with RTP (4) less likely to believe full time faculty should be able to apply only in one area (5) less likely to believe that applications should be considered at a University level for either T/TT faculty members or lecturers.

TOP

Although the response rate to the survey is high compared to average survey response rates, over 70% of Unit III employees did not respond to this survey. Generally, those taking the time to respond to a survey tend to differ from non-respondents in that population members who are more opinionated -- both positive and negative -- are most likely to respond. Because T/TT faculty responded disproportionate to their numbers in the Unit III population, and because T/TT faculty are more likely to apply for and receive the PSIs and thus to be more positive about them, the estimates in this report may be slightly biased in a positive fashion.

Nevertheless, those who responded are more likely to have knowledge of, and involvement with, the PSI process, and thus to have been relatively thoughtful about the PSI process. The fact that our respondents relatively frequently availed themselves of the "no opinion" option suggests that at least some potential respondents did not respond because they do not have knowledge of, and opinions about, the PSI process (for example, one lecturer who did respond wrote that she had not known that lecturers can apply for the PSI). Moreover, a disproportionate number of "opinion leaders" (members of a population who have many ties to other population members and tend to give a great deal of information and be asked for information and opinions by others) are more likely to have responded to the survey; as these opinion leaders in Unit III hold a disproportionate share of influence, their opinions may be somewhat representative of Unit III employee opinion in general.

Perhaps a more serious limitation of this study is the fact that this is a one-time study. It is difficult to predict how faculty opinions may or may not change over time based on one study. The descriptive results reported here cannot be tied definitively to causal factors. All readers are cautioned to be careful to consider alternative explanations for results, even if those alternative explanations are inconsistent with preexisting beliefs. For instance, do faculty not getting PSIs perceive the process negatively because it is unfair, or because these faculty are unlikely to get PSIs because they are not meritorious? While this study provides some tentative clues to the answers to this and other questions, studying the process over time should provide more definitive answers.

It is important to note that negative faculty opinions may arise mainly from a perception that merit pay is either unjust or inappropriate in the university environment existing at CSULB or may stem mainly from specific ill-conceived or poorly implemented policies or procedures. Of course, the most likely case is that both sets of factors play a role, a point of view generally supported by the pattern of evidence reviewed in this study. On one hand, the majority of faculty endorsed only 4 of 12 possible policy changes listed, perhaps indicating fewer objections to specific policies than to the general idea of merit pay. However, there do exist differences in the average and median responses to both specific possible policy changes and attitudes that are thus incompletely "explained by" or "attributable to" more general attitudes toward the idea of merit pay on a CSU campus. Nevertheless, this is an important distinction because, in the first instance, policy or procedural changes will do little to change opinions. In the second instance, policy or procedural changes could be very helpful.

Understanding the degree to which negative faculty opinions are the result of generalized opinions rather than specific concerns is critical. FPPC must realistically appraise the likely impact of any policy and procedure changes and realize that while such changes may slightly increase perceived procedural fairness, they may nevertheless do little to mitigate current negative faculty opinions about the PSI merit pay system.

Interestingly, the results appear to suggest that getting PSIs in multiple years lessens negative opinions about the PSI process. Moreover, there is no comparison of the PSI to other campus processes. For example, respondents appear to believe that PSIs take too much time of applicants and college committees; perhaps respondents would believe that other campus processes, such as SCAC, or applying for leadership awards, are also too time-consuming as well. It is also possible the resistance to the current PSI process represents a generalized resistance to any new initiatives on campus, especially those introduced by administrators, rather than being specific to the merit pay question. In the absence of specific data to address the degree to which the findings reported here represent (1) specific objections to PSI implementation or (2) general objections to the idea of merit pay at CSULB or (3) general resistance to administrative directives, we recommend cautious interpretation of our results.

TOP