What structures, policies, and processes are in place for assessing student learning outcomes?
Over the past dozen years the campus worked within the framework of decentralization to foster open communication and obtained widespread agreement to strengthen its capacity to assess student learning, which, in turn, led to the integration of assessment into the recently revitalized process for periodic program review.
In the late 1990s and early 2000s several initiatives were undertaken to instill assessment of student learning at the program level into the campus culture, but these did not succeed. The first, a draft plan written by the Provost in 1995, was basically a top-down conceptual paper that was never widely circulated and garnered very limited support. The second, a faculty-driven policy on assessment, was adopted in 1998 by the Academic Senate (#98-06). However, it was merely an endorsement of the nine principles espoused by the American Association for Higher Education (AAHE), which did not stimulate enough interest to create either an action plan or the resources to support assessment.
The third initiative was the appointment of a cross-divisional Assessment Committee in 1995, which was given a budget and a half-time assessment coordinator to support it. Being appointed and not elected left the Committee with no formal power; however, the Committee awarded small grants to encourage academic departments and student services programs to undertake assessment. It also held two well-received, campus-wide Assessment Retreats. The result was diffusion of assessment across campus, but at very different levels depending on academic disciplines. For example, in 2001 when the Assessment Committee asked all academic programs for information on assessment activities, only one-third responded, the majority of which had professional accreditation. Assessment was widely perceived as—and indeed functioned as—an optional process completely separate from program review, and one that met with considerable skepticism and occasional opposition.
In 1999 the CSU Chancellor’s Office called for biennial accounting from all CSU campuses beginning in 2000. This centrally mandated assessment directive led to CSULB’s fourth attempt to develop a viable process for assessing student learning. In March 2001, a subgroup from the CSULB Academic Senate developed a statement of Campus Accountability Goals to be implemented over the next four years. The goals for AY 2002-2003 included determining at least five measurable student learning outcomes for majors in all degree programs; choosing and implementing at least one method of assessment in each program; and identifying ways to use assessment results to guide program improvement. However, with no perceived rewards for compliance and no sanctions for non-compliance, the statement and its timeline were seen as just another unfunded mandate from the Chancellor’s Office, and they were largely ignored.
In 2004, a new Associate Vice President for Graduate and Undergraduate Studies took ownership of both program review and assessment, combining them for the first time. This included reviews of academic degree programs as well as of advising, learning assistance, and other related programs not generally evaluated in the past. The half-time assessment coordinator was increased to full time and given program review duties as well. Assessment plans were required of all proposed new degree programs. At the same time, the CSU Chancellor’s Office began asking each campus to provide the percentage of all their degree-granting programs that had identified student learning outcomes as well as the percentage that had begun to conduct assessment of student learning. CSULB was among the campuses with the lowest percentages in both categories, which put assessment on the campus agenda for discussion.
A survey of academic departments by the Assessment Committee revealed that the program review process had, in effect, broken down. Reviews were split between two independent Academic Senate councils (one undergraduate and one graduate). Each required its own external consultant for each program review, dragging the process out over several years. Lacking a formal policy, the guidelines for program review had been revised and re-interpreted so often that, in frustration, some academic units threatened to boycott the process. The Provost appointed a cross-campus task force with representation from faculty, department chairs and deans to draft a formal program review policy. The Associate Vice President and the Coordinator for Program Review and Assessment attended a meeting of department chairs in every college to communicate the intent and content of the new proposed policy and to build support for its adoption. The policy called for a minimum of required information while still allowing considerable room for customization of the process by each college and academic unit. It also combined the two separate program review and assessment processes into one. And for the first time, it allowed professional accreditation processes to serve as program review, which many colleges had wanted. Approved by the Academic Senate in 2005 (#05-11), the policy incorporated assessment of student learning as a required element in program review. Each program review now results in a Memorandum of Understanding signed by the program head, the college dean, and the Vice Provost. The MOU spells out what is expected of the program over the next review cycle as well as resources provided to support these goals. Expectations for assessment of student learning have figured in every MOU to date.
As part of the Academic Senate reorganization described in Section I, the Assessment Committee and the two program review councils were dissolved and a new Program Assessment and Review Council (PARC) was created. In fall 2005, the CSULB Academic Senate endorsed a new timetable for reaching the campus’ assessment accountability goals and adopted a mandatory annual assessment report. The Associate Vice President adopted a successful new strategy for assessment, first decentralizing responsibility to the deans and encouraging the appointment of a college assessment coordinator for each of the seven academic colleges, and then decentralizing the assessment budget to the colleges. The Coordinator for Program Review and Assessment worked with the college coordinators to address the best way to implement assessment in each college. (Criteria 2.4, 2.5, 2.10)
These initiatives resulted in the first successful steps toward institutionalization of assessment of student learning on the CSULB campus. Committed leadership and open communication built widespread support for policies and practices that accepted the differences among decentralized academic units. The adoption of a required annual report on assessment from each academic unit has resulted in substantial progress on assessment of student learning, as seen in the following tables:
Undergraduate Degree Programs (N=~84) |
2005-06 |
2006-07 |
2007-08 |
|---|---|---|---|
% Not reporting |
44% |
15% |
12% |
% Programs with Plans only |
49% |
30% |
16% |
% Programs with Data/Changes |
7% |
55% |
72% |
Graduate Degree Programs (N=~66) |
2005-06 |
2006-07 |
2007-08 |
|---|---|---|---|
% Not reporting |
71% |
32% |
12% |
% Programs with Plans only |
17% |
31% |
24% |
% Programs with Data/Changes |
12% |
37% |
64% |
In the first year of reporting (2005-06), some programs obtained a waiver for the annual assessment report if they were going through program review or professional accreditation. This accounts for the majority of the undergraduate degrees shown as not reporting; for example, most undergraduate engineering degrees were undergoing review by ABET. Most graduate programs did not understand that they were required to do assessment of student learning in every degree program every year.
In subsequent years, the percentage of units not reporting has decreased substantially, and the percentage of units that have moved from the planning stage to the results stage has been steadily increasing. These successes are celebrated with an annual Assessment Poster Fair, where programs from each college are invited to present their assessment results in poster form to the university community.
In sum, each academic unit now completes one annual report, which summarizes assessment activities undertaken over the past year and, where an MOU exists, progress on the recommendations of program review. This combined reporting process has been well received by the campus, and it documents a substantial, campus-wide increase in CSULB’s capacity to use evidence to improve program effectiveness. (Criteria 2.7, 2.10)
The assessment timetable and Program Review Policy can be found on the Program Assessment and Review Council web page at:
http://www.csulb.edu/divisions/aa/grad_undergrad/senate/committees/assessment/
CSULB has required program-level assessment of General Education (GE) since at least 1979. For many years, however, program-level assessment of GE lagged in practice. A variety of small-scale audits of GE were carried out to achieve specific purposes (e.g., ensuring a sufficient number of sections per course given student demand; examining enrollment trends to avoid bottleneck courses; ensuring that students take prerequisites). No overall learning outcomes were articulated for GE, and no comprehensive assessment plan was developed or implemented.
Following new guidelines adopted by the CSU Chancellor’s Office in the 1990s, a substantial revision of GE policy went into effect in 2000 at CSULB. For the next five years, the General Education Governing Committee (GEGC) was largely occupied with reviews of individual GE courses to ensure that each one complied with the revised policy (a large task, given that the number of GE courses grew to exceed 650). However, with no overarching program level learning outcomes for GE, the criteria for certifying courses for GE credit were not clear, and approval seemed at times to be influenced largely by the membership at the time of review.
Beginning in 2005, the first waves of courses certified for GE were due for their five-year review for recertification. The intent of the recertification process was to examine the evidence from course-level assessment of student learning. However, no workshops or tutorials had been provided to instructors of GE courses about expectations for assessment activities; the guidelines and procedures for recertification were perceived to be vague; and the GEGC was unable to articulate what was expected or to provide guidance in assessment procedures. In sum, most GE course recertification documents did not report on assessment.
During this period there was considerable turnover in responsibility for the GE program. Budget cuts and changes in personnel resulted in scant administrative guidance; during one 18-month period, there was no GE coordinator at all. However, a noticeable shift occurred when a tenured faculty member with expertise in assessment was appointed in 2005-2006 to a half-time position as General Education coordinator. Many initiatives were begun that year to facilitate, and ultimately to institutionalize, comprehensive assessment for the GE Program.
Reacting to the deficiencies of the intended recertification process, the GEGC began working on revising the GE policy in 2005-06. This latest revision of the GE policy, passed by the Academic Senate in fall 2007 and effective in fall 2008, clearly delineates a set of Student Learning Outcomes (SLOs) for the entire GE program. Learning objectives were derived from AAC&U's L.E.A.P. (Liberal Education and America's Promise) initiative which describes what GE should ideally achieve in the 21st century. All courses certified for GE credit will now be expected to show how each particular course is specifically contributing to the greater mission of general education. New recertification procedures were also created. A new, required Standard Course Outline (SCO) Template provided detailed instructions to faculty about what information should be included in an SCO to ensure curricular integrity, and a new Review Worksheet delineated a common rubric for evaluation of courses seeking GE recertification. The new procedures are being introduced to the campus community through a series of events begun during spring 2008. A variety of resources and tools are being developed to facilitate the success of the new procedures, including a GE web site, self-paced web-based faculty tutorials on assessment, and faculty professional development workshops.
In 2006-07, assessment models from GE programs across the country were considered with the aim of developing new, broader initiatives for program-level assessment of GE. In a radical departure from the past, individual course instructors would no longer be expected to plan and carry out comprehensive assessment of student learning in each course for the purposes GE. Rather, the process would be carried out by a small group of experts in the learning outcomes being assessed. This approach, called S.A.G.E. (Student Achievement in General Education), constitutes a manageable plan for program-level assessment of the large volume of courses that make up the GE program.
This approach will estimate the "value added" by the GE curriculum to students' mastery of each of the overall learning objectives. The plan employs a pre-test, post-test experimental design, and the GE Coordinator and GEGC will oversee the process. Simple random samples will be selected of both native and transfer students in both beginner and advanced GE courses. All students in these samples will be asked to perform a task (e.g., writing a short essay) to demonstrate their knowledge and ability related to a particular learning objective. Data will be analyzed to determine whether, and by how much, the advanced students outperform the beginner students on each learning objective, the dimensions of these complex skills on which the advanced students show greater achievement, and how the data can be used to better inform the design of the GE curriculum.
The data collected through the S.A.G.E. process will also serve as assessment of student learning in individual GE courses. Faculty teaching GE courses will be provided with the S.A.G.E. data for their courses as well as specific guidance on using the data to create sustained, evidence-based improvement in the educational effectiveness of their courses. This approach obviates the need to train some 600+ instructors in assessment of student learning; and assessment will be carried out by experts. As a result, instructor efforts will focus on using the evidence to "close the loop" for their specific GE course.
The S.A.G.E. process was piloted in 2006-2007, with consultation of faculty experts across campus. Two learning outcomes were selected to test this model of assessment, and faculty experts teaching in each of those two areas were brought together to develop a set of measurable student outcomes and to establish performance benchmarks for each of those two learning outcomes. To evaluate the robustness of the assessment plan itself, two of the more "ineffable" learning outcomes were selected: global competencies and U.S. intercultural competencies. Once the measures and benchmarks were developed, a writing prompt was created to collect student performance data. In fall 2007, pilot data for one of the learning outcomes (global competencies) was collected from a student sample. Preliminary analysis of the data demonstrates that S.A.G.E. shows promise as a process for measuring student achievement on learning outcomes across the GE program.
At this point, a culture shift in GE is occurring on the CSULB campus. This comprehensive plan for outcomes-based accountability represents a significant new direction for GE that will be refined and expanded for several years to come. GE will now participate in program review on a regular schedule and submit yearly annual reports on assessment, as do all other academic programs. Additionally, the new procedures will substantially strengthen the capacity of the GEGC to promote educational effectiveness. GEGC members will now be informed experts on general education and its learning goals. The GEGC will be able to gather data that puts a finger on the pulse of what is happening in GE as an entire program.
CSULB seeks to provide coherence to its GE curriculum by using the GE program learning objectives to anchor both program-level assessment (through S.A.G.E.) and course-level assessment (through recertification). These efforts are expected to facilitate, for the first time in the history of our campus, a common understanding among faculty and students about what general education is intended to add to the development of well-educated individuals.
CSULB has developed policies, structures and processes that hold the potential to assess what students are learning in the GE program, and to use that information for program improvement. In the Educational Effectiveness Review (EER), we will document how assessment information has been used to enhance teaching and learning.
What structures, policies, processes, technological resources, faculty and staff qualifications and support, and student support services are in place for our distance and off-campus programs?
The term “distance education” generally applies to instruction that is delivered to students in distant locations, most commonly through online, web-based, or other non-traditional instructional modes. While distance education at CSULB is not contingent upon a particular funding source, state-supported programs have, with few exceptions, chosen to focus on traditional, face-to-face instruction. In contrast, programs offered through University College and Extension Services (UCES) that are funded on a self-support basis have been more willing to serve off-campus constituencies and utilize non-traditional instructional delivery modes. Thus, distance education at CSULB is almost entirely offered through UCES on a self-support basis, and the term “distance education” is used to describe self-support programs delivered in traditional, non-traditional, and hybrid modes.
By the end of the 1990s, CSULB offered a few self-support distance education programs through UCES, nearly all of them master’s degrees in professional, accredited disciplines (e.g., MSW, MPA). In response to constricting budgets in the early years of the present decade, the CSU Chancellor’s Office endorsed self-support distance education degree programs, and CSULB academic programs were encouraged to explore this option to augment shrinking state resources. The flexible schedules, convenient access, and sophisticated technologies of these programs proved attractive to students, and enrollments increased rapidly between 2000 and 2005, particularly in programs whose enrollments were not constrained by professional or clinical accreditation requirements.
The rapid growth in enrollments prompted CSULB to examine the application of academic policies, structures and practices to self-support programs. CSU Executive Order 802 (1/21/02), which specifies the procedures to be followed in establishing self-support programs, requires that academic standards for self-support programs be identical to those of comparable state-supported programs. Similarly, CSULB does not have separate institutional policies that apply exclusively to self-support programs.2
Rather, all policies applicable to state-support programs are also applicable to self-support programs. These include curriculum and educational policies, personnel policies, program review and assessment policies, and so forth.
The organizational structures charged with responsibility for upholding academic standards for state-supported instruction are also responsible for overseeing self-support programs. In practice, however, the process for setting up, amending, or discontinuing self-support programs had devolved to the college level, and the academic unit had been responsible for ensuring that all applicable campus policies are followed. As a result of this decentralization, some campus policies were not uniformly applied to self-support programs. Instructor evaluation and program review were of particular concern. (Criterion 2.11)
In response to these concerns, two campus groups were convened to address this challenge: a Provost’s ad hoc committee in 2005 and an Academic Senate ad hoc committee in 2006. The College of Health and Human Services, with the largest number of distance education programs, also convened an ad hoc committee in 2006 to address these concerns.
A subsequent special session of academic deans endorsed the relevance of self-support programs and affirmed that self-support and distance education programs are subject to the same quality assurance mechanisms and practices as on-campus programs with regard to faculty deployment, grading standards, course content, and assessment of student learning. This commitment is backed by a robust new program review policy; a revised statement of responsibilities for assessment of student learning; and a new Program Assessment and Review Council charged with the implementation and evaluation of these activities for self-support academic programs as well as those provided on campus. In the EER, results of program review and student learning outcomes assessment will be analyzed with attention to distance education programs.
