CSU Long Beach College of Education Accreditation
2a.3 How is the unit assessment system evaluated?
The Unit Assessment System (UAS) is built around the core standards and concepts contained in the college Conceptual Framework, state guidelines, and NCATE and professional standards. Assessment plans for each program, together with an annual reporting cycle, form the foundation of the UAS and ensure that data are collected and reported regularly. A common program improvement process guides the collection, interpretation, and use of data in the unit and its programs. In fall 2007, the Assessment Committee adopted an assessment plan template for use by all programs in the college. The template provides a common format through which our diverse array of programs can articulate learning outcomes and also serves as a management tool for collecting and analyzing data.
Each program learning outcome is mapped via the assessment plan template to the Conceptual Framework and to state, national, and NCATE standards. Click here for individual program assessment plans. The mapping of student learning outcomes to standards through assessment plans allows programs to systematically examine candidate outcome data along a variety of dimensions. Moreover, it also provides a framework through which the college aggregates and analyzes data at the unit level and uses findings to guide program improvement efforts.
For example, these data are presented relative to the NCATE standards in a table prepared annually by the Assessment Office. This table is used in various forms to report candidate data in Standard 1.
The Assessment Office prepares a similar table that presents program-level data aggregated around particular components of the college's Conceptual Framework. One key difference between this figure and the example above is that the latter includes all programs in the College of Education and Affiliated Programs, regardless of NCATE status.
The assessment plan template also reflects the three transition points adopted by the Assessment Committee in fall 2007. These transition points are:
In addition to the assessment plan template, the Assessment Committee approved a more succinct flow chart. While the flow chart does not provide the comprehensive detail of the full Assessment Plan, its more streamlined appearance allows for an easy review of the transition points and related signature assignments and provides a good snapshot view of each program.
The table titled Transition Point Assessments by Program displays the data sources (signature assignments, surveys, and other measures) programs use to collect data on candidate performance at the unit's transition points.
To date, evaluation has been ongoing and informal, taking place in several ways as the UAS has been refined since fall 2007. A formal evaluation is planned in 2010 after the college and its faculty have more experience with full implementation.
First, the Assessment Committee, which was created in fall 2007 and formalized as a part of Faculty Council in spring 2009, established UAS goals that were used to structure the system and will be used in the spring 2010 evaluation of the UAS by the Assessment Committee. The committee included the desire to have a system that is sufficiently flexible to meet the needs of our diverse array of programs, provides meaningful data relevant to the faculty and program improvement, and is driven and owned by faculty. Our community partners (representatives from a school district and a community college) played a key role in shaping this system. The committee has engaged in ongoing reflection during the development and implementation processes relative to the core principles outlined above.
Second, throughout 2008-09, the UAS was evaluated through meetings and conversations in multiple forums. Assessment and the system were regular agenda items at department and program coordinator meetings. This feedback was channeled through the chairs to the Dean's Administrative Leadership Council (ALC) to identify and address common concerns. In addition, in spring 2009, each department chair and the assessment coordinator met with each program coordinator to review their materials and to solicit ideas for how to better support their assessment efforts. The Dean's Community Advisory Committee met in fall 2008, received a presentation on the UAS, and provided input to further its refinement. They offered guidance on how to work with community college partners to ensure that students taking pre-requisite courses at those institutions fulfill the learning outcomes we have defined in the college. Individual programs also involve the professional community in the assessment process. For example, both the School Psychology and Dual Language Development programs have engaged their advisory board in reviewing candidate performance data.
This communication yielded multiple changes. For instance, experience and feedback through multiple venues revealed that faculty were often unsure of timelines and confused about what to turn in and where to do so. This resulted in the creation of published timelines and ongoing efforts to streamline and improve communication through department chairs as well as an Assessment Newsletter.
In addition, an Assessment Committee review of annual reports submitted in fall 2008 (the first under this new UAS) revealed that some faculty lacked a clear understanding of the goals of the reporting process and needed more guidance on the report format. The committee worked with the assessment coordinator to design more detailed documents to help faculty prepare their annual reports. It also asked the coordinator to offer workshops for faculty to review the Annual Report template and supporting documents, as well as the goals of the reporting process.
Third, a more formal evaluation of the UAS by the Assessment Committee is planned for spring 2010.
Finally, the associate dean for assessment and two faculty members have conducted research on the effectiveness of Spring 2008 assessment workshops and overall UAS implementation. This research has already resulted in on professional presentation (AERA 2009) and will lead to multiple publications.
In spring 2008, the Assessment Committee directed all programs to develop and use rubrics for collecting data on all signature assignments and culminating experiences starting in fall 2008. In addition, the committee "strongly encouraged" programs to employ analytic rubrics for signature assignments given prior to the Advancement to the Culminating Experience transition point, since these rubrics enable faculty to give more detailed feedback to candidates and provide for nuanced data to inform program improvement.
These rubrics were developed collaboratively by program faculty based upon a shared understanding of the program learning outcome(s) each rubric was expected to assess. This collaborative process was designed to ensure program ownership as well as to create a rubric that was comprehensive, appropriate, accurate, and free of bias. Faculty use these common signature assignments and rubrics across sections of the same course.
To ensure the fairness, accuracy, and consistency of the assessment methods, program faculty have begun calibrating around a given signature assignment and rubric. Starting in fall 2008, faculty began collecting "exemplars" of candidate work at various levels of performance (examples of a 4, 3, 2, 1 and 0 as appropriate). These exemplars serve two purposes. First, they provide concrete products for faculty to review at their annual assessment meetings, allowing for a more detailed discussion of candidates' strengths and weaknesses around a particular learning outcome. Second, they provide a means for faculty to calibrate their own scoring across sections of a course, further ensuring that assessment of students is both fair and rigorous. Electronic versions of these exemplars are stored on the college's Candidate and Operations Support System (COSS) database, linked to programs' learning outcomes for easy reference. The extent of the calibration is uneven given that our energies have been devoted toward refining our UAS and we anticipate a concerted effort to work through the departments to move forward on this task in 2009-10.
Finally, we will begin reliability studies on signature assignment data once we have developed a critical mass of data sufficient for this purpose. We expect these studies to begin after the spring 2010 term.
Individual programs, supported centrally by the college, track candidate admissions data (including applications, admits, and yield) as part of their assessment plans and process, using these data to inform their outreach and recruitment efforts. For most programs, these data are pre-filled in programs' annual reports by the Assessment Office.
In addition, exit and follow-up survey data provide indirect evidence of candidate performance and information on the candidate experience in the college, and are used to guide decision-making. The Initial preparation programs receive data annually from the CSU Center for Teacher Quality (CTQ). The CTQ administers an annual Exit Survey to all candidates completing an initial credential program (Education Specialist, Multiple Subject, Single Subject) in the CSU System. In addition, CTQ annually administers a survey of graduates (Education Specialist, Multiple Subject, Single Subject) of these three programs at the end of their first year of teaching and a survey of their supervisors (Education Specialist, Multiple Subject, Single Subject). These exit and follow-up surveys, when triangulated with signature assignment data, student teaching evaluation data, and CalTPA data (high-stakes, standardized assessments required to be eligible for a teaching credential in California), provide important insights into candidate performance and program efficacy.
For advanced programs and programs for other school personnel, the college created and piloted an Advanced Programs Exit Survey in spring 2009. Survey data were analyzed in summer 2009 at the college and individual program levels, reported at the September 2009 meeting of the Graduate Programs committee, and shared with program faculty. College-level data are also being reviewed by the Assessment Committee, Faculty Council, and other relevant bodies (e.g., Technology Committee) in fall 2009. In addition, the Assessment Office and School Psychology program are collaborating to pilot a follow-up survey of graduates in fall 2009. We expect to implement this across the college in spring 2010.
At the college level, multiple data are continuously used in enhancing the operations of the College through multiple means. For example, the Administrative Leadership Council and department chairs regularly review admissions and retention data (ALC Minutes: 2009-03-10, 2008-12-02, 2008-11-04), as well as enrollment trends and projections, as part of systematic enrollment management efforts. Also, the college's associate deans use enrollment management data and tools provided by Institutional Research and Assessment to work with departments around low course enrollments.
In addition, administrative units such as the Credential Center or SERVE office prepare annual reports, through which they engage in a process of analysis and interpretation that leads to action. For instance, the Credential Center has provided Saturday hours of service based on candidate feedback.
In June 2009, an ad hoc "operational indicators" group was formed. This group, composed of key staff and leaders from the major departments and colleges in the office, was charged by the Dean with working to "identify key operational indicators that the college and its programs can use to monitor and improve program operations, the source providing the data, and the level(s) of analysis of those indicators." This group will report to the college's Assessment Committee in fall 2009.
| Data | Level of Analysis |
| Application, admission and yield rate |
|
| Enrollment, projections, and retention |
|
| Exit Surveys for All Programs |
|
| Follow-up Surveys for Initial Programs (planned for Advanced Programs in Spring 2010) |
|
| Operational indicators (varies by office) |
|
Agendas & minutes for meetings during which assessment issues were discussed.
