Standard 2: Assessment System and Unit Evaluation

2c. Use of Data for Program Improvement

2c.1 In what ways does the unit regularly and systematically use data to evaluate the efficacy of and initiate changes to its courses, programs, and clinical experiences?

2c.2 What data-driven changes have occurred over the past three years?

2c.3 What access do faculty members have to candidate assessment data and/or data systems?

2c.4 How are assessment data shared with candidates, faculty, and other stakeholders to help them reflect on and improve their performance and programs?

2c.5 Tables, figures, and a list of links to key exhibits related to the use of data for program improvement may be attached here.

2c.1.   In what ways does the unit regularly and systematically use data to evaluate the efficacy of and initiate changes to its courses, programs, and clinical experiences?

Interpretation and use of data takes place at both the program and unit levels.

In December 2007 the Assessment Committee directed all programs to:

• Analyze data on a minimum of two learning outcomes each year and "take actions relevant to their findings and interpretations."
• Analyze and act on all learning outcomes at least once within a three-year period.

Programs are free to review some or all of their outcome data more frequently, and some do, but the above are baseline expectations for program practice. The committee felt these expectations made the process manageable and allowed for a large enough sample size on specific outcomes that faculty would feel confident in the conclusions they were drawing.

The Assessment Committee also directed programs to prepare annual assessment reports based on their annual review of identified learning outcome and other data. Program faculty are expected to meet at least once per year to review data on selected learning outcomes and agree on next steps for program improvement. Prior to their data discussions, programs receive data reports from the Assessment Office. A Data Discussion Guide exists to guide these conversations, and these data discussions form the basis for the program's Annual Report.

The college has adopted a two-part reporting calendar. Initial and Advanced credential programs (i.e., Commission on Teacher Credentialing-accredited programs) are required to file an Annual Report each November/December and therefore generally meet in the fall to review data. Degree-only programs (not CTC-accredited) submit reports each May, generally meeting in the spring for data discussions.

This structure was chosen in large part because the CTC requires submission of a Biennial Report in December of every-other year. In those years when a Biennial Report is due (e.g., 2009), programs with a credential component submit this document in lieu of the college-required Annual Report. The college's Annual Report was developed based on the CTC Biennial Report template, in part to simplify reporting for programs. The Annual Report is linked closely to the Data Discussion Guide in an effort to make the report writing process as easy as possible.

To support this process, the Assessment Office offered workshops on how to prepare an Annual Report in spring 2009. This grew of out a desire by the Assessment Committee to help faculty recognize how data findings could be turned into actions and to make the process as clear and straightforward as possible. The workshops (PowerPoint slides and attendance sheets are available as exhibits) were attended by 26 faculty and contributed to stronger reports submitted in May 2009 than those submitted in fall 2008. In addition, the fall 2009 College convocation entitled "Access and Achievement: Using Data to Support Student Success", focused on the topic of data use for the purpose of program improvement, and provided a forum for faculty to engage in cross-program discussions on how to better use assessment data to enhance their programs.

Aggregated data are reviewed at the unit level by the Assessment Committee. The committee did an initial review of 2007-08 data in spring 2009 and directed the Assessment Office to make revisions and investigate several questions. At its September 11th 2009 meeting, the committee reviewed the revised 2007-08 data as well as the 2008-09 aggregated data. This step will put the Assessment Committee on its planned schedule to review data from the prior academic year early each fall semester.

^ back to top

2c.2.   What data-driven changes have occurred over the past three years?

To date, much of the benefit from 2007-08 revisions to the college's UAS have come in the form of changes in practice related to assessment. For example, as a result of developing curriculum maps, faculty have recognized that curricular alignment is something that needs to be addressed in some programs. Others have used the creation of rubrics as occasion for a conversation about performance expectations and standards for candidates.

We are pleased to see that programs and the unit as a whole are making changes based on the assessment process and the data they are collecting. Examples include:

• The Early Childhood Education program began (Summer 2009) a mixer to introduce new candidates to returning candidates and graduates. This is based on candidate feedback that they were unclear regarding program expectations, despite the fact that these expectations were covered at orientation and in individual advising sessions.
• Faculty in the Dual Language Development program are stepping up efforts to incorporate elements of literature reviews earlier in the program course sequence after faculty found candidates struggled on this portion of a signature assignment.
• The Multiple Subject program faculty are conducting a review of course outlines in an effort to ensure that candidates have a better understanding of the integration of assessment, planning, and learning when working with K-12 students. They are also developing a review module on assessment to deliver to candidates prior to the start of student teaching.
• The Educational Specialist Level I faculty felt candidates received a surprisingly low score on a particular signature assignment in the 07-08 data. This was due in part to a rubric that is being redesigned to track more closely to the learning outcome. However, faculty also recognize that the format of the assignment (a written exam) was different than ones their candidates are used to (projects), so they are looking at alternative formats for the assignment. This example, in particular, illustrates the connection between program and unit-level data review, since the Assessment Committee identified the average score on this outcome as an area of concern as did program faculty.
• The Educational Technology program is undertaking several efforts, including consideration of how to improve candidates' argumentation, mechanics, and APA skills, after reaching consensus that writing is an important need for their candidates.
• The Reading program has used its curriculum map to think carefully about what courses it would not offer in the wake of budget cuts.
• Concern over application numbers in Educational Psychology is prompting an effort to develop recruitment strategies starting in fall 2009 and working on curriculum revisions to enhance the clarity of the program focus.
• The Credential Office instituted Saturday service hours when a survey of candidates revealed some were not able to access the office easily on weekdays.
• As part of the college's Graduate Culture Initiative (flyer 08, flyer 09), the unit is offering advanced writing workshops and writing tutor support. This decision was informed by survey data identifying this as an area of need.
• Review of five-year trends in admission and yield data has informed enrollment targets for academic programs in the coming year. These data were also considered closely in making decisions regarding opening new tenure-track faculty lines.
• In response to an emerging need for our advanced programs to work with students in a supervised clinical setting, the college started Math and Reading Clinics housed in the Educational Psychology Clinic, which allow for supervised clinical field experiences for our candidates.

^ back to top

2c.3.   What access do faculty members have to candidate assessment data and/or data systems?

To date, faculty have had limited access to candidate assessment data. This access is restricted to the data as presented in various charts and graphs summarizing candidate data at the program level. These data are prepared by the Assessment Office and shared with program faculty through program coordinators at annual data meetings.

As the college builds its COSS system and transitions this database to a new generation of software, we will be enhancing faculty access to the system. Most immediately, we plan to be able to have faculty enter candidate performance data through a web interface. Following that, we plan to build-out the system in several ways, including:

• Giving faculty access to candidates in their program so that the faculty can look up status and performance of individual candidates
• Designing pre-formatted summaries of candidate performance data so that faculty can get real time data summaries at the program level
• Creating a system of "flags" or triggers that will automatically generate reports for faculty if a candidate crosses a particular threshold and might be considered at-risk of not succeeding.

^ back to top

2c.4.   How are assessment data shared with candidates, faculty, and other stakeholders to help them reflect on and improve their performance and programs?

One of the key benefits of the refined UAS has been the opportunity to more systematically involve the unit's part-time faculty, most of whom are practitioners in their fields. In spring 2009, for example, the Dean provided stipends for part-time faculty who attended data discussions. The result was a significant and valuable involvement on the part of our part-time colleagues.

In addition, programs such as the Dual Language Development and School Psychology programs, have involved external stakeholders or candidates in reviewing assessment data. However, this particular practice is, as yet, uneven in the unit. To some degree, this is likely because the program faculty themselves have only gone through one full cycle with the revised UAS and thus are just starting to appreciate the full process themselves.

One of the key goals for the 2009-10 is to build capacity within the unit to engage key stakeholders in the full assessment process. This includes not just sharing and reflecting on data, but making program SLOs transparent from the beginning of a candidate's program of study, helping stakeholders see how the curriculum relates to the learning outcomes, and engaging others in reviewing, interpreting and crafting action plans around candidate performance data.

Toward this end, several efforts have taken or will take place:

• The college's fall convocation focused on ways to put data to use for program improvement purposes. One of the speakers was a faculty member who has experience engaging external stakeholders in data discussions.
• Department chairs will be working with program coordinators to ensure that their Fall data discussion meetings include members of their advisory boards and/or other key stakeholders.
• The Assessment Office plans to develop a resource (likely a handout or tool) that faculty can access for ideas on how to go about engaging others.

^ back to top

2c.5.   Tables, figures, and a list of links to key exhibits related to the use of data for program improvement may be attached here.

N/A

^ back to top