1	Assessing Information Literacy using Technology By Dr. Lesley Farmer, CSULB
2	What is Information Literacy? Definitions: What is information – and literacy? Overlap: critical thinking, technology competency, problem-solving, research skills… Other terms: ICT Levels: functional, academic… The ability to access, evaluate, use, manage, and communicate information effectively and purposefully
3	Assessment Process: A Cycle of Inquiry Observe the world around you! Define the need for assessment Plan the assessment process Collect the data Analyze the data Report the results Plan the intervention
4	Educational Info Lit Issues How migration impacts IL (e.g., different educational practices and library resources) Links with visual literacy, aural literacy, media literacy, numeracy Who is responsible for teaching IL? Scope and sequence of IL in K-16 education Should IL be a separate curriculum or integrated within other curricula? Articulation of IL across grades/levels IL as a lifelong skill IL instruction in different types of libraries IL instruction collaboration between librarians and course teachers Info seeking behaviors by different age groups, in different countries Info seeking behavior for school vs. workplace vs. personal inquiry Formal and informal IL Information seeking behavior in physical space vs. cyberspace
5	Assessment System Input: staff, $, materials, facility Process: staff activity, equipment activity Output: services used, products used Outcome/impact: change in user Environment: university context
6	Assessment Questions WHO will assess? Who is being assessed? HOW will assessment be done? WHERE will assessment be done: in library, at school, at home? WHEN will assessment be done: at what stage; what timeframe? WHY is assessment being done: to describe, to improve, to show impact? What are the standards for IL? Can they be standardized across countries? What policy statements exist about IL?
7	AAHE Principles of Assessment Measure what you value. Use a variety of methods (POV, time). Have clear, specific ed goals. Consider both process and products. Assess on ongoing basis. Foster broad involvement. Focus on the use of assessment. Consider assessment as integral to big picture. Consider assessment as an accountability issue.
8	Types of Measures Records analysis: reference-fill %, reference questions (#, type) Observation: processes Interview: reference interview, planning partner, student response Questionnaire: reference service Test: reference skills
9	Factors in Choosing Measurement Tool Cost: forms, development, analysis Availability Size: sample population Scope: what is evaluated; what is impact Skills: to measure (I.e., interview), analyze Legalities: copyright, confidentiality Culture: school norms, cooperation
10	Assessment Instrument Issues Observation: not systematic Convenience vs. authenticity Age-appropriate measures Self-assessment limitations Complexity of information literacy
11	Current Trends in Assessment Instruments preference for performance assessment transition from tool-based to problem-based assessment transition from library-centric to content-centric assessment transition from multiple choice standardized tests to portfolios of evidence tension between high stakes testing and ongoing assessment acknowledgement of the affective domain’s role in information literacy assessment collaborative assessment between teacher librarian and classroom teacher use of simulations incorporation of technology embedded assessment within tutorials to give instant feedback and targeted remediation investigation of system-wide conditions for information literacy learning
12	Assessing Technology Literacies What is technology: digital, electric? What is being assessed: tools/operations, strategies, use of resources, communication, ethics? Self-assessment – and its limitations Tests – and their limitations Performance: skills vs. strategy, one-shot vs. process/progress, tool limitations E-portfolios: what are you measuring?
13	Visual Literacy Assessment Issues What is being assessed? Tools = media manipulation Elements = universal art Principles = composition Esthetics = cultural context Assessors need training!
14	Media Literacy Assessment Differentiation from visual literacy Universal aspects: goals, the language of each mass medium, “hooks” Cultural aspects: how language is used, cultural experiences and expectations What is being assessed: comprehension, criticism, response
15	Impact of Technology Collapse of time and space Worldwide access Communications Multimedia Interaction Management Repurposing Changes instruction
16	Matching Technology Tools with Assessment What are the critical features of each tool? What are the desired outcomes? What is process-centered, and what is product-centered? What are the parameters: time, resources, expertise…? Who is the learner?
17	Examples of Technology Incorporation Publishing for templates and repurposing Sortable databases Spreadsheets for analysis PDAs for quick observation and feedback Telecommunications for collaborative work – joint view of documents Video/images for scenarios Take advantage of asynchronous, cross-platform, secure environment
18	A Word about Learners Pre-engagement: background knowledge, capabilities, origin of need, development Contact: attention, motivation, assumptions Comprehension: decode, vocab, concepts, context Evaluation: agree/reject/change Use: relate, reform, synthesize, organize, apply Note: Levels of engagement Note: cognitive, affective, kinesthetic
19	Research Process Rubrics Australia: http://alia.org.au/advocacy/literacy.kit.pdf Wisconsin: http://www.waunakee.k12.wi.us/DPI_Standards/matrix.htm Kansas: http://www.ksde.org/outcomes/libstd52001.pdf Minnesota: http://www.isd77.k12.mn.us/resources/infocurr/benchmarks.pdf
20	Accessing Information Factors: location, keywords, info organization Recall vs. evaluation Janssen (2002) 4-factor scale for assessing the value of information literacy questions: 1) the complexity of the search needed to answer the question, 2) interest, 3) fact / explanation, 3) knowledge advancement. Asking questions: Copeland (2005) conditions for questioning (http://www.stenhouse.com/pdfs/0394ch05.pdf). Richard Paul (1990) Socratic questions (http://www-ed.fnal.gov/trc/tutorial/taxonomy.html)
21	Evaluating Information Assumptions: context, literal reading, dealing with conflicting info Issue of non pre-selected info no one source is the best for all learners; they need to can evaluate their own stage of knowledge and ability to use information so they can choose the most appropriate information for themselves for a specific task in a specific context. Chico uses authentic tasks (http://dewey.chs.chico.k12.ca.us/info-lit.html).
22	Communicating Information Socratic circles Arizona: http://www.fulton.asu.edu/~fcae/Insturments/Communication%20Instrument/Communicationinstrument.htm Canada: coding scheme that captures assessment data about affective, cohesive and interactive responses to thematic content, specifically in asynchronous learning environments (http://cade.athabascau.ca/vol14.2/rourke_et_al.html).
23	Metacognitive Skills Stages: Be aware of one's own cognitive ability, be able to explain tasks that improve performance, be able to suggest alternative strategies Study process: (teaching.polyu.edu.hk/datafiles/R75a.doc) Self-perceptions: (http://www.education.gov.ab.ca/k_12/curriculum/bySubject/focusoninquiry.pdf Cheuk (Singapore) developed an indicator-based scheme for assessing information behavior (including physical, cognitive, and affective domains)
24	Appreciation Skills Need to understand the language/protocol of the medium, understand the content, and critically compare it with other works esthetically Kansas: appreciation elements across the creative arts (http://www.ksde.org/outcomes/rubricfinal.pdf) Affective reading: (http://ksumail.kennesaw.edu/~jcope/WorkshopHandouts/2003Handouts.htm) Australian TV/film: (http://www.actf.com.au/learning_centre/school_resources/teaching_kits/btv/units/btv_lp.htm
25	Online Tutorials: Benefits Just-in-time instruction/learning Independent of time and place Supplements/complements info lit instruction Self-paced / self-control Instant feedback Easily mounted Easily repurposed (especially if designed in modules) Opportunity for interaction Opportunity for data collection Flexible instruction: group or individuals ANYWHERE
26	Online Tutorials: Design Consider student outcomes AND ease of use Best for skills Collaborate Sections: intro, info, task, assessment Try to include branching Think visual Remember ADA and tech limitations Pilot test
27	Online Tutorials: Examples http://tutorials.sjlibrary.org/ http://www.marcopolo-education.org/ http://www.merlot.org http://www.lib.umb.edu/webtutorial/index.html http://www.rlc.dcccd.edu/lrc/ppt.htm http://oncampus.richmond.edu/academics/library/list/ http://www.lib.berkeley.edu/TeachingLib/BIResources.html http://www.library.cornell.edu/olinuris/ref/research/tutorial.html
28	Online Tutorials: Assessment Gather data about process and product Analyze data: content, format, delivery Make modifications: instruction, tutorial, learning activities
29	Levels of Assessment: Library UK scenarios: http://www.teachernet.gov.uk/teachingandlearning/resourcematerials/schoollibraries/?3300545056b4385-a1d7191d-3302-4529-8db5-17eb5900db37) Quebec: (http://www.qfhsa.org/pdf/library_questionnaire_nov04.pdf) Evaluating library program/librarian: (http://www.sldirectory.com/libsf/resf/evaluate.html#forms)
30	Levels of Assessment: Classroom Canada’s ICT: (http://www.education.gov.ab.ca/k_12/curriculum/bySubject/focusoninquiry.pdf Switzerland :beliefs and practices related to interactive learning systems (http://www.scil.ch/seufert/docs/cultural-perspectives.pdf) New Zealand: (http://www.tki.org.nz/r/assessment/atol_online/self_review_e.php)
31	Site-level Info Lit Audit Identify resources and services that support information literacy Identify gaps and dysfunctional use of information literacy-related resources Identify needs of key information literacy stakeholders Map information flow within the system and between the system and its surrounding environment Link information literacy and governance issues Identify existing and needed information literacy-related policies (henczel, 2001) The New South Wales (Australia): (http://www.oit.nsw.gov.au/Guidelines/4.3.12.g-IM-Audit.asp)
32	National/International Assessments ETS: (http://www.ets.org/research/ictliteracy) UNESCO Bangkok: (http://www.unescobkk.org/index.php?id=662) PISA framework: (http://www.ictliteracy.info/rf.pdf/PISA%20framework.ppt)
33	Some Preliminary Findings Students self-assess inaccurately Students enter unprepared Students need explicit training Students undervalue/underuse databases Site focus on information literacy is uneven Assessment administration, analysis, and follow-up can be difficult
34	Use of Findings Leverage findings to plan strategically Acquire resources Coordinate levels/areas of curriculum Diagnose/place incoming students Incorporate into curriculum Provide domain-specific training Collaborate with teachers in terms of resources and tutorials Coordinate with tech services and course packaging
35	Webliography http://www.csulb.edu/~lfarmer lfarmer@csulb.edu