California Learning Resource Network

clrn.org

Inputs Matter

What is CLRN?

California Learning Resource Network State-funded Education Technology Service

Initial charter: Review supplemental electronic learning resources

Open Educational Resources

6000+ free and commercial-free resources

Aligned to CCSS and other CA standards

Governor Schwarzenegger’s Digital Textbook Initiative

California Digital Textbook Initiative

Three Phases: Free or Open Source

High School Math, Science, & History

Review for Content Standards Only

California Legislation

AB 1398, relating to the use of textbook funds

Redefines “technology-based materials” to include electronic equipment required to use them

California Legislation

SB 247 relating to high school textbook purchases

Textbook funds may be used to purchase electronic versions

Districts must ensure all students have access at home & school

Recent CA Legislation

SB 185 / AB 133

Requires publishers to provide textbooks in digital format

Three Digital Textbook Paths

Self-contained digital Flat: Epub, Kindle, PDF Digital with interactive components

Online, subscription-based Some digital or interactive components

Online Courses

What is CLRN?

Online Course ReviewsSix subject areas

Standards Alignment Common Core State Standards California’s other standards Next Generation Science Standards

iNACOL Standards for Quality Online Courses

Publisher Eight Step Entry

Step 2: Content Standards

Standards CorrelationAlgebra I CCSS

cv cv cv

Step 3: Course Standards

Step 4: Instruction

Type

Length

Accreditation

Step 5: Technology

Step 6: Features

Step 7: Abstract

CLRN Review Sites

Six, subject-specific sites

County office based

Managed by COE content expert

20-25 active teachers

Meet monthly during school year

Reviewer Training & Norming

CLRN Review Process

Social Content Criteria

Online Course Standards

CLRN OCR Training Videos

OCR Retraining & Norming

CLRN Course Review Process

Nine-step process

1. Team assignment

2. Content (CCSS) Standards review Develop, Practice, Assessment

3. Minimum Requirements Reading levels, spelling/grammar, content accuracy, etc.

4. Social Content Review 13 areas, including male/female roles, ethnic/cultural

groups, & advertising.

CLRN Review Process

5. Online Standards

6. Features

7. Profile

8. Abstract

Step 4: Social Content

Male/Female Roles

Ethnic/Cultural Groups

Older Persons/Aging Process

Disabled Persons

Religion

Brand Names/Advertising

iNACOL Standards for Quality Online Courses

Online courses participate in the teaching/learning process

iNACOL Course Standards

Content

Instructional Design

Student Assessment

Technology

Course Evaluation and Support

Step 5: Course StandardsReviewers vet 24 of the 52 OCR standards

Content

Content depth and breadth Information literacy skills Learning resources and materials Communication process between

teachers, parents, and students Content accuracy and bias

Instructional Design

Course design and organization

Meaningful and authentic learning experiences

Multiple learning paths for students to master the content

Higher-order thinking skills

Instructor-student and student-student interactions; and supplemental tools and resources.

OCR Standards/Considerations

B5. The course provides opportunities for students to engage in higher-order thinking, critical reasoning activities and thinking in increasingly complex ways. ★

Assignments, activities, and assessments provide opportunities for student to elevate their thinking beyond knowledge and comprehension into the realm of analyzing situations, synthesizing information, or evaluating an argument. …Opportunities for group work, decision‐making, and finding patterns should also be included in the course.

Student Assessment

Alignment between the course goals and activities and its assessment strategies

Insure that there are adequate and appropriate methods to assess students

Assure that students are constantly aware of their progress.

Technology

Course architecture

User interface

Accessibility

Interoperability

Course Evaluation and Support

Evaluating course effectiveness

Accreditation

Teacher and Student preparation and support

CLRN Central Review

Validation & Norming Work the course

Review/update all 52 course standards

Standardize review comments

Notes inform reviewer retraining

CLRN Central Review

Editing & Proofreading Review, modify, and standardize content standards

rating & comments

Publisher Feedback Seven-day window New evidence required for Re-Review One Re-Review permitted per course.

Publishing Reviews valid for three years

Texas H.B 4294

eTextbook publishers may submit updated content for review

Districts/schools may select a subscription-based electronic textbook

Re-Review Policy

For Updated Courses Updated, not new (New=>30% new content)

New content and/or course standards.Strengthen alignment to the content

standards and/or the course standards. Publishers provide specific evidence.

Once per year

CLRN HOme

List View

Not Met standards

CLRN-Certified

University of California PartnershipA-G Approved Courses

CLRN-Certified 80% content standards 80% online course standards

15 power standards

OCR Standards & Considerations

CLRN Certified Information

U.C. Online Learning Home

What steps does your school/district take to validate that the rigor and quality of online or blended learning courses meets or exceeds that of “traditional” courses taught in a brick and mortar classroom?

Answers and Questions

We examine the course outlines with teachers and administration.

Matching up the state standards with course, requiring minimums for passing scores

None required

More Answers and Questions

Data supplied by curriculum provider.

Student input on rigor compared to last course or class taken.

Ensure alignment to state standards – we need to be doing more

What factors did you consider when selecting courseware?

None

Selected CLRN Certified courses.

Colleague recommendation

Vendor demonstration

Data supplied by provider

Examined course outlines

U.C. A-G list

Compared to content standards

Price

0 5 10 15 20 25 30 35 40 45

77%

60%

60%

55%

42%

42%

The Good News

CLRN Reviews inform course improvement

Re-Reviews

Content Re-reviews

Additional content

New activities

Additional components to meet course standards Syllabi, policies, resources, etc.

Analysis of the 460 published

63% certified Up from 46%

20% teach < 80% content standards Range from 4% met to 78% met

12% only missing captions or transcripts

Certification by Subject

ELA:52%

History: 64% Up from 27%

Math: 55% Up from 47%

Science: 58% Up from 28%

World Languages: 89%

Accessibility

You’re only a lawsuit away from making your courses accessible

DOJ Story

D10: Course materials and activities are designed to provide appropriate access to all students…

Closed-Captions/Transcripts

Edgenuity (e2020)

BYU

Compass Learning

K12, Inc.

(Word has gotten out)

Why Inputs Matter

Resubmit an Assignment Story

Ed Code

Ed Code 49011 states that school districts and schools shall not establish a two-tier educational system …through payment of a fee or purchase of additional supplies that the school district or school does not provide; and

What is not a great online course

Class 1: Watch lecture. Complete worksheets. Take a multiple choice test. Repeat

Class 2: Read. Take a multiple choice test. Repeat

Class 3: Read. Print and take quiz. Grade your quiz. Take multiple choice test. Repeat

Class 4: Read a physical book. Take quiz from a physical book. Take part in online discussion. Repeat

Moving a textbook online is not a foundation for a great course.

What is not a great online lesson.

“Writing an introduction” slide show

Vocabulary exercises. Students then:

READ 12 textbook pages about verbs

READ 13 pages of textbook reading about verb agreement

Not a great online lesson, continued

READ 10 pages of textbook reading about verb tense

READ 6 textbook pages about “verbs made easy”

READ a one-page textbook page about writing a first draft, and

READ two textbook pages about simple narratives.

What should we expect from great online courses?

High-quality course

Stimulating lectures

Just-in-time reading

Active learning

Engaging activities that go beyond knowledge and comprehension

Writing

Students create, evaluate, and analyze.

Variety of assessment types

Make use of the medium

Rich mediaLecture clips, video demonstrations and

clips, variety of multi-media, simulations, …

Ease of useLMS helps inform instruction.

Students/teachers always know where they are.

California Learning Resource Network

clrn.org

Inputs Matter