1

Portfolios for Lifelong Competence Assessment

2

Assessment of Competence - More than

One ApproachTraditional (primarily external scrutiny)Emerging (more internal, self-directed)Research (educationally-driven)

3

Tests standardized where possible to maximize reliability (e.g. simulations - MCQ/ virtual, SP)Narrowly focused on knowledge and reasoningHigh stakes, summativeMay focus on individuals (“assessment”) or programs (“evaluation”)All about minimal competencePreferred approach of regulators (viz. societal protection)

Assessment of Competence - The

Traditional Approach

4

Assessment of Competence - The

Emerging Approach Tests tailored to the actual practice and learning needs of the individualAssessment beyond knowledge (e.g. includes skills and behaviors)Low stakes, formative, abundant feedbackUsually focuses on the individual All about driving continuous improvementImpelled by innate professionalism (internal locus of accountability), not just external scrutiny

5

Assessment of Competence - The Research Approach

Seeking better pedagogies, best practicesBroadening base of assessment, shaping learningFinding effective balance between traditional and emerging approachesFocus may be on individual, but more often programmatic Seeking balance between baseline competence and continuing improvementPrimarily the realm of professional educatorsDriven by need to define effective educational approaches

6

Availability of Data Sources and

Methodologies

For the Traditional approach - GoodFor Emerging & Research approaches - Poor

7

Needed: Access to Many New Data Sets - Some

ExamplesWork learning experience - scope of clinical practice and changes over time Clinical performance - processes of care, outcomes, managementBehaviors and skillsAchievement of competency milestonesLongitudinal research data set

8

Comes the Portfolio aka eFolio - Definition

An mechanism to collect electronic data sets in near real time (i.e. A Process)A particular collection of data and data sets specific to purpose - “Assessment of competence” (i.e. A Thing)Both a Process and a Thing (i.e. A “Proing”)

9

The Good News: Bad News Debate Healthcare professionals leave a massive contrail of electronic data in their daily work

BUT relevant data are very fragmented, scattered across a galaxy of repositories

HIT now allows the potential for sourcing and compilation of data in near real time

BUT the relevant repositories have scant tradition nor appetite for data-sharing

10

The eFolio Connector Vision

Data Commons A collection of Portfolios and Repositories that

can provide data for many different applications

eFolio ConnectorSuite of apps compiling data reports for the learner, and for other viewers authorized by

the learner

Data Sourcing

11

Data Commons A collection of Portfolios and Repositories that

can provide data for many different applications

eFolio ConnectorSuite of apps compiling data reports for the learner, and for other viewers authorized by

the learner

eFolio Connector: Initial Data Sets Planned

eFolio ConnectorSuite of apps compiling data reports for a

variety of viewers authorized by the learner identified

Data Commons Bridging Many Portfolios and Repositories, and providing data for many different applications

Data Sourcing

12

Data Commons A collection of Portfolios and Repositories that

can provide data for many different applications

eFolio ConnectorSuite of apps compiling data reports for the learner, and for other viewers authorized by

the learner

eFolio ConnectorSuite of apps compiling data reports for a

variety of viewers authorized by the learner identified

Some Other Important Issues

Data Sourcing “VirtualRepository”

Reporting out to authorized

viewers specifications

Trusted Agent Hub

13

The “Virtual Repository”: Affiliated RepositoriesSign standard data-sharing agreements (Commons likely to become separate entity) Generically willing to share data, but decide on case by case basis Follow data custodian best practices Expose data in XML, comply with technical specifications for each data setMay consume data, with authorization of the learner identifiedMight charge fee to Apps for data provided

14

More on Trusted Agent HubProvides a backbone of connectivity

across Data Commons infrastructureIncludes array of System Tools (Apps) for

ID matchingDe-identificationAuthenticationBusiness rulesPermissionsTransaction tracking and auditing

Channels of DistributionViewers (authorization by learner or de-identification for identified

data)

Data Commons

Apps and Data Compilations

More on Report-Out to Viewers

Report Out

16

Compiling Data Sets: How Sausage Really Gets

MadeClearly define purpose (e.g. Understanding my practice)Devise necessary schema (e.g. to codify narrative)Do use cases to define content, context, use of data setFind reliable sources of data requiredExecute agreement with repositories to share data Build technical specification to facilitate interoperability Source data from repositories using web servicesPay any agreed-upon fees to repositories providing data

Data Commons

eFolio

Examples of New Data Reports In Development

Data Sourcing

18

What If?There is no current source for data required?

Add needed data to existing repository(ies)Build a new repository, affiliate with Commons

Permanent access to data required, but repository(ies) unwilling or unable to maintain

Arrange with repository to provide archivingBuild central archiving function outside Commons

Learner says that data in a report is wrong?

Build central process for reconciliation of data

19

What if? -2Legal discovery is attempted on one of the data sets generated?

Compiled data sets are expunged by policy immediately after report-out (cf. Mandalas) and no data values are stored in the system

A request is received for identified data from someone other than the relevant learner

No date are moved, compiled or reported out without authorization of the learner

20

Evolution of the eFolio Connector - Feasibility

Work2007 Invitational Conference (sponsored by AAMC, ACGME, FSMB and NBME)2008 Pilot Organizational Meeting; User advisory group established2010 Feasibility Pilot - built ANSI-conformant international technical specification for “Educational Trajectory” data set

21

Evolution of the eFolio Connector: Pre-Implementation

2011At least two more specifications (likely “Assessment of Milestones” and “My Practice”)Marketing, pricing, and business modelPrototype featuring 3-4 reports for feedback

22

ConclusionsThe eFolio Connector can source data from a diverse galaxy of repositories affiliated with the Data Commons and provide potentially useful data reports that

bridge the continuum provide feedback on competence and inform lifelong learning

Reports are available in near real time to learner and any other viewers authorized