Upload
dinhdieu
View
223
Download
0
Embed Size (px)
Citation preview
Substantially Equivalent Competency Assessment
Competencies in Relation to Internationally
Educated Nurses’ Applications for Registration
February 2016
1
Learning From Experience Project
SEC Assessment Data Analysis Report
Executive Summary
The employment of internationally educated nurses (IENs) is a widely-used strategy to
address the global nursing shortage. As a regulated profession, all registered nurses
(RNs) are required to obtain licensure/registration before they can practice, with all
Alberta RN applicants required to provide evidence of competence to practice in the
profession. Yet differences in nursing education programs, health care systems, and
nursing scopes of practice between countries can cause challenges in assessing an
IEN’s nursing competencies.
The Learning from Experience: Improving the Process of Internationally Educated
Nurses’ Applications for Registration (LFE) project sought to use evidence to increase
the efficiency of the IEN application for registration process, and included a systematic
analysis of IEN application data which led to adoptions of policy and practice revisions to
support a better assessment process.
The Substantially Equivalent Competency (SEC) Assessment is a competency-based
assessment designed to evaluate nursing knowledge, skills, attributes, values and
judgments to determine if an IEN possesses substantially equivalent competencies to an
Alberta RN. This statistical analysis linked SEC Assessment data and LFE registration
process data to examine the SEC Assessment competencies in relation to each other, to
application characteristics, and to subsequent outcomes in the IEN application for
registration process. The project team used a variety of statistical analysis methods,
including statistical modeling, to investigate research questions.
In analyzing the strongest and most challenging competency categories, both
Aggregated Competency Mean Scores and statistical modeling identified Professional
Responsibility and Accountability competencies as being strongest amongst the study
population, followed by Ethical Practice, Self-Regulation, Service to the Public, and
Knowledge-Based Practice: Specialized Body of Knowledge. As well, there was
statistically significant evidence to show that all of the applicants had the most
challenges in the Knowledge-Based Practice: Competent Application of Knowledge
competency category.
Applicants had similar SEC Assessment results regardless of their age, gender, basic
education credential, number of countries of experience or practice currency. However,
applicants educated in a country with a similar scope of nursing practice to Canada and
those with a Canadian licensed practical nurse registration had higher mean scores on
the SEC Assessment than applicants who did not have those characteristics.
The trend across competency categories was also generally the same for all of the
education country groups represented in the study population with applicants
demonstrating stronger competencies in Ethical Practice, Professional Responsibility
and Accountability and Self-Regulation, and more challenges in the Service to the
Public, Knowledge-Based Practice: Specialized Body of Knowledge and Knowledge-
Based Practice: Competent Application of Knowledge competency categories. Across all
2
Learning From Experience Project
SEC Assessment Data Analysis Report
of the competency categories applicants educated in the United Kingdom and United
States education country groups had better odds of meeting competencies whereas
applicants educated in the India and Asia education country groups had the lowest odds
of meeting competencies.
In relation to courses assigned, applicants who were referred to each type of course had
more challenges meeting all of the competencies than the applicants who were not
referred to that type of course. The exception was for Nursing Skills courses where there
was no discernable difference between the results of applicants who were referred to
Nursing Skills courses compared to applicants who were not referred. This may be
attributable to the educational requirement to complete Nursing Skills courses as
prerequisites to Clinical Practice courses.
Applicants who passed the national entry-to-practice exam had higher competency
scores across all of the categories than applicants who failed the exam and statistical
modeling confirmed that applicants with high SEC Assessment competency scores were
more likely to pass.
Conversely, ineligible applicants displayed noticeable lower scores across all of the
competency categories compared to applicants who obtained an initial RN registration.
The results of this analysis will be used to support evidence-informed policies at CARNA
surrounding the assessment based on the SEC Assessment report and application
documentation. The findings also have the potential to inform policy and practice
adaptations at SEC Assessment Centres, educational institutions, and other regulatory
bodies.
3
Learning From Experience Project
SEC Assessment Data Analysis Report
Contents
1.0 Introduction ............................................................................................................ 4
1.1 Background ................................................................................................... 4
1.2 Project Purpose and Objectives ..................................................................... 5
1.3 Research Questions ...................................................................................... 5
2.0 Methods ................................................................................................................. 5
2.1 Data Collection and Variables ........................................................................ 5
2.1.1 SEC Assessment Data ......................................................................... 5
2.1.2 IEN Application Data ............................................................................ 6
2.1.3 Data Cleaning and Linkage .................................................................. 7
2.2 Data Analysis................................................................................................. 7
2.2.1 SEC Assessment Competency Analysis ............................................. 7
2.2.2 IEN Application Characteristics and Outcomes Data Analysis ............. 8
2.2.3 SEC Assessment Competencies and Outcomes Data Analysis .......... 9
3.0 Results ................................................................................................................. 10
3.1 Applicant Demographics .............................................................................. 10
3.2 Strongest and Most Challenging SEC Assessment Competencies .............. 11
3.3 SEC Assessment Competencies in Relation to Application Characteristics . 14
3.4 SEC Assessment Competencies in Relation to Outcomes........................... 16
3.4.1 SEC Assessment Competencies in Relation to Courses ................... 16
3.4.2 SEC Assessment Competencies in Relation to Exam Results .......... 17
3.4.3 SEC Assessment Competencies in Relation to Ineligibility ................ 18
4.0 Discussion ............................................................................................................ 19
4.1 Summary ..................................................................................................... 19
4.2 Limitations ................................................................................................... 20
4.3 Moving Forward ........................................................................................... 21
Appendix A: Data Analysis Framework.......................................................................... 22
Appendix B: Research Team and Acknowledgments .................................................... 23
4
Learning From Experience Project
SEC Assessment Data Analysis Report
1.0 Introduction
1.1 Background
The employment of internationally educated nurses (IENs) is a widely-used strategy to
address the global nursing shortage1. As a regulated profession, all registered nurses
(RNs) are required to obtain licensure before they can practice, with all Alberta RN
applicants required to provide “evidence of competence in the practice of the
profession”2. Yet differences in nursing education programs, health systems, and nursing
scopes of practice between countries can cause challenges in assessing an IEN’s
nursing competencies.
Upon receiving an application for registration from an IEN, the College and Association
of Registered Nurses of Alberta (CARNA) assesses the applicant’s education,
experience and other qualification documentation to determine if the applicant’s
professional knowledge, skills, attributes, values and judgments (competencies) can be
deemed substantially equivalent to the competencies required to fulfill the roles and
responsibilities expected of RNs in Alberta. If a determination cannot be made based on
the application documents, an applicant may be referred to a Substantially Equivalent
Competency (SEC) Assessment.
The SEC Assessment uses a variety of strategies to assess RN competencies including:
multiple choice and short answer exams, interview-based case management situations,
clinical lab role play situations and self-assessment. It identifies whether the applicant’s
competencies are substantially equivalent to an entry-level RN in Alberta or if any
competency gaps exist and to what extent. The resulting SEC Assessment report
informs CARNA as to whether the applicant can be deemed substantially equivalent, if
he or she needs to complete bridging education to address competency gaps or if the
gaps are too extensive and the applicant is not eligible to practice in Alberta.
The Learning from Experience: Improving the Process of Internationally Educated
Nurses’ Applications for Registration (LFE) project sought to use evidence to increase
the efficiency of the IEN application for registration process. One component of the
project analyzed SEC Assessment competency data linked with data from the analysis
of application variables to develop an understanding of the relationship between SEC
Assessment competencies, and application characteristics and outcomes.
1 Canadian Nurses Association (2009). Tested Solutions for Eliminating Canada’s
Registered Nurse Shortage. Canadian Nurses Association: Ottawa, ON. 2 Government of Alberta (2000). Health Professions Act. Government of Alberta:
Edmonton, AB. Section 28 (2).
5
Learning From Experience Project
SEC Assessment Data Analysis Report
The research project was led by the College and Association of Registered Nurses of
Alberta and funded by Alberta Health through Health Canada’s Internationally Educated
Health Professionals Initiative. The views expressed herein do not necessarily represent
the views of Health Canada or Alberta Health.
1.2 Project Purpose and Objectives
The LFE research project sought to improve the efficiency of the internationally educated
nurse application for registration process while upholding CARNA’s commitment to
public safety and to the principles of fairness and transparency. The LFE project strived
to:
Develop an evidence-based model for the assessment of IEN applications for
registration.
Make CARNA processes as transparent, simple and efficient as possible from
receipt of a complete application to eligibility for RN registration.
Clearly articulate requirements and expectations of applicants.
Build capacity and leadership in the area of nursing regulation.
The SEC Assessment data analysis sought to:
1. Provide an understanding of the SEC Assessment competencies in relation to
each other.
2. Clearly understand if and how the SEC Assessment competencies relate to IEN
application for registration process characteristics and outcomes.
1.3 Research Questions
The research questions for the SEC Assessment data analysis were as follows:
1. What competencies did applicants have the most challenges with?
2. What competencies were applicants the strongest in?
3. Are there any trends in competencies that applicants were strongest in or had the
most challenges in according to education country group?
4. How did the competencies relate to the courses assigned?
5. Were there competencies that were related to passing or failing the national
entry-to-practice exam?
6. Were there competencies that were related to applicants being deemed
ineligible?
2.0 Methods
2.1 Data Collection and Variables
2.1.1 SEC Assessment Data
SEC Assessment data was collected at Mount Royal University (MRU) as part of routine
reporting in the IEN application for registration process. The SEC Assessment data in
6
Learning From Experience Project
SEC Assessment Data Analysis Report
this study included competency scores from applicants who completed an SEC
Assessment between May 2008 and September 2012. MRU SEC Assessment Centre
staff retroactively entered the data manually into an electronic database, between April
and June 2013.
There are multiple types of SEC Assessments categorized as:
General – assessment of general nursing entry-to-practice competencies in six
competency categories:
o Professional Responsibility and Accountability (“Professional Responsibility”);
o Knowledge-Based Practice: Specialized Body of Knowledge (“Knowledge”);
o Knowledge-Based Practice: Competent Application of Knowledge
(“Application of Knowledge”);
o Ethical Practice;
o Service to the Public (“Service to Public”); and
o Self-Regulation.
Focused (Maternal/Newborn, Child Health, Mental Health) – assessment of
specialized nursing entry-to-practice competencies in the specified area of focus.
General plus Focused – general assessment with one or two focused areas.
Complete – general assessment with all three focused areas.
Individual competencies in each competency category are measured for all the types of
SEC Assessments, however many of the competencies permeate throughout the
different focused SEC Assessments and therefore are only recorded under the General
competency category when a “Complete” or “General plus Focused” assessment is
performed. Therefore, all the applicants in the data had general SEC Assessment
competency information, but not everyone had focused area assessment results.
2.1.2 IEN Application Data
The LFE Project baseline data included 3504 complete application files received from
IEN applicants between January 1, 2008 and December 31, 2011. Information was
collected as part of routine data collection during the application for registration process
and was reported within the CARNA operational database and/or paper applicant files.
Data available only from paper files was entered manually into the database and data
entry was audited to ensure accuracy.
Data from applicants meeting the following criteria were included in the LFE Project:
Internationally Educated Nurse (nursing entry-level education completed outside
of Canada).
Completed application (all application documents received) between January 1,
2008 and December 31, 2011.
Assessed under the Health Professions Act.
Applications that were active, assessed and lapsed (24 months of inactivity) after
the application was considered completed.
7
Learning From Experience Project
SEC Assessment Data Analysis Report
The following consideration was also applied:
Re-applications: The applicant’s most current data was used in place of the
original application data.
Each file was individually reviewed to ensure consistent data entry. The project team
entered and cleaned the data from the files from March to November 2012. IEN
application for registration outcome data for the applicants captured in the SEC
Assessment data analysis was updated in June 2015.
2.1.3 Data Cleaning and Linkage
The SEC Assessment data and LFE Project baseline data were linked based on unique
stakeholder numbers. Data from 1425 SEC Assessment reports were received from
MRU. After linkage, there were 1104 SEC Assessment reports identified for 1076
applicants in the LFE baseline data. Some applicants completed more than one SEC
Assessment. For applicants with multiples reports, only the most complete SEC
Assessment results were included for analysis therefore 28 SEC Assessment reports
were excluded from the linked data. Ultimately, data from 1076 SEC Assessment reports
were analyzed.
2.2 Data Analysis
SEC Assessment competency data was analyzed to identify areas of strengths and
challenges between competencies and competency categories. Demographic and
outcome statistics were also identified for the study sample. Modeling was conducted to
link the competency data and the LFE data, analyzing relationships between the
competencies, and characteristics and outcomes. Analyses were conducted using SAS
9.3. Appendix A outlines the data analysis framework for the SEC Assessment data
analysis.
2.2.1 SEC Assessment Competency Analysis
A three point rating scale comprised of “Met”, “Partially Met”, and “Not Met” ratings was
used for the SEC Assessment until January 2011. It was revised to a four point rating
scale to offer further delineation of the “Partially Met” rating category. This leveling of the
partially met category allowed for greater definition of the degree of a candidate’s
performance in the areas of knowledge, skills, attitude, and judgment requiring growth.
After February 2011, ratings used to describe candidate performance were categorized
according to “Met”, “High Partially Met”, “Low Partially Met” and “Not Met”.
In order to accommodate the differences between the 3-point scale and 4-point scale,
the SEC Assessment competency data was standardized or weighted to allow for a one
scale system as described in Table 2.2.1. It is important to note that “Partially Met”
(score 3) in the analysis scale was not necessarily better than “Low Partially Met” (score
2) or worse than “High Partially Met” (score 4). The analysis scale scores (5 point scale)
were used for statistical analysis.
8
Learning From Experience Project
SEC Assessment Data Analysis Report
Table 2.2.1: Rescaling of SEC Assessment Scales
Not Met Low
Partially Met
Partially Met
High Partially
Met Met
3 Point Scale 1 - 2 - 3
4 Point Scale 1 2 - 3 4
Analysis Scale 1 2 3 4 5
SEC Assessments are conducted in four major areas: General, Maternal-Newborn, Child
Health and Mental Health. The latter three areas are considered specialty areas while
General assessments focus on general nursing competencies. All of the applicants in
the study had General assessment competency data. Some applicants also had
competencies in specialty areas assessed. In the General assessment, there were 78
individual competencies which were assessed in the six aggregated competency
categories described in Section 2.1.1. In order to identify the most challenging and
strongest individual competencies, mean scores were used for analysis. Comparisons of
aggregated competencies were based on Cochran-Mantel-Haenszel tests and
proportional odds models.
Specialty assessments are often interconnected with the General assessment
competencies, with assessors focusing on specialty competencies that are typically not
demonstrated in the General assessment. Therefore specialty assessment data is
typically focused in the Knowledge and Application of Knowledge competency
categories. Since specialty area competency data is inconsistent, statistical models were
only built on General SEC Assessment results.
Means for all individual competencies were calculated and compared within each
aggregated competency category. The highest mean individual competency score
indicated the strongest individual competency in the category for all applicants. In
contrast, the lowest mean individual competency score indicated the most challenging
individual competency for all applicants.
Proportional odds models were used to estimate the odds ratios of aggregated
competency categories. The SEC Assessment competency scores were grouped into
three groups based on practice: scores 1 and 2 were grouped together as “Not Met”;
score 3 remained separate as “Partially Met” and scores 4 and 5 were grouped together
as “Met”. The consideration of the 5 scores as 3 groups reflected the typical practice of
CARNA registration assessors when they reviewed an IEN's SEC Assessment results.
2.2.2 IEN Application Characteristics and Outcomes Data Analysis
Descriptive statistics (frequency, proportion, mean, histograms) were generated for each
application characteristic and outcome variable with continuous variables evaluated and
further categorized to describe frequency and conduct cross tabulations. The variables
represented the applicants’ demographic, education, employment and external
registration information, as well as variables representing courses, process outcomes
9
Learning From Experience Project
SEC Assessment Data Analysis Report
and competency scores. The p-values based on variance from cross tabulation and chi-
square tests were used to assess significant associations between variables and
outcomes in the bivariate analyses. Only General SEC Assessment results were used
for the linked analysis.
Aggregated Competency Median and Average Scores were calculated for each of the
applicant characteristic variables based on the 78 individual competencies for each
applicant. Therefore, every applicant would have one overall score to represent all of
their SEC Assessment competencies. Descriptive statistics (mean, median, minimum,
maximum, 90th percentile) were generated for each variable.
Trends in competencies by education country group were modeled by using a random
intercept proportional odds model. Aggregated competency groups and education
country groups were used as covariates, and their interaction term was added to the
model.
2.2.3 SEC Assessment Competencies and Outcomes Data Analysis 2.2.3.1 SEC Assessment Competencies and Courses Data Analysis
Descriptive statistics (frequency, mean, histograms) were generated for each course
type by aggregated competency category. Spearman correlations between the
aggregated competency category and each course type were also produced to evaluate
the independently paired associations between the competency categories and courses.
2.2.3.2 SEC Assessment Competencies and National Entry-to-Practice Exam
Results Data Analysis
Cross tabulation between the exam results and the SEC Assessment competency
categories was conducted using chi-squared tests. In addition, the bivariate analysis
between the exam results and the mean score of aggregated SEC Assessment
competency categories was conducted.
Logistic regression models were used to explore the independent associations between
the exam results (pass or fail) and SEC Assessment multiple choice and short answer
percentage scores, and the mean scores of the SEC Assessment competencies within
each aggregated competency category. The model diagnostic was based on Hosmer-
Lemeshow tests and a Receiver Operating Characteristic curve was produced to assess
the predictive power of each variable.
2.2.3.3 SEC Assessment Competencies and Ineligibility Analysis
Aggregated Competency Mean Scores were calculated for applicants who were deemed
ineligible as compared to applicants who received an initial RN registration.
10
Learning From Experience Project
SEC Assessment Data Analysis Report
3.0 Results
Following a description of key applicant demographics, the data analysis results are
presented in response to each of the six research questions.
3.1 Applicant Demographics
Of the 1076 applicants included in the data analysis:
The mean age of applicants was 32 with 86.43% of applicants younger than age
40 when they applied.
82.16% of the applicants were female and 17.84% were male.
The majority of applicants were educated in the Philippines (61.52%) and India
(12.08%).
16.54% of applicants had a basic education credential similar to an Alberta
baccalaureate (3-4 years post-secondary generalist nursing education after 12
years of elementary and secondary education) and 82.99% were considered to
have a diploma or degree not similar to an Alberta baccalaureate.
10.50% were educated in an area where the scope of nursing practice is similar
to that of Canada.
15.06% were registered with a licensed practical nurse regulatory body in
Canada at the time of application.
30.48% of applicants had transition (migration) experience, working in additional
countries other than the one they were educated in prior to applying to CARNA.
Only 6.78% of the applicants did not have practice currency (graduation or 1125
hours of practice in the last 5 years) at the time of application.
The average amount of time since the applicants last practiced was 0.71 years.
Figure 3.1.1 outlines the applicants’ pathway through the IEN application for registration
process including the frequency of outcome decisions at each phase. Following the SEC
Assessment, the majority of applicants (66.17%) were referred to bridging education
while 30.02% were deemed ineligible. At the time of data extraction, 40.71% had been
deemed eligible to write the national entry-to-practice exam and work in Alberta with
temporary registration to fulfill the reference requirement (“TP Eligible”).
Subsequent to being deemed TP Eligible, 34.01% of applicants had written the national
entry-to practice exam and 27.88% of applicants had obtained an initial RN registration.
Over the course of the process 31.13% of the applicants had been deemed ineligible for
RN registration in Alberta, and 7.16% of the applicants had no forward movement in two
years and were lapsed.
11
Learning From Experience Project
SEC Assessment Data Analysis Report
Figure 3.1.1: SEC Assessment Data Analysis Population Flow Chart
3.2 Strongest and Most Challenging SEC Assessment
Competencies
Mean competency scores were calculated for each of the 78 general competencies with
the highest mean scores indicating the strongest competencies and the lowest mean
scores indicating the most challenging individual competencies. Therefore, research
questions one and two were addressed together.
Research Question 1: What competencies did applicants have the most
challenges with?
Research Question 2: What competencies were applicants the strongest in?
Table 3.2.1: Strongest and Most Challenging Individual Competencies in the General SEC Assessment
Category
Strongest Individual Competency
Most Challenging Individual Competency
Professional Responsibility
Accountable and responsible for own actions, decisions, including personal safety.
Demonstrates critical inquiry processes in relation to new knowledge and technologies.
12
Learning From Experience Project
SEC Assessment Data Analysis Report
Category
Strongest Individual Competency
Most Challenging Individual Competency
Knowledge Has knowledge of relational communication and understands that relational practice is the foundation for all nursing practice.
Understands the role of primary health care in health delivery systems (accessibility, health promotion, public participation, appropriate use of technology, intersectoral collaboration) and its significance for population health.
Application of Knowledge
Completes assessment in a timely manner.
Verifies when clients have essential information and skills.
Ethical Practice Understands ethical and legal considerations related to maintaining client confidentiality in all forms of communication.
Takes into consideration the spiritual and religious beliefs and practices of clients.
Service to Public Manages physical resources to provide effective and efficient care (equipment, supplies, medication, linen).
Participates and contributes to nursing and health care team development by: building partnership, knowing and supporting the full scope of practice of various team members, and using appropriate channels of communication.
Self-Regulation Understands the significance of the concept of fitness to practice in the context of individual self-regulation and public protection.
Distinguishes between the legislated scope of practice for the RN individual scope of practice based on own level of competence.
13
Learning From Experience Project
SEC Assessment Data Analysis Report
Figure 3.2.1: Score Distribution Among General Aggregated Competency Categories
Figure 3.2.1 shows the mean aggregated competency scores for all of the competency
categories. The Professional Responsibility category has the highest percentage of
“Met” scores and lowest number of “Not Met” scores, whereas the Application of
Knowledge category has similar percentages of “Met” and “Not Met” scores. Focusing on
the “Met” scores, the analysis of the aggregated mean competency scores would
indicate that the competency categories from strongest to most challenging are
Professional Responsibility, Ethical Practice, Self-Regulation, Service to Public,
Knowledge and Application of Knowledge. The trend in scores for all of the competency
categories is similar.
Proportional odds modeling also shows that the strongest aggregated competency
category for all the applicants was Professional Responsibility, followed by Ethical
Practice, Self-Regulation, Service to the Public, Knowledge and Application of
Knowledge. There was statistically significant evidence to show that all applicants had
more challenges in Knowledge and Application of Knowledge compared to other
competency categories. This aligns with the findings from the aggregated mean
competency scores.
0.00
5.00
10.00
15.00
20.00
25.00
30.00
35.00
40.00
45.00
1-Not Met 2-Low PartiallyMet
3-Partially Met 4-High PartiallyMet
5-Met
Pe
rce
nt
Score
Professional Responsibility Knowledge Application of KnowledgeEthical Practice Service to Public Self Regulation
14
Learning From Experience Project
SEC Assessment Data Analysis Report
3.3 SEC Assessment Competencies in Relation to Application Characteristics
Of the 1076 applicants, the Median Competency Mean Scores were similar amongst the
age groups, genders, basic education credential, number of countries of experience
other than country of education, and practice currency. However, applicants with a
Canadian LPN Registration had a higher Median Competency Mean Score than
applicants who did not report an LPN registration in Canada.
Research Question 3: Are there any trends in competencies that applicants were
strongest in or had the most challenges in according to education country group?
In analyzing applicants’ education countries, some countries had very few observations
therefore, in order to have more balanced data, countries were grouped into 13
Education Country Groups. The Research Team identified several countries where
larger numbers of applicants were educated (e.g. the Philippines, India, the United
States) and where there was interest in individual education country analysis. Other
countries were grouped by geographical region with the assumption that education
programs were similar in each region. Applicants who received their entry-level
education in the United Kingdom, the Middle East, Australia/New Zealand, and the
United States had high Median Competency Mean Scores compared to the other
education country groups. Applicants who were educated where the scope of nursing
practice is similar to Canada also had a higher Median Competency Mean Score than
applicants who were educated in other areas.
15
Learning From Experience Project
SEC Assessment Data Analysis Report
Figure 3.3.1: Aggregated Competency Mean Scores by Education Country Group
The trend across competency categories was similar for all of the education country
groups with higher Aggregated Competency Mean Scores in Ethical Practice,
Professional Responsibility and Self-Regulation, and lower scores in Service to Public,
Knowledge and Application of Knowledge. There were a few exceptions. The
Australia/New Zealand country group had high Service to Public scores, similar to those
of Ethical Practice and Self-Regulation. As well, the Eastern Europe, the
Tropics/Caribbean, Asia and India country groups had Self-Regulation scores lower than
their Service to Public scores. The study population included only two applicants
educated in the South America education country group therefore the country group was
not included in this analysis due to the small sample size.
The random intercept proportional odds model confirmed the trends identified by the
Aggregated Competency Mean Scores. Throughout each competency category, the
United Kingdom and United States had the highest Aggregated Competency Mean
Scores and highest odds ratios indicating that applicants educated in those country
groups had better odds of meeting all of the competencies. Applicants educated in the
India country group had the lowest mean scores and lowest odds ratios. The Asia
education country group also had low odds ratios in Knowledge, Professional
Responsibility, Application of Knowledge, and Self-Regulation.
16
Learning From Experience Project
SEC Assessment Data Analysis Report
3.4 SEC Assessment Competencies in Relation to Outcomes
3.4.1 SEC Assessment Competencies in Relation to Courses
Research Question 4: How did the competencies relate to the courses assigned?
Following an SEC Assessment, all of the types of courses, except for Nursing Skills and
Ethics, had more than half of applicants referred to that type of course. The highest
percentages of applicants were referred to Health Assessment (80.53%), Mental Health
(78.33%) and Professional Nursing in Canada (76.87%) courses.
For all the courses except Nursing Skills courses, applicants who were referred to the
courses had discernibly lower Aggregated Competency Mean Scores than applicants
who were not referred to the course. There were no individual competency categories
that were associated with referral to any of the courses except the Ethics courses, rather
referral to a particular course was related to lower Aggregated Competency Mean
Scores across all of the competency categories. The largest differences between scores
for competency categories were around 1.00-1.07 for most of the courses. Applicants
referred to Ethics courses had an Aggregated Competency Mean Score in the Ethical
Practice competency category that was 1.70 lower than applicants who were not
referred to Ethics courses.
There was no discernible difference between the Aggregated Competency Mean Scores
of applicants who were referred to Nursing Skills courses compared to applicants who
were not referred. This may be related to the educational requirement to complete
Nursing Skills courses as prerequisites to Clinical Practice courses. Assessors may
determine that the applicant needs a Clinical Practice course even if the applicant’s
nursing skills are satisfactory in which case they would be referred to both. This may
occur if a) the applicant can demonstrate competencies related to nursing skills but has
challenges with critical thinking competencies; b) the applicant has been out of practice
a number of years or is close to losing currency as Clinical Practice courses are
considered to meet the currency requirement, or c) the applicant has limited clinical
experience.
The Spearman Correlation indicated that applicants referred to Professional Nursing in
Canada, Health Alterations and Therapeutics, Clinical Practice, and Pediatrics courses
were all moderately negatively correlated to all of the competency categories. Thus the
better applicants did on the SEC Assessment; the less likely they were to be assigned to
take those four types of courses.
17
Learning From Experience Project
SEC Assessment Data Analysis Report
3.4.2 SEC Assessment Competencies in Relation to Exam Results
Research Question 5: Were there competencies that were related to passing or
failing the national entry-to-practice exam?
Figure 3.4.1: Aggregated Competency Mean Scores in Relation to Exam Results
Applicants who passed the national entry-to-practice exam had higher Aggregated
Competency Mean Scores than applicants who failed the exam across all competency
categories. The range in differences between mean scores shows the smallest
difference for Service to Public and the largest difference for Self-Regulation.
Logistic regression modeling showed that all if the competency categories have a
significant impact on the exam results. Applicants with strong SEC Assessment results
were more likely to pass the national entry-to-practice exam. The strongest predictive
competency categories for exam results are Ethical Practice, Knowledge and
Professional Responsibility; however the six competency categories are highly
correlated.
4.29
3.80 3.69
4.19 3.97
4.14
3.74
3.20 3.14
3.61 3.54 3.50
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
ProfessionalResponsibility
Knowledge Application ofKnowledge
Ethical Practice Service to Public Self-Regulation
Agg
rega
ted
Co
mp
ete
ncy
Me
an S
core
Competency Category
Pass (N=325) Fail (N=41)
18
Learning From Experience Project
SEC Assessment Data Analysis Report
3.4.3 SEC Assessment Competencies in Relation to Ineligibility
Research Question 6: Were there competencies that were related to applicants
being deemed ineligible?
Figure 3.4.2: Aggregated Competency Mean Scores for Ineligible Applicants and Applicants who Received an Initial RN Registration
Ineligible applicants had discernibly lower Aggregated Competency Mean Scores than
applicants who received their initial RN registration across all competency categories.
There were no individual competency categories related to ineligibility, rather ineligible
applicants had low scores across all competency categories. Ineligible applicants had
Aggregated Competency Mean Scores lower than 2.50 across all competency
categories whereas applicants who received an initial RN registration had Aggregated
Competency Mean Scores higher that 3.50 in all of the competency categories. The
range in differences between mean scores shows the smallest difference for Service to
Public and the largest difference for Self-Regulation.
2.37
1.93 1.79
2.44 2.26
2.15
4.27
3.78 3.67
4.17 3.96
4.13
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
ProfessionalResponsibility
Knowledge Application ofKnowledge
Ethical Practice Service toPublic
Self-Regulation
Agg
rega
ted
Co
mp
ete
ncy
Me
an S
core
Ineligible Applicants RN Registered Applicants
19
Learning From Experience Project
SEC Assessment Data Analysis Report
4.0 Discussion
4.1 Summary
To our knowledge, this is the first study to examine SEC Assessment competencies in
relation to the larger IEN application for registration process. Typically characteristics of
the IEN applicants are withheld from SEC Assessment assessors to prevent bias during
the assessment. As well, CARNA registration assessors make the decision on the SEC
Assessment review outcome based on the SEC Assessment report and the applicant’s
education and experience documentation. This study used data from the SEC
Assessment Centre and from CARNA’s applicant database to explore the relationship
between SEC Assessment competencies, and application characteristics and outcomes.
The analysis revealed that trends in the competency categories were similar, with
applicants demonstrating stronger competencies in Professional Responsibility, Ethical
Practice and Self-Regulation, and having the most challenges in demonstrating
Application of Knowledge competencies across all of the data analyses. Both the
Aggregated Competency Mean Scores and statistical modeling showed that the
competency categories from strongest to most challenging were Professional
Responsibility, Ethical Practice, Self-Regulation, Service to Public, Knowledge and
Application of Knowledge. There was statistically significant evidence to show that all
applicants had the most challenges in the Application of Knowledge competency
category. Difficulties demonstrating skills and abilities in the presence of an assessor
may contribute to the factors that cause challenges in this competency category.
Similar Median Competency Mean Scores were observed regardless of IEN applicant
demographic characteristics including age, gender, basic education credential, number
of countries of experience, and practice currency. However, those educated in a country
with a similar scope of nursing practice to Canada and those with a Canadian LPN
registration had higher Median Competency Mean Scores than those who were
educated elsewhere or who did not have a Canadian LPN registration.
The trend across competency categories was also generally the same for all of the
education country groups with higher Aggregated Competency Mean Scores in Ethical
Practice, Professional Responsibility and Self-Regulation, and lower scores in Service to
Public, Knowledge and Application of Knowledge. Across all of the competency
categories the United Kingdom and United States education country groups had the
highest Aggregated Competency Mean Scores and had better odds of meeting
competencies according to statistical modeling. This finding is not surprising considering
the similarities of nursing scopes of practice and health care systems between these
countries and Canada. In contrast, applicants educated in the India and Asia education
country groups had the lowest mean scores and lowest odds ratios for meeting
competencies.
20
Learning From Experience Project
SEC Assessment Data Analysis Report
In the analysis of the SEC Assessment competencies in relation to courses assigned,
there was no individual competency category related to referral to any of the courses,
rather referral to each course was related to lower Aggregated Competency Mean
Scores across all of the competency categories. For all courses except Nursing Skills,
applicants who were referred to each type of course had discernibly lower Aggregated
Competency Mean Scores than those who were not referred to that type of course. The
lack of detectable difference between the Aggregated Competency Mean Scores of
applicants who were referred to Nursing Skills courses compared to applicants who were
not referred may be attributable to the educational requirement to complete Nursing
Skills courses as prerequisites to Clinical Practice courses.
The Aggregated Competency Mean Scores were higher for applicants who passed the
national entry-to-practice exam than for applicants who failed the exam across all
competency categories. Logistic regression modeling confirmed that applicants with
strong SEC Assessment results were more likely to pass. The strongest predictive
competency categories for exam results were Ethical Practice, Knowledge and
Professional Responsibility. However the six competency categories are highly
correlated and modeling showed that all of the competency categories have a significant
impact on the exam results.
There was no individual competency category that was related to applicant ineligibility,
rather across all of the competency categories, ineligible applicants had Aggregated
Competency Mean Scores that were less than 2.50 whereas applicants who received
their initial RN registration had Aggregated Competency Mean Scores above 3.50.
4.2 Limitations
LFE Project data was collected as part of routine data collection during the IEN
application for registration process. As the data were extracted from an active database
which was not designed for research purposes, it was not entered and cleaned in
isolation and variations in the data extraction may exist. Following extraction from the
CARNA database, the Research Team, Data Entry Clerks and Data Analyst thoroughly
reviewed and cleaned the data used for this analysis.
In May 2013, CARNA transitioned to a new client management database with different
functionality and coding. This required the development of a new data query to extract
the project data. As the LFE Project data in this analysis spanned a time period which
included both databases, an alignment of data codes was required.
The timeframe for the SEC Assessment data analysis was different than the timeframe
for the LFE Project baseline data, therefore data linkage was complex with some of the
SEC Assessment records falling outside of the baseline data period. Data from 1425
SEC Assessment reports was received from MRU however, after linkage with the LFE
Project data the study population included data from 1076 applicants.
21
Learning From Experience Project
SEC Assessment Data Analysis Report
As well, due to the various education and employment experiences for each applicant,
not every applicant was referred to a complete SEC Assessment (both general and
specialty areas). Therefore data in specialty areas were only available for some
applicants.
The shift in the SEC Assessment scale from a 3 point scale to a 4 point scale in the
midst of the data set required both scales to be converted to a 5 point scale for analysis.
Therefore in order to interpret these findings in relation to either scale a conversion was
made.
4.3 Moving Forward
Findings from the SEC Assessment data analysis provide an enhanced understanding of
SEC Assessment competencies in relation to IEN application for registration
characteristics and process outcomes.
Knowledge surrounding strong and challenging competency areas may inform SEC
Assessment Centres in the development of assessment tools and approaches, and
guide IENs in preparing for the SEC Assessment. Additionally, findings surrounding
strong and challenging competency categories, and the course related analysis may
provide guidance to registration assessors in assigning courses, or to educators in the
development of bridging education curriculum. Registration assessors may also apply
findings related to the exam results and ineligibility to decision-making surrounding the
review of SEC Assessment competency scores.
Overall, the results of this analysis can be used to support evidence-informed policies at
CARNA to improve the decision-making processes surrounding the SEC Assessment.
The findings also have the potential to inform policy and practice adaptations at SEC
Assessment Centres, educational institutions offering bridging education and other
regulatory bodies.
22
Learning From Experience Project
SEC Assessment Data Analysis Report
Appendix A: Data Analysis Framework
23
Learning From Experience Project
SEC Assessment Data Analysis Report
Appendix B: Research Team and Acknowledgments
Working Group Jennifer Kwan, LFE Project Manager Mengzhe (Maggie) Wang, Data Analyst Winston Qui, Biostatistician Dr. Pam Nordstrom, Director, School of Nursing and Midwifery, Mount Royal University College and Association of Registered Nurses of Alberta Cathy Giblin, Registrar/Director, Quality Assurance Loreta Suyat, Deputy Registrar Nan Horne, RN Consultant Nancy MacPherson, RN Consultant Roxanne Ferrier, RN Consultant Barry Bissoondatt, Manager Operations, Registration Services Gillian Lemermeyer, RN Policy Associate Crystal Komanchuk, Communications Coordinator University of Alberta Dr. Greta Cummings, Professor, Faculty of Nursing
ACKNOWLEDGEMENTS
The LFE Research Team would like to acknowledge the dedicated work of the Data Entry Clerks in entering and cleaning IEN application data, and Jessica Schaub in the development of this report. We would like to thank the Mount Royal University SEC Assessment Centre for entering and providing the SEC Assessment competency data, as well as valuable information on the assessments. Additionally, we recognize the ongoing support of the CARNA Registration Services Department and Leadership Team. The Learning from Experience: Improving the Process of Internationally Educated Nurses’ Applications for Registration project was funded by Alberta Health through Health Canada’s Internationally Educated Health Professionals Initiative. The views expressed herein do not necessarily represent the views of Health Canada or Alberta Health.