Embed Size (px)
Citation preview
Monitoring Report Online Competency Assessment (OCA) Project
1 of 12
Board Direction Motion 5533 (May 21, 2015)
THAT a monitoring report be prepared to demonstrate how creation of a new online competency assessment system by Engineers Canada will support constituent associations in advancing the engineering profession and its self‐regulation in the public interest at a cost that is justified by the results. The monitoring report will provide details including, but not limited to:
• the terms and conditions of the funding agreement with the Federal Government;
• the use of the online system by CA's as agreed to with the Federal Government;
• demonstration that the online system with achieve the goals of the Federal Government as announced April 13, 2015;
• the plan for updating the program upon completion; and
• a demonstration that the work does not conflict with any of:
o EL‐2 which shall not cause or allow conditions, procedures or decisions which impede productive relations with constituent associations:
o EL‐5 which shall not permit planning that allocates resources in a way that deviates materially from Ends priorities; and,
o EL‐7 which shall not permit the Board to be uninformed or unsupported in its work.
Author Kim Allen
Date Submitted September 30, 2015
Policies requested EL‐2 The chief executive officer shall not cause or allow conditions, procedures or decisions which impede productive relations with constituent associations.
EL‐5 The chief executive officer shall not permit planning that allocates resources in a way that deviates materially from Ends priorities, that risks fiscal jeopardy, or that is not derived from a multi‐year plan.
EL‐7 The chief executive officer shall not permit the Board to be uninformed or unsupported in its work.
Other relevant policies
Monitoring Report Online Competency Assessment (OCA) Project
2 of 12
This report is presented in four sections to address the specific requests and each of the policies. The project is in progress, so evidence of outcomes or actual expenditures are not available. The report therefore focuses on the planned project activities, schedule and budget. Policy interpretations in this report refer solely to aspects of the Online Competency Assessment (OCA) Project.
Background:
The history of the competency‐based assessment project started in 2008 with Alternative Methods of Licensure, a project to investigate recommendation #9 of the From Consideration to Integration project which called for Engineers Canada to “study the feasibility of alternative system of evaluating an applicant’s professional competency for licensure in comparison to the existing system.” This recommendation arose from studying the experiences of engineering immigrants and a recognition that it was necessary to improve the licensure process and outcomes for qualified international engineering graduates (IEGs). At the time, IEGs found the licensure process to be frustrating, time‐consuming, bureaucratic, complicated, inconsistent, difficult to navigate, and in some cases unfair.
Through Alternative Methods of Licensure, Engineers Canada studied the licensure requirements in each jurisdiction, investigated licensure models in other countries, and concluded that moving to a competency-based assessment for the engineering work experience portion of licensure would provide the most value in terms of simplifying, clarifying and making more consistent the process by which engineers are licensed in Canada and in terms of improving the integration of foreign trained professionals into the Canadian engineering profession. The project concluded by recommending seven core competencies that all engineers would be required to demonstrate through their engineering work experience. It also recommended that a future project develop a competency assessment tool; develop training for staff, volunteers and applicants; and conduct pilot projects. These recommendations were endorsed by the Engineers Canada Board on May 28, 2010 via motion #5120.
The Competency-Based Assessment of Engineering Work Experience Project began in 2010 to implement those recommendations. It successfully developed the required tools and conducted pilot projects in Ontario and Saskatchewan which led to further refinements. The project resulted in three key recommendations for the Board.
1. Adopt the proposed competencies as the definition of the work experience requirement for licensure as a professional engineer.
2. Endorse the proposed competency‐based assessment process as the baseline for national harmonization of the work experience requirement.
3. Support the progressive implementation of the proposed solution by Engineers Canada to:
• Coordinate the design, development and deployment of an online system.
• Create a national steering committee to provide oversight for the development of the online tool and implementation by associations
• Assist each association through the adoption phase
Monitoring Report Online Competency Assessment (OCA) Project
3 of 12
All three recommendations were endorsed by the Engineers Canada Board on March 1, 2013 via motion #5320. Since that time, Engineers Canada has been working to secure funding from the federal government (Employment and Skills Development Canada – ESDC). When ESDC indicated that they would not fund the entire project, since it benefited both foreign and domestic applicants, the Engineers Canada Board endorsed the use of $1.0 M from reserve funds for our portion of the costs. This decision was made at the May 23, 2014 Engineers Canada Board meeting via motion #5441.
On April 1, 2015 ESDC informed Engineers Canada that they were willing to fund 50% of the project. A federal government press release on April 13, 2015 stated, in part:
“The Honourable Pierre Poilievre, Minister of Employment and Social Development, released a report today outlining new and better ways to integrate immigrants into the Canadian workforce.
The Minister shared highlights from the report, authored by the Panel on Employment Challenges of New Canadians, during his speech at The Conference Board of Canada’s Canadian Immigration Summit 2015, which kicked off in Ottawa today.
The Minister also announced funding for two related projects that will see internationally trained doctors and engineers have their foreign credentials more quickly and effectively recognized by eliminating red tape and taking advantage of new online tools.
The two projects, one led by the Medical Council of Canada and the other by Engineers Canada, will help address some of the challenges noted by the Panel that newcomers face when trying to obtain employment. These challenges include problems getting foreign qualifications recognized, a lack of Canadian work experience, inadequate pre‐arrival information and a mismatch of skills to region.”
For additional information on the project please also see https://www.engineerscanada.ca/competency‐based‐assessment
SECTION I – Requests from the Board
Provide details including but not limited to: • the terms and conditions of the funding agreement with the Federal Government; • the use of the online system by constituent associations as agreed to with the Federal Government; • demonstration that the online system with achieve the goals of the Federal Government as announced April
13, 2015; • the plan for updating the program upon completion;
Chief Executive Officer’s Interpretation, in the context of the OCA project
The press release of April 2013, 2015 did not explicitly state the goals of the Federal Government. Based on the press release quoted above (the full text is available here), those goals are interpreted as:
1. Foreign credentials recognized more quickly and effectively
2. Online tools are implemented so that assessments can begin from anywhere in the world
Justification of Reasonableness
The press release makes reference to the recommendations from the report of the Panel on Employment
Monitoring Report Online Competency Assessment (OCA) Project
4 of 12
Challenges of New Canadians, but only states that the Minister will “study them”, not that they are goals of the Federal Government’s. In the two paragraphs that speak directly to the funding of the project, only the online tool and the recognition of foreign credentials are items that are also included in the agreed‐to scope of deliverables for the Online Competency Assessment project description that is part of the funding agreement with ESDC.
Requirement: Evidence:
Terms and conditions of the funding agreement with the Federal Government
The federal government has agreed to fund 50% of the project costs over the 24 month project duration. The project objectives are to:
“- develop and implement an online tool necessary for the competency-based assessment of engineering work experience for professional engineering licensure. This tool will provide orientation to applicants and assessors, and incorporate an on-demand help capability at all critical points in the workflow;
- develop and implement an online assessor training tool, with a learning management system critical to improving consistency and objectivity of licensure decisions.”
Engineers Canada must provide quarterly progress reports to ESDC including achievement of the desired outcomes:
“a fully standardized and automated process that is fair, and transparent, and standard in assessing the work experience of foreign-trained engineers”
The achievement of these outcomes will be measured through qualitative surveys of applicants, assessors and staff; and via quantitative analysis of assessments.
Use of the online system by the constituent associations as agreed to with the Federal Government
The project has committed to conduct pilots with two to three constituent associations of different sizes.
While the budget and schedule allow for three final implementations with one to two constituent associations per implementation (i.e. between three and six CAs), there is no commitment made as to the final number of constituent associations who will use the system.
Achievement of the goals of the Federal Government as announced on April 13, 2015
1. Development of an online tool – are all online tools (assessment, training, on-line support) developed and
Monitoring Report Online Competency Assessment (OCA) Project
5 of 12
piloted?
2. Quicker assessments – are foreign-trained applicants able to self-navigate, understand the requirements, and submit the necessary documentation in less time than they currently are? Are the constituent associations able to make decisions about the work experience portion of licensure in less time than they currently are, considering all staff and volunteer time?
3. More effective assessments – is the new system better in terms of clarity, consistency and transparency?
The goals will be measured through qualitative and quantitative surveys of applicants, assesors and constituent association staff.
The plan for updating the program upon completion
The plan for updating the program is contingent on who provides the IT solution, where it is hosted, how much hosting and maintenance cost, and the terms of the service agreements which are established with the participating constituent associations.
All of these items are currently unknown, although the decision related to each one is already included as a planned project activity.
Once all of the items are established, the final plan can be developed and agreed to with all constituent associations.
Monitoring Report Online Competency Assessment (OCA) Project
6 of 12
SECTION II – EL-2 Treatment of Constituent Associations
EL‐2 The chief executive officer shall not cause or allow conditions, procedures or decisions which impede productive relations with constituent associations. Further, without limiting the scope of the above statement by the following list, the chief executive officer shall not: 1. Permit constituent associations to be without easy access to clear information about the privileges and
responsibilities of membership. Not applicable to OCA Project. 2. Use methods of collecting, reviewing, storing or transmitting constituent association information that fail to
protect against improper access to the information elicited. 3. Operate without consideration of constituent association viewpoints regarding operational means which
directly impact the constituent associations. 4. Operate without a process to ensure that constituent association comments and complaints are responded
to fairly, consistently, respectfully, and in a timely manner. 5. Permit constituent associations to be without timely information regarding assessment fees, or change in
assessment fees. Not applicable to OCA Project. 6. Neglect to advise constituent associations in a timely manner about issues that Engineers Canada is aware
of which may impact the profession and/or constituent associations. 7. Neglect to facilitate collaborative development of shared resources, including those developed by the
constituent associations to achieve the Ends.
Chief Executive Officer Interpretation, in the context of the OCA project
EL‐2 The OCA project must work with all constituent associations, and use effective communication, project management and change management techniques to ensure productive relationships with all constituent associations.
EL‐2.2 The online tools developed through OCA must respect privacy laws and ensure data security and integrity.
EL‐2.3 The OCA project must understand the requirements and limitations of each constituent association and work to develop customizations which respect those constraints.
EL‐2.4 The OCA project must work collaboratively with all interested constituent associations and develop tools which meet their needs and respect the goals of greater consistency, clarity and objectivity.
EL‐2.6 The OCA project must keep all constituent associations informed of progress and of new requirements (if any) raised by the Federal Government.
EL‐2.7 The OCA project must make use of existing systems and tools that have been developed by the constituent associations, where possible.
Justification of Reasonableness
Like all Engineers Canada initiatives, programs and tools, the OCA project must provide value for the constituent associations. The only way to do this is to maintain open communication, and develop the final product
Monitoring Report Online Competency Assessment (OCA) Project
7 of 12
collaboratively to make sure that it meets the needs of each constituent association while respecting the overall goals of online competency assessment project.
Compliance will be achieved when: Evidence of compliance:
The OCA project must work with all constituent associations, and use effective communication, project management and change management techniques to ensure productive relationships with all constituent associations
The project activity breakdown includes change management and communications as an ongoing aspect of the project, as follows:
Change management and communications strategy (June 2015 – April 2017)
‐ Design and integrate change management activities into the master project plan
‐ Distribute “Case for Change” document to facilitate decision making at all levels
‐ Maintain pro-active communications to the associations over the entire lifecycle of the project (e.g. project reports, webinars and demonstrations)
‐ Identify and empower change agents within each association to facilitate deployment of the solution within their respective associations
‐ Solicit feedback from the associations at all stages and respond appropriately
The online tools developed through OCA must respect privacy laws and ensure data security and integrity
The project activity breakdown includes analysis of constituent association privacy and security requirements and development of software design requirements to match, as follows:
Analyze existing processes and IT Infrastructure of the 12 constituent associations (July 2015 – Oct 2015)
‐ Identify privacy and security concerns and requirements
Analyse software design requirements (Jan 2016 – Feb 2016)
‐ Determine detailed requirements for applicant orientation
‐ Develop application tracking requirements
The OCA project must understand the requirements and limitations of each constituent association and work to develop customizations which respect those constraints
The project activity breakdown includes several activities to understand constituent association needs, as follows:
Analyze existing processes and IT Infrastructure of the 12
Monitoring Report Online Competency Assessment (OCA) Project
8 of 12
constituent associations (July 2015 – Oct 2015)
‐ Establish liaison with IT staff of the constituent associations
‐ Identify workflow and document management required for tracking applicant submissions and decision making by each association.
‐ Identify web architecture requirements and standards
‐ Understand interface requirements to integrate with the existing databases of the associations
Assess and obtain concurrence on strategy for limited licences (Oct 2015)
‐ Seek agreement on the scope of limited licences within the competency framework
‐ Seek agreement on a standard format for the scope definition of limited licences
The OCA project must work collaboratively with all interested constituent associations and develop tools which meet their needs and respect the goals of greater consistency, clarity and objectivity
The project activity breakdown includes several activities to work collaboratively with the constituent association, as follows:
Readiness analysis (June 2015 – Oct 2015)
‐ Assess the implementation readiness of constituent associations to facilitate setting deployment targets
‐ Identify key issues and address within the communications strategy
Analyze existing processes and IT Infrastructure of the 12 constituent associations (July 2015 – Oct 2015)
‐ Establish liaison with IT staff of the constituent associations
Assess and obtain concurrence on strategy for limited licences (Oct 2015)
‐ Seek agreement on the scope of limited licences within the competency framework
‐ Seek agreement on a standard format for the scope definition of limited licences
Develop concept prototype and demonstrate nationally (Mar 2016 – June 2016)
‐ Develop a proof-of-concept prototype based on analysis of all design requirements
Monitoring Report Online Competency Assessment (OCA) Project
9 of 12
‐ Demonstrate the prototype nationally to reinforce buy-in of the associations and further refine or validate the design requirements
‐ Update the design requirements based on the feedback received from the demonstration of the prototype
Implement assessor recruitment and certification program (Aug 2016 – Dec 2016)
‐ Liaise with constituent associations to facilitate recruitment of assessors
Implement adoption strategy (Dec 2016 – Jan 2017)
‐ In accordance with change management strategy finalize pilot locations for first deployments
‐ Draft service standard agreements between Engineers Canada and the associations
‐ Conduct pro-active deployment planning meetings with associations
‐ Provide support to associations during deployment for policy, technical and business process changes
Deploy pilot solution to select associations (Feb 2017 – Apr 2017)
‐ Monitor operationalization of pilots
‐ Refine best practices and document lessons learned
Progressively deploy the software solution to the next wave of associations (Apr 2017 – June 2017)
‐ Use the "Ready-Set-Go" process to deploy the solution to additional associations
‐ Facilitate decision making and business process change
‐ Facilitate technical support between association IT staff and the vendor as appropriate
‐ Facilitate internal communications by supplying associations with communications tools
Validations of operations and service standards model (May 2017 – June 2017)
‐ Support the associations during the first 60 to 90 days of initial operations
‐ Concur on Service Standard agreements
Monitoring Report Online Competency Assessment (OCA) Project
10 of 12
‐ Gather feedback and refine the solution
The OCA project must keep all constituent associations informed of progress and of new requirements (if any) raised by the Federal Government
The project activity breakdown includes ongoing communications that will allow for progress reporting and to address any changes that may arise, as follows:
Change management and communications strategy (June 2015 – April 2017)
‐ Maintain pro-active communications to the associations over the entire lifecycle of the project (e.g. project reports, webinars and demonstrations)
The OCA project must make use of existing systems and tools that have been developed by the constituent associations, where possible
Since the original project plan was developed in 2013, APEGBC has successfully launched their competency‐based assessment tool for all applicants for engineering licensure. It appears that this platform will be able to support the Engineers Canada competency assessment model which differs from the model used at APEGBC. APEGBC is therefore the preferred vendor for the provision of the competency assessment tool. The ESDC funding agreement still requires a competitive bid process, however.
Engineers Canada has held three teleconferences with APEGBC (May 11, May 21, and July 14, 2015) to discuss the project.
Engineers Canada informed ESDC on June 12, 2015 that APEGBC is a potential supplier for this portion of the IT solution.
Monitoring Report Online Competency Assessment (OCA) Project
11 of 12
SECTION III – EL-5 Planning
EL‐5 The chief executive officer shall not permit planning that allocates resources in a way that deviates materially from Ends priorities, that risks fiscal jeopardy, or that is not derived from a multi‐year plan.
Chief Executive Officer Interpretation, in the context of the OCA project
The OCA project must establish a budget, and respect that budget. Deviations to the budget must be tracked and reported to ESDC and the Board.
Justification of Reasonableness
The OCA project must follow project management principles and track budget.
Compliance will be achieved when: Evidence of compliance:
The OCA project has a budget which is tracked and reported to ESDC and the Board.
• A high level project budget has been established and agreed to with ESDC.
• The project budget will continue to be refined as the project progresses and costs are known.
• Quarterly status reports to ESDC include progress, budget and schedule updates. These updates will be provided to the Board.
• Deviations to the budget will be reported to ESDC and to the Board.
Monitoring Report Online Competency Assessment (OCA) Project
12 of 12
SECTION IV – EL-7 Communication and Support to the Board
EL‐7 The chief executive officer shall not permit the Board to be uninformed or unsupported in its work.
Chief Executive Officer Interpretation, in the context of the OCA project
The Board must be informed of the project progress.
Justification of Reasonableness
The Board must be informed of the progress of the project, and any changes in direction or objectives.
Compliance will be achieved when: Evidence of compliance:
The Board will be informed of the project progress
Updates on the project progress will be included in every Board update.
The Board receives copies of the quarterly status reports sent to the Federal Government
Quarterly status reports will be submitted to the Board.
Resources
• Engineers Canada / ESDC funding agreement (attached) Adobe Acrobat
Document
• OCA project activities and schedule (attached) Adobe Acrobat
Document
• OCA project budget for portion funded by ESDC (attached) Adobe Acrobat
Document
• http://www.engineerscanada.ca/competency‐based‐assessment Certification
I certify that the above information is accurate. Kim Allen, FEC, P.Eng., July 21, 2015
