Quality Assurance and Peer Review

Professor Alan Paterson OBE

CPLS, University of Strathclyde

The ILAG Family 2019

Concepts and levels of Quality

➢Historically, synonymous with Excellence

➢Japanese manufacturers – quality as:

a) conforming to international standards, or

b) “fitness for purpose”, or

c) “value for money”.

➢Binary notion of quality replaced by a continuum.

The Spectrum of Quality

EXCELLENCE

COMPETENCE PLUS

THRESHOLD COMPETENCE

INADEQUATE PROFESSIONAL SERVICES

NON PERFORMANCE

Figure 1: The Performance Continuum

© Paterson & Sherr

Setting the Pass Mark

➢ Best practice

➢Good practice

➢ To pass a peer review assessment

➢ Fitness to practice – threshold competence. The discipline standard. Does this change with time?

➢ The negligence standard for specialists

➢ The negligence standard

➢Minimum requirement of Art 6 Jurisprudence ECHR

➢ To permit an “ineffective assistance of counsel” appeal

INPUTS OUTCOMES

STRUCTURE

PROCESS

Education

Qualification

Licensure

Experience

Knowledge

Skill sets

Property

Equipment

Library

Training

IT

Contacts

Management Allocation

Supervision

TrainingSystems

Performance

Advice

Correspondence

Decisions AdvocacyStrategy

Fact Gathering

Legal Analysis

Client Handling

Practice Management

Financial

Social

Public

Private Inaction

Action

Sentence

Damages

International Developments

UN Principles and Guidelines on Legal Aid (2012)

Principle 13

“All States to put in place mechanisms to ensure that all legal aid providers possess the education, training, skills and experience that are commensurate with the nature of their work “

• Criteria for accreditation of the legal aid providers (Guidelines 15);

• Quality standards for legal aid providers and paralegal (Guidelines 16 and 14);

• Adequate training, and supervision by qualified lawyers (Guideline 14 –mentioned in the context of paralegals only, though this would probably be appropriate for lawyers as well);

• Monitoring and evaluation mechanisms to assess the quality of the services provided by lawyers and paralegals (Guidelines 16 and 14)

International Developments II

EU Commission

Recommendation of 27 November 2013 on the right to legal aid for suspects or accused persons in criminal proceedings

Section 3 on effectiveness and quality of legal aid establishes that “Legal assistance provided under legal aid schemes should be of high quality in order to ensure the fairness of proceedings. To this end, systems to ensure the quality of legal aid lawyers should be in place in all Member States.”

This includes accreditation of legal aid providers and appropriate training; appointment of lawyers while taking into account the wishes of the accused and allowing accused to choose a provider from a list; and replacement of lawyers who fail to provide adequate legal assistance.

[ Input & Structure ]

EU Directive 2016/1919 of the European Parliament and of the Council of 26 October 2016 on legal aid for suspects and accused persons in criminal proceedings and for requested persons in European arrest warrant proceedings, establishes in Article 7 ‘Quality of legal aid services and training’:

1. Member States shall take necessary measures, including with regard to funding, to ensure that: (a) there is an effective legal aid system that is of an adequate quality; and (b) legal aid services are of a quality adequate to safeguard the fairness of the proceedings, with due respect for the independence of the legal profession.

[ Input & Structure ]

International Developments III

UN Global Study findingsImproving quality of legal aid services was identified as the number one priority for Member States in the Global Study on Legal Aid (UNODC,2016)Monitoring/ Evaluation mechanism of Quality of legal aid?Bar Association 35%Legal Aid Board 20%Ministry of Justice 20%None 11%However, 57% of responding states indicated that the mechanism used to assess quality was client complaints.Only 26% reported taking any proactive steps to assess quality – the most frequent measure being client satisfaction surveys.

Publication of the UNODC Handbook on Ensuring Quality of Legal Aid Services in Criminal Justice Processes (2019 )

➢ Global Study: Most pressing technical assistance need cited was “development of quality criteria for legal aid providers”

➢ To respond to this need, as a first step UNODC aided by Miri Sharon and Alan Paterson with 50 other experts from all regions developed a new Handbook to provide an overview and guide on measures to constantly improve and monitor the quality of criminal legal aid services

What we know about what is out there ?

Nikartas and Limante: Tools and Criteria for Measuring Legal Aid Quality: Guidelines for EU Member States (2018)

Accreditation and Training

Complaints

Client satisfaction

Peer Review

“The evaluation of the serviceprovided against specified criteria and levels of performanceby an independent lawyer with significant current or recent practical experience in the areas of law being reviewed”

Peer Review

The Spread of Peer Review England and Wales, Scotland, Canada (Ontario), Finland, Georgia, Moldova, South Africa, Chile, China,

New Zealand, Ukraine, The Netherlands and Australia

14

MECHANISMS OF CONTROL AND MEASUREMENT

• Peer review,

• External Audits,

• Claims

• Semiannual and Final Reports

In addition, the Public Criminal Defense has other measurements and instruments

that help to control the service, evaluate its performance and account for it.

INDICATORS OF CRIMINAL DEFENSE DELIVERY

EXTERNAL AUDITS OF USER SATISFACTION

GLOBAL QUALITY INDEX

INTEGRATED DATA ANALYSIS

BUSINESS INTELLIGENCE TOOLS

Lessons from Chile

Peer Review SystemGeneral flow chart

Annual plan

Peer reviewsschedule

Preliminaryreview

Fieldwork, sample and

analysis

ReportObjections

Districtsupervision

Follow up

Evaluation

The aim of Peer Review in the UK

• To enhance efficiency and effectiveness?• To assess the quality of performance and

outcomes achieved by legal aid providers?• To reassure the public and the state as to quality

and value for money?• To enhance choice for the public?• To exclude poor performers from the list of legal

aid providers?• To attain continuous improvement by providers?• To change the culture of legal aid providers?

Competing Perspectives

• Appointed by panel of laypersons and lawyers after an open recruitment process.

• Peer reviewers are experienced solicitors.

• Peer reviewers receive training for 3 to 4 days on reviewing and marking files according to the criteria.

• Training fosters consistent application and interpretation of the criteria.

• Once accepted as peer reviewers, they have to go through a peer review themselves.

• Cannot review a competitor firm or one they have links to.

Scottish Peer Reviewers

File selection and Firm selection

• 5 random files (OR 10% of a lawyer's files where the client is likely to be vulnerable, adult incapacity; mental welfare or immigration cases ) for each lawyer in the firm are assessed against 14 published peer review criteria. These include:

• Effectiveness of lawyer’s initial fact and information gathering skills.

• Effectiveness of communications.

• Whether the lawyer identified the need for appropriate experts or counsel.

• Evidence of adequate preparation for each stage of the case.

• Evidence of the lawyer taking steps agreed with the client with a reasonable time.

• Whether the case was concluded effectively.

Criteria are Client Centred

Peer Review Cases and Criteria - CIVIL

• 8 random files for each lawyer are assessed against published peer review criteria for less serious criminal cases (e.g. assault), serious criminal cases (robbery) and appeals. Criteria include:

• Whether the lawyer made an effective application for bail,

• Whether special defences were considered,

• Adequate preparation for trial,

• Advice given to the client on how to plea,

• Advice given to client about sentence,

• Advice give to client about appealing,

Criteria are client centred

Peer Review Cases and Criteria - CRIMINAL

Peer Review in less important Cr cases

Client contact at initial interview stage whether in custody or in an office 1. How effective were the solicitor’s initial fact and information gathering skills,

including identifying any additional information required and taking steps necessary to obtain it?

1 2 3 C N/A

This should also include consideration of any advice given to the client during a police interview at a police station, where appropriate.

2. Did the solicitor give the client correct and appropriate advice about the defence case including whether it is stateable and whether an early plea should be considered? Has a discount for an early plea been canvassed with the client?

1 2 3 C N/A

File and Lawyer Scoring

54321

EXCELLENCECOMPETENCE PLUS

THRESHOLD COMPETENCEPOOR

NON-PERFORMANCE

“That’s it? That’s Peer Review?”

Points of Good Practice

Good Terms of Engagement letters

Use of easy to understand language used in letters

Keeping clients well informed of the development of their cases

Sensitivity shown when dealing with child witnesses

Clear and easy to follow correspondence

Excellent initial letter of instructions - supplied master copy for

circulationFile keeping

Good court minutes endorsed on the front cover of the file

Detailed notes with a careful attitude to preparation

Use of a checking system

The solicitor carries out file reviews and records these

Where appropriate the solicitor will record comprehensively the

narration given by the Crown: this could be important in the event of

an Appeal

Communications

Points of Good Practice II

Good and detailed gathering of information

Early resolutions in cases benefitting both clients and the wider justice system

Application of S71 procedure to identify and resolve matters prior to trial in

Solemn cases

Copy legislation on file, copy case law on file

Solicitor negotiating the discontinuation of proceedings with Crown

Full and detailed notes taken after Full Committal where solicitor discussed

bail with client followed by careful notes taken upon sentence in case of

appeal

Legal work

The application for legal aid was very detailed and demonstrated that the solicitor had a detailed knowledge of the client’s background and relevant issues.

Legal aid

Areas for Improvement

Client meetings being held too infrequently

Style and contents of letters poor

Sparse instructions to agents

Letters to clients containing wrong dates and incorrect information

Lack of letters to clients updating them that disclosure was available to discuss

Communications

Conflict of interest when acting for accused and co-accused

Failure to attend to conduct a trial

Absence of S196 early plea discount

Limited information obtained regarding the defence.

Defence witnesses not cited when they should have been

Legal work

Areas for Improvement II

Conflicting information about client's financial circumstances (eg working, not working)

Lack of vouching of client’s financial circumstances for the legal aid application

Legal aid

Files not being kept chronologically

Papers being kept loose leaf

Lack of printed LA applications on file to see what had been submitted to the Board

File keeping

Lessons from Peer Review

Quality is not binary

The pursuit of Quality can have different objectives

Quality, Peer Review and the independence of the profession

Quality, Peer Review and client confidentiality

Peer Review and expense

The Quality floor is a dynamic rather than a static target

Lessons from Peer Review

Quality is not binary

The pursuit of Quality can have different objectives

Quality, Peer Review and the independence of the profession

Quality, Peer Review and client confidentiality

Peer Review and expense

The Quality floor is a dynamic rather than a static target

Lessons from Peer Review

Quality is not binary

The pursuit of Quality can have different objectives

Quality, Peer Review and the independence of the profession

Quality, Peer Review and client confidentiality

Peer Review and expense

The Quality floor is a dynamic rather than a static target

Lessons from Peer Review

Quality is not binary

The pursuit of Quality can have different objectives

Quality, Peer Review and the independence of the profession

Quality, Peer Review and client confidentiality

Peer Review and expense

The Quality floor is a dynamic rather than a static target

• Scots cost of QA in 2013/14 was 220, 000 Euros (2.3% of Scots expenditure on administration) ( 123 Euros per file)

• (Complaints 2016/17 2,582 Euros per case) .

Costs and Benefits

Lessons from Peer Review

Quality is not binary

The pursuit of Quality can have different objectives

Quality, Peer Review and the independence of the profession

Quality, Peer Review and client confidentiality

Peer Review and expense

The Quality floor is a dynamic rather than a static target

Quality Targets are Dynamic

1) Technological advances and innovation mean that the requirements of good practice change e.g. the digital discovery of evidence, the probabilistic basis of DNA profiling and the hidden biases of AI algorithms

2) Twenty years ago the model was “the professional knows best”. Patient care and client communication were not person centred but professional centred. Today, case law and professional ethics requires professionals to be client centred – the bar has clearly been raised.

2012-19 File Stats All ReviewersMarker FM F % F D % D

T Allan 550 29 5.3 - 92 16.7

S Brown 698 16 2.3 124 17.8

I Cruikshank 273 20 7.3 22 8.0

K Douglas 638 47 7.4 60 9.4

C Dunipace 701 23 3.3 82 11.7

T Gallanagh 571 17 3.0 46 8.1

D Henderson 407 67 16.5 29 7.1

M Hutchinson 408 10 2.5 14 3.4

R Lindsay 225 26 11.6 35 15.6

F Mccready 220 13 5.9 22 10

I MccLelland 314 9 2.9 12 3.8

J McKay 608 86 14.1 99 16.3

L McPhie 129 4 3.1 9 7.0

P Ralph 353 46 13 66 18.7

J Rafferty 138 3 2.2 30 21.7

G Robertson 666 8 1.2 109 16.4

A Ross 568 40 7 97 17.1

S Walker 417 18 4.3 57 13.7

G Wiggins 777 89 11.5 78 10.1

R Yuill 146 1 0.7 18 12.3

Total 8,661 (11,307) 571 1,084

Overall Av. 433 28.6 6.6% 54.2 11.7%

Civil 3rd Cycle review scoresReviewer PM FM %F %D

1 69 338 13.3 16.3

2 70 365 10.7 22.5

3 32 172 2.9 23.8

4 69 356 8.4 13.8

5 51 235 10.2 17.0

6 118 626 12.0 19.5

7 126 656 8.5 16.5

8 124 607 14.3 12.7

9 65 316 16.5 6.0

10 89 458 10.5 8.5

11 175 1009 8.4 11.9

12 129 642 17.1 8.9

13 98 565 10.3 11.2

14 48 259 6.6 24.3

15 81 369 18.2 29.0

16 53 264 12.9 9.9

17 73 334 11.7 13.5

18 39 251 7.6 12.0

19 103 596 6.9 15.9

20 44 189 16.9 9.5

21 101 567 19.6 5.1

22 108 555 15.5 15.7

TOTALS 1865 9729

Overall Av 84.8 442.2 11.9 % 14.1%

• Evidence that peer review is driving up standards.

• Fail rate has reduced on criminal.

• Numbers of Final Reviews and Special Reviews has fallen on civil.

• Numbers of solicitors achieving the highest marking has increased.

• Survey of solicitors in 2013 showed that 84% of respondents had a positive or neutral opinion on whether the QA scheme was an effective way of ensuring quality.

• A few solicitors who were doing very small amounts of legal aid, or had failed reviews, withdrew from our register of suppliers. The great majority of lawyers/firms that fail their initial review pass the next time round

Driving up Standards?

Practice Unit Identified

8 Files/ Practitioner Selected

Reviewer(s) Selected

Practice Unit Informed

Routine File Review Pass Fail

Extended Review

Or

Deferred Extended

Review

Pass Fail

Final Review

(3/6 Months) Pass

Fail

SLAB Notified

Consider Withdrawing

Compliance Certificate

Compliance

Certificate

Issued

Monitoring of

Practice Unit

Special Review

Compliance

Certificate

Issued

Compliance

Certificate

Issued

Continued for

Comment

from Practice

Unit

The Role of the Quality Assurance Committee

1) All Scots assessments are reported to a Quality Assurance Committee (QAC ) or Criminal Quality Assurance Committee (CQAC). Each comprises of:

3 solicitors appointed by the Law Society

3 non-legal members (representing the interests of the consumer)

3 members appointed by SLAB

2) Meets monthly to consider all summary reports andfailed file reports from peer reviewers. It is the QAC , not the reviewer to decides whether a practitioner or firm passes or not.

The Role of the CQAC a) Pass ( with feedback ) provider with a good or average grade.

Result: Not reviewed again for several years;b) Award a marginal pass with detailed feedback

Result: Reviewed again in 12-18 months c) Continue consideration of the lawyer for comment from the lawyer as to the report of the reviewer including flaws in earlier reviews;

Result : Lawyers expected to address past flaws even if passed last review – continuous improvement

e) Award a marginal fail with feedback and order a “deferred extended review” in nine months –

Result: Lawyers given early chance to improvef) Award an immediate fail and order an immediate “extended review”

Result: Public protected from really poor providersg) Order an immediate special review;

Result: Public purse protected

CRIMINAL CIVIL

2012-2019 2012-2018

Registered firms 613 ( 454 ) 622

Registered practitioners 1,655 ( 687 ) [Some do no legal aid or have no files ]

Reviews undertaken 1149 602

Reviews continued re comments 122 (11%) 299 (48%)

Extended Review / deferred ER 6/69 ( 7%) 21/35 (9%)

Practitioners to final 8 (1%) 12 (2%)

Special reviews 4 4

Marginal passes 111 (10%) ??

Quality Assurance Scheme Peer Reviews

Solicitor 1 2(a) 2(b) 2(C) 3(a) 3(b) 3(C) 3(d) 4 5 6 7(a) 7(b) 8 9(a) 9(b) 10 11 12 13(a) 13(b) 14 Overall

I MCCulloch (KD) 2 2 2 1 1 2 2 2 2 2 N/A 2 2 2 2 2 1 1 C 2 2 2 2

2 C 2 2 1 2 2 N/A 1 2 N/A N/A N/A 2 2 2 1 1 N/A 2 2 2 2

3 3 2 2 1 2 2 2 3 3 N/A N/A N/A 3 2 2 3 3 N/A 2 2 2 4

2 C 2 1 1 2 2 2 2 2 N/A 2 N/A N/A 2 C 2 C N/A N/A N/A N/A 2

2 2 1 2 2 2 2 N/A 2 2 N/A N/A N/A N/A 2 2 2 2 N/A N/A N/A N/A 3

G McIntyre (KD) 2 2 2 2 2 2 2 N/A 2 2 N/A N/A N/A N/A 2 2 2 2 2 2 N/A 2 3

2 2 2 2 2 2 2 N/A 2 2 2 N/A N/A N/A 2 2 2 2 2 N/A N/A N/A 3

2 2 2 2 2 2 N/A N/A 2 2 N/A N/A N/A N/A N/A N/A 2 2 N/A N/A N/A N/A 3

3 2 2 2 2 2 2 2 3 3 N/A N/A N/A N/A 2 3 3 3 3 N/A N/A N/A 4

2 2 2 2 2 2 2 2 2 2 N/A N/A N/A 2 2 2 3 3 N/A 2 2 2 4

A Mackie (KD) 2 2 2 2 2 2 2 3 3 3 N/A N/A N/A 2 2 2 2 3 2 2 2 1 3

2 2 2 2 2 2 2 N/A 2 2 N/A N/A N/A N/A 2 2 2 1 N/A 2 N/A 2 3

2 2 2 2 2 2 2 2 2 2 N/A N/A N/A N/A 2 2 2 2 2 N/A N/A N/A 3

2 2 2 2 2 2 2 N/A 2 2 2 N/A N/A N/A 2 2 2 2 N/A 2 N/A C 3

2 2 2 2 2 2 2 N/A 2 2 2 N/A N/A N/A 2 2 2 2 N/A 2 N/A 2 3

Case

Report

Analysis

Total Noper criteria 1 2(a) 2(b) 2(c) 3(a) 3(b) 3(c) 3(d) 4 5 6

7(a)

7(b) 8 9(a) 9(b) 10 11 12 13(a) 13(b) 14

1 43 58 290 237 638 25 10 21 196 53 32 6 66 25 199 98 227 167 13 83 14 134

2 5920 6004 5646 4547 4680 6385 5647 4215 5359 5174 3431 1649 3697 2326 4490 5068 5514 5301 2153 4092 1433 3702

3 548 298 14 33 122 53 68 156 415 515 73 63 22 196 45 369 275 415 96 8 22 10

C 170 308 494 759 1107 133 85 72 105 57 139 44 221 41 226 188 94 201 79 445 85 583

N/A 19 31 256 1125 147 99 890 2237 626 901 3023 4939 2695 4113 1741 973 591 616 4360 2070 5146 2266

Overall CR file scores 2012-19

1 43 (0.4%)

2 653 5.8%)

3 9,281 (82.1%)

4 1,271 (1.2%)

5 60 ( 0.5%)

Criminal Peer Review E & WApril 2017 – March 2018 April 2018 – March 2019

•The most common rating awarded to Crime providers in 18/19 was ‘Threshold Competence’ (3). This was consistent

with provider performance reported in 17/18.

Rating

Number

of

providers

%% Below

Competence

5 16 3.48%

4 159 34.57%

3 228 49.56%

2 57 12.39%12.39%

1 0 0%

Total460

Ratings

Number

of

provider

% Below

Competence

5 11 2.18%

4 189 37.43%

3 256 50.69%

2 48 9.5%9.7%

1 1 0.20%

Total 505

• Typical causes of fails:• Delays in taking action or applying for legal aid

• Poor communication with clients relating to the operation of the costs rules for legally aided persons

• Poor file notes of phone call or interviews

• No terms of engagements letters on files.

• Occasional examples of abuse:• Private charges to legally aided clients or padding of

accounts.

Results from peer reviews

Running a pilot programme

1) Appoint a body to oversee the pilot programme partnership

2) Identify the standards of criminal practice 3) From these derive the peer review criteria for

assessing process and outcome4) Identify the assessment system5) Run a selection competition for peer reviewers6) Decide on what should be assessed – performance in

court, files7) Decide on whether additional factors should be taken

into account e.g. client satisfaction; complaints