Upload
others
View
5
Download
0
Embed Size (px)
Citation preview
© 2019 National Association of Insurance Commissioners 1
CASUALTY ACTUARIAL AND STATISTICAL (C) TASK FORCE Casualty Actuarial and Statistical (C) Task Force Apr. 6, 2019, Meeting Minutes Casualty Actuarial and Statistical (C) Task Force Mar. 22, 2019, Conference Call Minutes (Attachment One) Casualty Actuarial and Statistical (C) Task Force Mar. 12, 2019, Conference Call Minutes (Attachment Two) Casualty Actuarial and Statistical (C) Task Force Feb. 12, 2019, Conference Call Minutes (Attachment Three) Comment Letter on the Statement of Actuarial Opinion Instructions (Attachment Three-A) Comments on Best Practices for Regulatory Review of Predictive Analytics White Paper (Attachment Three-B) Casualty Actuarial and Statistical (C) Task Force Jan. 29, 2019, Conference Call Minutes (Attachment Four) Casualty Actuarial and Statistical (C) Task Force Jan. 8, 2019, Conference Call Minutes (Attachment Five) Casualty Actuarial and Statistical (C) Task Force Dec. 18, 2018, Conference Call Minutes (Attachment Six) W:\National Meetings\2019\Spring\TF\CasAct\contents.docx
Draft Pending Adoption
© 2019 National Association of Insurance Commissioners 1
Draft: 4/18/19
Casualty Actuarial and Statistical (C) Task Force Orlando, Florida
April 6, 2019 The Casualty Actuarial and Statistical (C) Task Force met in Orlando, FL, April 6, 2019. The following Task Force members participated: Steve Kelley, Chair, represented by Phil Vigliaturo (MN); James J. Donelon, Vice Chair, represented by Rich Piazza (LA); Lori K. Wing-Heier represented by Michael Ricker (AK); Keith Schraad represented by Erin Klug (AZ); Ricardo Lara represented by Lynne Wehmueller (CA); Andrew N. Mais represented by Wanchin Chou (CT); Stephen C. Taylor represented by David Christhilf (DC); David Altmaier represented by Sandra Starnes (FL); Colin M. Hayashida represented by Gerald Hew (HI); Doug Ommen represented by Travis Grassel and Andria Seip (IA); Robert H. Muriel represented by Judy Mottar (IL); Vicki Schmidt represented by Nicole Boyd (KS); Eric A. Cioppa represented by Sandra Darby (ME); Anita G. Fox represented by Kevin Dyke (MI); Chlora Lindley-Myers represented by Cynthia Amann (MO); Mike Causey represented by Kevin Conley (NC); Marlene Caride represented by Mark McGill (NJ); Barbara D. Richardson represented by Stephanie McGee (NV); Jillian Froment represented by Thomas Botsko (OH); Glen Mulready represented by Andy Schallhorn (OK); Andrew Stolfi represented by TK Keen (OR); Jessica Altman represented by Michael McKenney (PA); Raymond G. Farmer represented by Michael Wise (SC); Kent Sullivan represented by J’ne Byckovski (TX); and Mike Kreidler represented by Eric Slavich (WA). 1. Adopted its March 22, 2019; March 12, 2019; Feb. 12, 2019; Jan. 29, 2019; Jan. 8, 2019; Dec. 18, 2018; and 2018 Fall
National Meeting Minutes Mr. Vigliaturo said the Task Force met March 22, 2019; March 12, 2019; Feb. 12, 2019; Jan. 29, 2019; Jan. 8, 2019; and Dec. 18, 2018. During these meetings, the Task Force took the following action: 1) adopted statistical reports; and 2) adopted a comment letter to the Executive (EX) Committee’s ad hoc group regarding the Statement of Actuarial Opinion instructions related to the definition of Qualified Actuary. The Task Force also met March 19, 2019, in regulator-to-regulator session, pursuant to paragraph 3 (specific companies, entities or individuals) of the NAIC Policy Statement on Open Meetings, to discuss rate filing issues. The Task Force viewed a Casualty Actuarial Society (CAS) livestreaming event March 26, 2019, in lieu of its typical Predictive Analytics Book Club conference call. The CAS provided access to its livestreaming event from its Ratemaking, Product and Modeling (RPM) Seminar, which included the following sessions: Machine Learning and Artificial Intelligence; The Predictive Modeling Cooking Show; The Changing Role of the Actuary in the Face of Disruption; Insurance Claims Prevention Using New Predictive Analytics; and Auto Insurance: What Happened and What Happens Next? Mr. Dyke made a motion, seconded by Mr. Piazza, to adopt the Task Force’s March 12, 2019 (Attachment One); Feb. 12, 2019 (Attachment Two); Jan. 29, 2019 (Attachment Three); Jan. 8, 2019 (Attachment Four); Dec. 18, 2018 (Attachment Five); and Nov. 15, 2018 (see NAIC Proceedings – Fall 2018, Casualty Actuarial and Statistical (C) Task Force) minutes. The motion passed unanimously. 2. Adopted the Report of the Actuarial Opinion (C) Working Group Mr. Vigliaturo said Julie Lederer (MO) provided a written report for the Actuarial Opinion (C) Working Group. The Working Group has not met in 2019. In mid-March, the Financial Examiners Handbook (E) Technical Group asked the Working Group to review the property/casualty (P/C) reserves and claims handling exam repository and provide feedback by May 31. Mr. Vigliaturo appointed Anna Krylova (NM) as vice chair of the Working Group. Ms. Mottar made a motion, seconded by Mr. Botsko, to adopt the report of the Actuarial Opinion (C) Working Group. The motion passed unanimously.
Draft Pending Adoption
© 2019 National Association of Insurance Commissioners 2
3. Adopted the Report of the Statistical Data (C) Working Group Mr. McGill said the Statistical Data (C) Working Group is working on the formulas in the Report on Profitability by Line by State, primarily regarding allocation of investment gain. Mr. Piazza made a motion, seconded by Mr. Botsko, to adopt the report of the Statistical Data (C) Working Group. The motion passed unanimously. 4. Discussed the Appointed Actuary Attestation of Qualification and the Three-Year Experience Requirement Proposals Mr. Vigliaturo said there was discussion during the March 22 hearing with the Executive (EX) Committee’s ad hoc group about the Statement of Actuarial Opinion instructions, written comments received and distributed edits. He said the changes the Task Force needs to consider for adoption relate to the attestation and experience charges. He said the other changes will be decided by the ad hoc group. The Task Force discussed the need to review its proposed changes within a document showing all proposed changes, whether the qualification documentation should include specific reference to companies’ insurance lines and scope of business, additional information to be included in instructions and/or in the annual regulatory guidance, and a concern about potentially rushing the project with lower quality instructions. The Task Force decided to meet via conference call in regulator-to-regulator session to consult with NAIC staff in order to gain a better understanding of how the Task Force’s proposed instructions fit with the ad hoc group’s proposed instructions. 5. Discussed the Predictive Analytics White Paper Mr. Piazza said the white paper was exposed for a 60-day public comment period ending Feb. 12. That exposure period was then extended another week until Feb. 19. The Task Force held two discussions during its Feb. 12 and March 12 conference calls. Following those calls, volunteer drafters mapped the comments to paragraphs of the paper to aid evaluation. Mr. Piazza asked for any comments on the 16 best practices. He said he would like to consolidate and reword the best practices. He suggested that the best practices and knowledge statements should be focused on the regulatory practice and not at the industry. Birny Birnbaum (Center for Economic Justice—CEJ) said the charge to the Task Force is limited to rate filings, but state insurance regulators should consider the review of pricing more generally. After marketing and underwriting, the data is already refined. If the company uses factors of concern to state insurance regulators—such as credit scoring, employment, occupation, etc.—then state insurance regulators should be concerned that the rate filing does not fully capture the pricing factors being used through underwriting tiers, etc. He said one of the fundamental areas of regulatory review is the data. State insurance regulators should ask about the data used to develop the model and judge whether the data is incomplete, biased or faulty in some way. He said that would be a best practice in review of the ratemaking model. Mr. Piazza said state insurance regulators will be looking at the data and evaluating whether there are biases in the data. He said they will not be evaluating how the companies may have targeted their products. Mr. Birnbaum clarified he does not suggest the review of models for marketing or underwriting in the scope of the charge; he said the scope to review ratemaking predictive models should include understanding the data used to develop, test and produce ratemaking results. Mr. Piazza agreed that bias is an issue identified in the knowledges. Ralph Blanchard (Travelers) said the data described by Mr. Birnbaum is not biased. He said data has a characteristic. He said life insurance companies do not use mortality trends of the U.S. population because that data does not match to its underwriting. He said the data for ratemaking must have the same characteristic underwriters are using to underwrite. Mr. Birnbaum said if the data describes the book of business and all attributes of those customers, then that would be fine. But if pricing factors have been applied in the underwriting process, then it is important to understand the situation, and companies should be disclosing that information. Mr. Keen asked if Mr. Birnbaum’s suggestion includes advising state insurance regulators to look at variables in the model, but also evaluating variables excluded from the model and the reason why. Mr. Birnbaum said to look for unfair discrimination state insurance regulators need to know if the ratemaking data has been refined, especially if underwriting or rating tiers were used. He said the underwriting tiers might include risk variables that cause regulatory concern.
Draft Pending Adoption
© 2019 National Association of Insurance Commissioners 3
David Kodama (American Property Casualty Insurance Association—APCIA) said the APCIA’s member companies suggest that the regulatory review of models be consistent in approach with other requirements in the rate filing process. He said state insurance regulators should not impose different standards and requirements. He said state insurance regulators should give proper deference to existing Actuarial Standards of Practice (ASOPs). He said the APCIA supports consistency but not the imposition of new standards onto the process. Mr. Piazza said the charge is to provide guidance to the states. He said the aim is to use standards that the states already have. He said many states need help to better understand and improve reviews. He said one potential result of the work might be for companies to provide information upfront in the filing rather than the states having to request information, which could eliminate numerous additional weeks in the review process. Mr. McGill said state insurance regulators would like companies to provide support of models similarly to how the companies meet standards to support traditional rate filings. Mr. Birnbaum added that state insurance regulators are not creating new standards. He said state insurance regulators are responding to companies’ new data usage and techniques by developing new regulatory tools and techniques in response. Mr. Piazza said the next steps will be for the drafting group to review the comments and provide suggested changes. Using the suggested changes, the white paper will be redrafted and exposed for comment. He said the process will likely result in the identification of some policy or other issues that are not actuarial. He said those issues may be brought to the attention of other NAIC groups. He said he expects the drafting and review processes might need to be repeated a couple of times, with an aim to submit the white paper to its parent committee and the Big Data (EX) Working Group by the Fall National Meeting. He said he would also like to be able to propose revisions to the Product Filing Review Handbook and state guidance at the same time. 6. Discussed NAIC Activities Regarding Casualty Actuarial Issues Ms. DeFrain said predictive analytics sessions will be held June 6–7 at the NAIC/NIPR Insurance Summit in Kansas City, MO. She said the training will include case studies. Mr. Grassel said Iowa’s Global Insurance Symposium (GIS) is being held April 23–25, with predictive analytics training for state insurance regulators being conducted in the days before and after the GIS. Mr. Chou said the NAIC is providing training to enhance the Own Risk and Solvency Assessment (ORSA) reviews on Section 3 of the ORSA, which includes economic capital models. 7. Heard Reports from Actuarial Organizations Kathleen C. Odomirok (American Academy of Actuaries—Academy) said the Academy’s Committee on Property and Liability Financial Reporting (COPLFR) educates practicing actuaries on regulatory requirements around the Statement of Actuarial Opinion. She said around 80 actuaries attended the annual seminar for opinion writers. Around 150 registrants participated in a follow-up webinar on selected topics, which was held Feb. 1 and focused on effective report writing. She said the COPLFR will be contacting state insurance regulators to help with updating the annual Loss Reserve Law Manual, updating the P/C practice note on Statements of Actuarial Opinion, and updating the practice note on risk transfer with respect to reinsurance contracts. Lisa Slotznick (Academy) provided a list of activities from other Academy’s Casualty Practice Council groups. She said the Property and Casualty Risk-Based Capital Committee works with the NAIC. The P/C Extreme Events and Property Lines Committee will soon be publishing a paper on wildfires, is working on a catastrophe bond paper and is working on flood insurance issues. Other groups are providing predictive analytics training at the NAIC/NIPR Insurance Summit in Kansas City, MO, and are working on cybersecurity, cyber insurance and the federal Terrorism Risk Insurance Act (TRIA) program. Ms. Slotznick said there is a lot of research being conducted regarding cyber, wildfires and a climate risk index. Godfrey Perrott (Actuarial Board for Counseling and Discipline—ABCD) discussed the ABCD’s investigations of complaints and requests for counseling. Kathleen A. Riley (Actuarial Standards Board—ASB) discussed exposure and adoption actions taken on various ASOPs. Information can be found on the ASB website.
Draft Pending Adoption
© 2019 National Association of Insurance Commissioners 4
Providing information on P/C actuarial research, R. Dale Hall (Society of Actuaries—SOA) presented the SOA’s general insurance actuarial research and education, and Mr. Blanchard presented the CAS’ P/C actuarial research. Information about this research can be found on the SOA and CAS websites. Having no further business, the Casualty Actuarial and Statistical (C) Task Force adjourned. W:\National Meetings\2019\Spring\TF\CasAct\4-6 CASTF Min.docx
Attachment One Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 1
Draft: 4/1/19
Casualty Actuarial and Statistical (C) Task Force Conference Call March 22, 2019
The Casualty Actuarial and Statistical (C) Task Force met via conference call March 22, 2019. The following Task Force members participated: Steve Kelley, Chair, represented by Phillip Vigliaturo (MN); James J. Donelon, Vice Chair, and Rich Piazza (LA); Lori K. Wing-Heier represented by Mike Ricker (AK); Jim L. Ridling and Daniel J. Davis (AL); Michael Conway represented by Mitchell Bronson and Sydney Sloan (CO); Andrew N. Mais represented by Wanchin Chou (CT); Stephen C. Taylor represented by David Christhilf (DC); Doug Ommen represented by Travis Grassel and Andria Seip (IA); Robert H. Muriel represented by Judy Mottar (IL); Vicki Schmidt represented by Nicole Boyd (KS); Eric A. Cioppa and Sandra Darby (ME); Anita G. Fox represented by Kevin Dyke (MI); Chlora Lindley-Myers represented by Julie Lederer (MO); Marlene Caride represented by Mark McGill (NJ); John G. Franchini represented by Mark Hendrick and Anna Krylova (NM); Mike Causey represented by Arthur Schwartz (NC); Barbara D. Richardson and Gennady Stolyarov (NV); Andrew Stolfi represented by David Dahl (OR); Jessica Altman represented by Kimberly Rankin (PA); Raymond G. Farmer represented by Will Davis (SC); Kent Sullivan represented by Nicole Elliott, Miriam Fisk and Walt Richards (TX); Mike Kreidler represented by Eric Slavich (WA); and James A. Dodrill represented by Joylynn Fix (WV). 1. Heard a Report on the Appointed Actuary Project Commissioner Ridling asked NAIC staff to provide some background on the Appointed Actuary Project. Kris DeFrain (NAIC) said, in 2017, the Executive (EX) Committee decided to revise the property/casualty (P/C) Statement of Actuarial Opinion (SAO) instructions to create an objective definition of a “qualified actuary” and make other changes in line with guidance presented by an NAIC consultant. To do this work, the Committee did three things: 1) appointed an ad hoc group of commissioners to oversee the Appointed Actuary Project (the ad hoc group now includes Commissioner Ridling, Superintendent Cioppa and Commissioner Donelon); 2) asked NAIC staff to work with actuarial organizations and volunteers to complete the project; and 3) assigned three charges to the Task Force.
Working with a consultant and numerous volunteers, NAIC staff completed the first phase of the Appointed Actuary Project: the Job Analysis Project. The output was a list of knowledge statements defining what a P/C Appointed Actuary needs to know and do to provide an expert actuarial opinion to accompany the annual statutory financial statement.
NAIC staff then began the second phase of the Appointed Actuary Project: the Educational Standards and Assessment Project. This second phase used the knowledge statements from the Job Analysis Project to develop minimum actuarial educational standards. These standards determine the syllabi and educational content of an actuarial education program to provide minimum basic education for a P/C Appointed Actuary. The project also documented numerous areas where the P/C actuary is expected to need additional experience and continuing education (CE) to do the Appointed Actuary job.
With those minimum standards in place, the NAIC is now assessing the P/C actuarial education offered by the Casualty Actuarial Society (CAS) and the Society of Actuaries (SOA). With a few months of the assessment remaining, the result due in mid-May will determine which actuarial credentials, under specific terms, will be accepted as the new NAIC accepted actuarial designations. These designations will form the foundation of the new “qualified actuary” definition in the SAO instructions. At the same time NAIC staff worked on these projects, the Task Force worked on its three charges related to the attestation, experience and continued competence. The result of the first two charges are included in the SAO instructions being discussed today. The work on competence will continue into 2019, as it does not require an immediate change to the SAO instructions.
Ms. DeFrain said the purpose of the hearing today is to invite comments on the revised SAO instructions. These revised instructions combine the proposals of the Committee’s ad hoc group and the Task Force. These instructions were released for a 60-day public comment period beginning Dec. 15, 2018, and ending Feb. 15, 2019.
The Committee’s ad hoc group discussed the written comments and agreed to make numerous changes in the SAO instructions to address a majority of the written comments. Those changes were made to the SAO instructions last week. Ms. DeFrain discussed that document briefly, as it is believed that it resolves many comments and clears the way for the agenda today.
Attachment One Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 2
Ms. DeFrain said the document provided in the materials for the conference call includes the insertion of comments to the side in order to explain where changes were made and, sometimes, where changes were considered and alternative action is to be taken. The change with the biggest impact is that one of the components on the definition of “qualified actuary” was removed. That former subpart read “has sufficient experience and knowledge obtained through basic education, continuing education, or experience to understand reserving for the company’s lines of business and business activities.” It was agreed that this would be removed from the definition, given that it would remain in the documentation for the attestation.
The top four remaining issues were identified and placed on the agenda. Ms. DeFrain said additional changes to the instructions might be made after the hearing. Ms. DeFrain said the agenda is also split between the ad hoc group’s proposals and the Task Force’s proposals. She referred to the initially proposed and exposed instructions for the identification of which group proposed which changes. Commissioner Ridling agreed that the purpose of the hearing is to hear oral comments to decide whether to make additional changes to the SAO instructions. 2. Heard Comments on the Proposal to Re-expose the SAO Instructions After Completion of the Educational Standards and
Assessment Project Ms. Mottar, Ms. Lederer and Mr. Vigliaturo said they and the Task Force support the re-exposure of the SAO instructions after the NAIC completes the CAS and SOA’s actuarial examination assessments and determines any restrictions on the actuarial designations. 3. Heard Comments on the Proposal to Require Academy Membership in the Qualified Actuary Definition Mary Miller (American Academy of Actuaries—Academy) said it is simpler to require “a member of the Academy” than the current professionalism wording used. She said that while the CAS and SOA are international bodies, the Academy is the U.S. actuarial association and home of professionalism in the U.S. She said almost every other country would require membership in their countries’ associations to do statutory work in that country. Mr. Dyke said the Task Force outlined arguments in favor and against including Academy membership as a requirement. He said he supports Academy membership as a requirement in the definition. He said the Academy does a lot of work for the NAIC and to support the overall profession, particularly in respect to the qualification of actuaries. He said the Academy is the U.S. national association and it would be simpler to refer to membership in the Academy than the way it is written now. He referred to the Task Force’s letter. Ralph Blanchard (Travelers) expressed concern regarding the Casualty Practice Council reviewing qualifications of non-members. He said he is not comfortable with the Academy playing a quasi-regulator role. Ms. Miller said the Academy would not review qualifications of anyone who is not a member of the Academy. Mr. Stolyarov said the Task Force agreed on a proposal to add a qualifier to add Academy membership in order for the Casualty Practice Council to review. Mr. Schwartz said an actuary does not have to be a member of the Academy to be held to the Actuarial Standards of Practice (ASOPs) and the Actuarial Board of Counseling and Discipline (ABCD). Mr. Stolyarov said the Task Force identified reasons why Academy membership should not be a requirement. He said the professionalism framework applies to actuaries regardless of Academy membership. Academy membership is not a current requirement in the SAO instructions. Academy membership has many benefits, but making it a requirement has a cost of $675 in membership dues. Mr. Stolyarov said adding Academy membership is beyond the scope of the current project to create an objective definition. He said the vast majority of appointed actuaries are U.S. residents, so state insurance regulators should not require Academy membership for all actuaries when there might be an issue with a non-U.S. resident. He referred to additional points in the comment letter.
Attachment One Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 3
Commissioner Ridling said the definition of “qualified actuary” must be harmonized before the states are harmonized. 4. Heard Comments on the Proposal to Modify the Grandfathering Clause Mr. Stolyarov said the Task Force agreed to propose revised wording for the grandfathering clause to note that the restrictions that would apply to designations would also apply in grandfathering. He said the grandfathering clause would be clearer without changing the spirit of what is intended. Ann Weber (SOA) said a note about the restrictions seems fair and makes sense. She said any changes to the way grandfathering is written needs careful consideration for fairness. She said alternative language proposed during Task Force discussions would be unfair because CAS actuaries that came through before the end date would be deemed qualified, whereas SOA actuaries would not. She said the current process is intended to be fair to all actuaries. Mr. Blanchard said his concern with grandfathering seems premature to comment before the assessment is complete. Commissioner Ridling said that is in line with those who have requested a re-exposure. Mr. Stolyarov said it is important not to deprive an actuary solely because of the date the designation is inferred. He said the syllabi are updated continuously and that will continue. Ms. Lederer expressed concern regarding whether the assessment will find the designation deficient in certain areas. She said she does not believe it appropriate to grandfather the designation when it is shown to not meet minimum standards. Ms. Miller said the qualification standards do provide a method for people to become qualified if there are gaps in their education. Mr. Stolyarov said that if there are some deficiencies in the syllabus, there is no way to say an individual did not obtain the education through another method. He said a preemptive bar, because of the timing of completion, seems punitive. 5. Heard Comments on the Commissioners’ Power to Accept Non-Qualified Actuaries as Appointed Actuaries
Ms. Miller said the term “Qualified Actuary” in the SAO instructions has been changed to “Appointed Actuary” because of the definition change in order to state that a commissioner can approve someone to be an Appointed Actuary even if that person is not a Qualified Actuary. She said the instructions cannot give or take away a commissioner’s authority, but past drafters did not want to emphasize this option because it seems to weaken the work to define “Qualified Actuary.” Mr. Stolyarov said the issue did not receive much discussion, but he favors keeping the wording as-is. He said this is a rare situation. He said it is helpful to have this explained for commissioners to understand and to not have it hidden in other parts of the instructions. Ms. Miller said commissioners have the power without it being encouraged in the instructions. Commissioner Ridling questioned whether it would be better to not address that particular issue. Mr. Schwartz suggested changing the description from “insurance regulatory official” to “commissioner.” He also expressed concern that there would be a verbal approval. Mr. Piazza pointed out that the restrictions require written approval. He said that if the sentence is retained in the instructions, then the term “insurance regulatory official” should be modified to explain who that is. Commissioner Ridling agreed.
6. Heard Comments on the Attestation and Experience Proposals Mr. Schwartz expressed concern that the documentation of qualifications related to the lines of business and company structure is too restrictive. He said actuaries must do the “look in the mirror” test. Ms. Miller said there is little help to a board of directors determine whether the actuary is qualified. She said an earlier version of the SAO instructions referenced the Academy’s attestation document. Otherwise, there is quite a burden placed on the board of directors and state insurance regulators. She agreed that discussing the lines of business and company structure will be
Attachment One Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 4
confusing to boards. The qualification standards already require the actuary to have the experience and qualifications to do the work. She added that the paragraph should use the word “attestation.” Mr. Stolyarov said he has no opinion on what the qualification documentation should be called. Regarding the documentation requirement, he said it does not need to be annual because once experience is attained, it cannot be unattained. He said he reviews qualifications of actuaries and has clear instructions. State insurance regulators need to be able to review the documentation as they see fit. Ms. Miller said she believes the documentation should be in the board of directors’ documentation and not in the actuarial report. The ad hoc group will meet to determine next steps before the CAS and SOA assessments are completed. The Task Force will discuss its proposed instructions at the Spring National Meeting. Having no further business, the Casualty Actuarial and Statistical (C) Task Force adjourned. W:\National Meetings\2019\Spring\TF\CasAct\3-22 CASTF min.docx
Attachment Two Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 1
Draft: 3/29/19
Casualty Actuarial and Statistical (C) Task Force Conference Call March 12, 2019
The Casualty Actuarial and Statistical (C) Task Force met via conference call March 12, 2019. The following Task Force members participated: Steve Kelley, Chair, represented by Phillip Vigliaturo (MN); James J. Donelon, Vice Chair, represented by Rich Piazza (LA); Lori K. Wing-Heier represented by Mike Ricker (AK); Jim L. Ridling represented by Daniel J. Davis and Jerry Workman (AL); Ricardo Lara represented by Mitra Sanandajifar and Lynne Wehmueller (CA); Michael Conway represented by Sydney Sloan (CO); Andrew N. Mais represented by Susan Andrews, Qing He and Wanchin Chou (CT); David Altmaier represented by Howard Eagelfeld and Sandra Starnes (FL); Doug Ommen represented by Travis Grassel (IA); Robert H. Muriel represented by Judy Mottar (IL); Vicki Schmidt represented by Nicole Boyd (KS); Eric A. Cioppa represented by Sandra Darby (ME); Anita G. Fox represented by Kevin Dyke (MI); Chlora Lindley-Myers represented by Gina Clark, Brent Kabler and Julie Lederer (MO); Marlene Caride represented by Mark McGill (NJ); John G. Franchini represented by Anna Krylova (NM); Barbara D. Richardson represented by Gennady Stolyarov (NV); Jillian Froment represented by Tom Botsko (OH); Glen Mulready represented by Nicolas Lopez (OK); Andrew Stolfi represented by David Dahl and Ying Liu (OR); Raymond G. Farmer represented by Will Davis (SC); Kent Sullivan represented by J’ne Byckovski, Brock Childs, Nicole Elliott, Miriam Fisk, Eric Hintikka and Elizabeth Howland (TX); Mike Kreidler represented by Eric Slavich (WA); and James A. Dodrill represented by Joylynn Fix (WV). Also participating was: Gordon Hay (NE). 1. Discussed Comments Received on the Best Practices for Regulatory Review of Predictive Analytics White Paper Mr. Vigliaturo said comments were received on the draft white paper and attached to the Task Force’s Feb. 12 minutes (see NAIC Proceedings – Spring 2019, Casualty Actuarial and Statistical (C) Task Force, Attachment Three). He said anyone who submitted written comments would be allowed to present those during this conference call. Adam Pichon (LexisNexis) said Lexis Nexis supports some standardization, such as checklists, to provide filing information for state insurance regulators to review models. One area of concern is that the white paper is focused on generalized-linear models (GLMs), yet there are other models that are starting to be accepted for use. He said there should not be so much rigor that the “art” side of modeling, such as the evaluation that attributes have plausible causality or rationally make sense, is hindered. He said another example occurs when running a GLM with small amounts of data and evaluating p values. He said some variables are known to be predictive even if the p value is greater than 5%, which is a generally accepted threshold for p values. He said confidentiality to protect models and techniques used to develop the model is important. Mr. Vigliaturo said the Task Force’s intent is to limit the scope by focusing on only GLMs for private passenger auto and homeowners. He said this was decided in order to make progress on the work. Mr. Hay said his comments concern the definition of “unfairly discriminatory,” and the states needing their own definition. Richard Gibson (American Academy of Actuaries––Academy) said the Academy would like to see more explicit recognition of Actuarial Standards of Practice (ASOPs). Michael Woods (Allstate) said Allstate supports a more consistent framework. He said Allstate believes in transparency and attempts to pre-empt as many comments as possible. He suggested removing the request to provide intuitive arguments for why a rating variable is related to an outcome because it conflicts with ASOP No. 12 (Rating Classification). Alternatively, state insurance regulators could request that the Actuarial Standards Board (ASB) change ASOP No. 12. He said state insurance regulators should not try to reproduce the model, but the white paper should focus on the review of procedures and outcomes, rather than on rebuilding the model. He said the raw data is considered proprietary, noting that there are security risks to sending raw data. The white paper places a large emphasis on state-specific results, yet most rating plan models are built using multistate datasets for credibility purposes. The white paper seems to emphasize univariate indications, but multivariate indications are more accurate. He said some sections ask for earned premium, but that would require bringing all premiums to a common basis. He suggested that the Task Force focus on the goal of the white paper to help states with limited resources. He said the current framework creates a large scope and would make the review more difficult for the states with limited resources.
Attachment Two Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 2
David Kodama (American Property Casualty Insurance Association—APCIA) said the APCIA supports the effort. He said the APCIA wants a product to clearly outline and describe priority areas for state insurance regulators. The APCIA believes that best practices could foster comprehensive upfront dialogue between the filing company and state insurance regulators that support an efficient and effective review appropriately focused on ensuring compliance with the applicable statutory or regulatory rating standards. He said the regulatory reviews need to allow insurers to innovate and make improvements in risk-assessment practices. Mr. Kodama said there are five key priority areas for the AICPA: 1) use practices developed and employed by state insurance regulators and leverage ASOPs; 2) do not extend the statutory scope of the filing review process beyond rates being “not excessive, inadequate nor unfairly discriminatory” (i.e., do not create a requirement for rating variables to adhere to some subjective intuitive relationship standard); 3) do not create a “one-size-fits-all” restrictive checklist; 4) identify essential information to be included in the filing to reflect the priority areas to the state insurance regulator in determining how the model impacts the methodology and output (i.e., there will be cases where more information is required, but the APCIA urges that the current number of elements or information required should be significantly reduced); and 5) protect insurers’ financial and intellectual property. Ms. Wehmueller said California provided technical edits and supports development of the white paper. She said it might be worthwhile to emphasize the purpose of the white paper to be a guide and that state insurance regulators will not be taking the 92 items as a requirement in their review of models. She asked if there is a way to pare down the list to the priority items for the states that do not have a large staff to review these models. Patrick Foltyn (The Cincinnati Insurance Companies) said the rules seem prescriptive, noting that a principle-based approach would be more prudent. He said flexibility is needed. Robert Curry (Insurance Services Office—ISO) said the ISO has made rate filings with GLMs for 10 years, and no state insurance department has ever requested all the information included in the white paper; as such, he said some streamlining is recommended. He said there is some concern about the data underlying the analysis required; i.e., the data is proprietary and could have personally identifiable information. He asked whether state insurance regulators need the data, given that traditionally the policy-level information to trace through an analysis has never been requested. The focus should be on the filing and what model is being used. The state insurance regulator should not be looking to see if a better model could have been developed by a different modeler. He said the Task Force appears to be trying to extend federal Fair Credit Reporting Act (FCRA)-type protections to non-FCRA-type data. ASOP No. 12 does not require an actuary to establish a cause-and-effect relationship. He said it is good to attempt to do that, but it should not be a requirement, given that it is not always possible to do so. Mr. Kabler said data-mining and similar techniques differ from the traditional scientific method. With data-mining, there is not necessarily a known causal understanding or a hypothesis to test. The relationships and what they mean are not known. They may go through thousands of variables, resulting in many false positives. The p value will say the relationship is not likely random even when it is random. He said the ability to explain causality is greatly diminished. He said there is a causal relationship between youth and driving; however, no one knows the causal explanation for credit scoring. He said ASOPs raise the issue of causality. He said it is not that actuaries should dispense with causality entirely; only that actuaries are not required to prove causality. He said random correlations are going to make their way into rating systems, and this merits more attention in the white paper. He said the American Statistical Association (ASA) has discussed these issues, and it recommends not using data-mining. He said he does not recommend that, because causality is not the primary concern. Mr. Davis said, in the rush to market, some of these variables may be wrong; therefore, they could easily be unfair. Mr. Kabler said there may be remedial measures that could be incorporated into the review process, such as the adjustment of p values to account for having many variables and using hold-out datasets. Michelle Rogers (National Association of Mutual Insurance Companies—NAMIC) said the white paper is thorough, but it might be getting too prescriptive with the large number of items of requested information. She said there needs to be an emphasis in the white paper on flexibility. She said the cost of rate filing reviews could get too costly, impact speed to market and innovation and, ultimately, impact consumers. She suggested identifying what is truly essential in every rate review, eliminating those that are unnecessary, and identifying other information that could be requested if the need is demonstrated.
Attachment Two Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 3
Mr. Stolyarov said consistency across the states, rather than uniformity, should be the goal. He said the states have common areas of focus. He said the states are not going to request all the information listed. He said the correlation-only argument is a misunderstanding of ASOP No. 12. While ASOP No. 12 says it is not necessary to establish a cause-and-effect relationship, an actuary should still select risk characteristics that are related to expected outcomes. He said senior modelers recognize a need to consider causality, why a variable might be predictive and/or what behaviors the variables are trying to represent. He said the use of large datasets will result in false positives and treatments that do not make sense. He added that the states protect confidentiality. He said some items do not make sense, such as penalizing for paying bills on time, not having auto loans or not carrying a balance on credit cards. He said the white paper does not prescribe any particular treatment but sets forth questions state insurance regulators could ask. Ryan Purdy (Merlinos & Associates, Inc.) asked how the state insurance regulators are going to review and adopt the guidelines and best practices if they do not have access to a modeler. He questioned whether there might be a reasonable approach where the states without access to experts could have alternative guidelines, such as evaluating controls over the process. He asked whether the controls are in place. He also asked whether the company’s experts are qualified. He said this would be similar to the risk-focused approach used in financial analysis and examination. Mr. Serbinowski said the project will prove useful to state insurance regulators. He has less concern with correlation errors, given that insurers are not interested in spurious correlations because they are detrimental to the insurers’ business, so those would be weeded out over time. He said he is concerned with consumers being able to edit the data. Gender and credit scoring have been used for a long time, and no causality has been explained in a rigorous way. Mr. Vigliaturo said the volunteer drafters will sort through over a hundred pages of comments and propose changes to the white paper. He said the Task Force will discuss a revised version at the Spring National Meeting. Having no further business, the Casualty Actuarial and Statistical (C) Task Force adjourned. W:\National Meetings\2019\Spring\TF\CasAct\3-12 CASTF min.docx
Attachment Three Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 1
Draft: 3/1/19
Casualty Actuarial and Statistical (C) Task Force Conference Call
February 12, 2019 The Casualty Actuarial and Statistical (C) Task Force met via conference call Feb. 12, 2019. The following Task Force members participated: Steve Kelley, Chair, represented by Phillip Vigliaturo and Connor Meyer (MN); James J. Donelon, Vice Chair, represented by Rich Piazza and Lawrence Steinart (LA); Lori K. Wing-Heier represented by Mike Ricker (AK); Jim L. Ridling represented by Daniel J. Davis and Jerry Workman (AL); Ricardo Lara represented by Giovanni Muzzarelli and Mitra Sanandajifar (CA); Michael Conway represented by Mitchell Bronson (CO); Paul Lombardo represented by Susan Andrews (CT); Stephen C. Taylor represented by David Christhilf (DC); David Altmaier represented by Howard Eagelfeld and Robert Lee (FL); Colin M. Hayashida represented by Randy Jacobson (HI); Doug Ommen represented by Travis Grassel and Andria Seip (IA); Kevin Fry represented by Judy Mottar (IL); Vicki Schmidt represented by Nicole Boyd (KS); Eric A. Cioppa represented by Sandra Darby (ME); Anita G. Fox represented by Kevin Dyke (MI); Chlora Lindley-Myers represented by Julie Lederer (MO); Marlene Caride represented by Mark McGill (NJ); John G. Franchini represented by Mark Hendrick (NM); Barbara D. Richardson represented by Gennady Stolyarov (NV); Jillian Froment represented by Thomas Botsko (OH); Glen Mulready represented by Andrew Schallhorn (OK); Andrew Stolfi represented by David Dahl and Ying Liu (OR); Raymond G. Farmer represented by Michael Wise (SC); Kent Sullivan represented by J’ne Byckovski, Brock Childs, Nicole Elliott, Miriam Fisk, Eric Hintikka, Elizabeth Howland and Jennifer Wu (TX); and Mike Kreidler represented by Eric Slavich (WA). Also participating was: Gordon Hay (NE). 1. Received a Report from the Actuarial Opinion (C) Working Group Mr. Vigliaturo reported that he appointed Ms. Lederer as chair of the Actuarial Opinion (C) Working Group. Ms. Lederer said the Working Group did not propose changes to 2019 Statement of Actuarial Opinion instructions because of the actuarial qualifications project. The Working Group has not met in 2019. 2. Received a Report from the Statistical Data (C) Working Group Mr. Vigliaturo reported that he appointed Carl Sornson (NJ) as chair of the Statistical Data (C) Working Group. Mr. McGill said the Working Group plans to continue to review the formulas in the Report on Profitability by Line by State (Profitability Report). There are no other planned changes to other reports. Mr. Lee said the “dwelling fire” data in the Dwelling Fire, Homeowners Owner-Occupied, and Homeowners Tenant and Condominium/Cooperative Unit Owner’s Insurance Report (Homeowners Report) is only fire, and it does not include allied lines. He said the allied lines include all the weather-related categories, and they are the majority of what Florida would consider to be “dwelling fire.” Mr. McGill said that is a new issue, and he will inform Mr. Sornson. 3. Discussed its 2019 Charges and Work Plan
Mr. Vigliaturo addressed the Task Force’s 2019 charges and top priorities. He said the first three charges include monitoring other groups. He said the Task Force will continue to monitor other work that might be of interest and report back to the Task Force. He said he will ask Wanchin Chou (CT) to continue on as a liaison with the NAIC’s Own Risk and Solvency Assessment (ORSA) work. To facilitate regulatory discussion regarding filing issues, Mr. Vigliaturo said there will be monthly conference calls. He asked regulators to forward any agenda items or topics to be addressed. Mr. Vigliaturo said the Task Force had three 2018 charges to address appointed actuary issues, which were generally referred to as: 1) the attestation; 2) the three-year experience period; and 3) the continued competence charges. He said the currently exposed 2019 Statement of Actuarial Opinion instructions includes the Task Force’s proposals to address the attestation and the three-year experience period charges. He said with adoption of those changes, the charge will be considered completed.
Attachment Three Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 2
Mr. Vigliaturo said the 2018 continued competence charge was revised for 2019. In 2018, the Task Force adopted a plan presented jointly by the Casualty Actuarial Society (CAS) and the Society of Actuaries (SOA). He asked Mr. Dyke to continue to liaise with the CAS and the SOA to work on the 2019 charges. Mr. Dyke agreed. Mr. Vigliaturo said a major focus for the Task Force this year is the project requested by the Big Data (EX) Working Group to draft best practices for the review of predictive models and analytics filed by insurers to justify rates. He said the Task Force decided to begin this work by drafting a white paper.
Mr. Vigliaturo said the predictive analytics “Book Club” conference calls are scheduled for the rest of this year. He said there are currently no selected papers or articles to read nor speakers scheduled, so he asked for Task Force members to send suggestions to NAIC staff. He asked interested parties to volunteer to speak.
4. Adopted a Comment Letter on the Statement of Actuarial Opinion Instructions
Mr. Vigliaturo said the property/casualty (P/C) Statement of Actuarial Opinion instructions were released for a public comment period ending Dec. 15, 2018. The Task Force held calls Jan. 29, 2019; Jan. 8, 2019; and Dec. 18, 2018, to discuss the changes proposed by the Executive (EX) Committee’s ad hoc group. During the Jan. 29 conference call, the Task Force discussed a potential comment letter to send to the ad hoc group. Some volunteers on the Task Force redrafted the comment letter in light of that discussion.
Mr. Vigliaturo said the aim of the Task Force is to present agreed-upon changes and then include a second part of the letter to provide accurate information on both sides of the potential change to require American Academy of Actuaries (Academy) membership. The aim is not to debate the two sides of the issue, but it is only to fairly and accurately represent both sides of the issue. The Task Force discussed the grandfathering clause. The Task Force agreed that the general idea is that actuaries who were qualified prior to the revised definitions should continue to be qualified. There was question as to whether the grandfathering should be broader to include anyone who would have qualified previously if the new definition had been in place. Mary Downs (Academy) said the actuarial credentials were deemed by WorkCred to not be equivalent. She said there will likely be confusion with appointed actuaries regarding what they should do with the current wording. Mr. Stolyarov said the WorkCred study is not considered authoritative by Nevada because the study has not been reviewed. He said the word “grandfathering” could be changed if that is confusing. Mr. Dyke said the WorkCred study cannot be ignored without further understanding or some deliberation by the ad hoc group. He said if the exam requirements are substantially similar to before the WorkCred study was completed, then the current study could be applied retroactively, but he is not sure that is the case. Mr. Davis said the executive summary was distributed, and it seems to support Ms. Downs’ statement. Mr. Stolyarov said he would need to see the complete study and not just the executive summary. Mr. Davis said changes will be made in the NAIC’s Educational Standards and Assessment Project so that both organizations will be able to make changes and meet the minimum educational standards. He said his understanding is the assessment project would have looked at prior years and been able to advise about the syllabi in place each year. Mr. Stolyarov said the current project is extensive, objective and rigorous, and it will be public. Kris DeFrain (NAIC) said the current study is a point-in-time assessment of whether the syllabus meets the standards. Mr. Davis said it seems there might need to be additional information to determine the appropriate grandfathering clause. Mr. Hay said if the bar was not cleared on the first pass, then there would be an evolution. Ms. DeFrain said that is accurate, in that the CAS and the SOA will have two years to make any changes to their syllabus to meet the NAIC’s minimum standards. Ralph Blanchard (Travelers) said an Associate Casualty Actuarial Society (ACAS) was all that used to be needed to be qualified; now, the ACAS needs an additional reserving exam. He said restrictions have been in place in the past.
Attachment Three Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 3
Mr. Stolyarov said the Task Force should also consider that professionals take continuing education and gain experience after they receive their designation, so professionals are not asked to retake the current exams when the syllabi change. The Task Force noted significant reservations in regard to proposing acceptable wording without knowing the restrictions to be placed on the NAIC accepted actuarial designations and/or any other changes the ad hoc group might make. The Task Force agreed to ask for another consultation with the ad hoc group once those restrictions and all other changes are known. Mr. Stolyarov and Mr. Dyke presented proposed changes to the second part of the letter regarding the potential change to require Academy membership. Mr. Stolyarov made a motion, seconded by Mr. Piazza, to adopt the comment letter as revised (Attachment Three-A). The motion passed unanimously. Mr. Vigliaturo asked NAIC staff to finalize the letter and submit it to the ad hoc group by the Feb. 15 comment deadline. 5. Discussed the Best Practices for Regulatory Review of Predictive Analytics White Paper Mr. Vigliaturo said Mr. Piazza will stay in the lead to complete the white paper this year. He said the Task Force discussed the white paper during an Oct. 9, 2018, conference call, and it exposed the paper for a 60-day public comment period ending Feb. 12, 2019. The comment period was subsequently extended until Feb. 19, 2019. Numerous comments were received (Attachment Three-B). Mr. Piazza said the target completion date for the white paper is the Fall National Meeting. He said the next steps will be to take the hundreds of pages of comments and get them put in order for easier review. The drafting group will then document the comments and propose changes to the white paper. Birny Birnbaum (Center for Economic Justice—CEJ) said other groups have addressed a large number of comments such as these by using tables and grouping comments on the same section together. Mr. Piazza agreed. He said the Task Force can re-expose the paper. 6. Discussed the Global Insurance Symposium Answering Mr. Piazza’s question, Ms. Seip said Dorothy Andrews (Insurance Strategies Consulting LLC) will be providing regulator-only predictive analytics training around the Global Insurance Symposium, which is being held April 23–25 in Des Moines, IA. Ms. Darby said the training is before and after the symposium. Ms. Seip said she would send NAIC staff additional information for distribution. Mr. Stolyarov asked for a more detailed description of the training. Having no further business, the Casualty Actuarial and Statistical (C) Task Force adjourned. W:\National Meetings\2019\Spring\TF\CasAct\2-12 CASTF min.docx
Attachment Three-A Casualty Actuarial and Statistical (C) Task
4/6/19
To: Executive (EX) Committee Ad Hoc Group — Commissioner Jim L. Ridling, Commissioner James J. Donelon, Superintendent Eric A. Cioppa
From: Phil Vigliaturo, Chair of the Casualty Actuarial and Statistical (C) Task Force
Date: February 12, 2019
RE: Exposure of Property/Casualty Statement of Actuarial Opinion Instructions
On December 18, 2018, January 8, January 29, and February 12, 2019, the Casualty Actuarial and Statistical (C) Task Force (“CASTF”) held conference calls that were open to the public, for the purpose of discussing the exposure draft of the revisions to the Property/Casualty (P/C) Statement of Actuarial Opinion Instructions, currently exposed through February 15, 2019. The focus of CASTF in these discussions was on the changes proposed by the Executive (EX) Committee’s ad hoc group.
The purpose of this letter is to document items of consensus that emerged during those conference calls, as well as a major area of difference among CASTF members regarding one matter relevant to the exposure draft.
Items of Consensus Among CASTF Members
The CASTF members achieved agreement regarding the following four desirable changes to the pending draft:
1. In 1A Definitions, item iii: Leave the “or” in the first line.
2. Change the paragraph after 1A Definitions, item i-iv, to read:
“An exception to parts (i) & (ii) of this definition would be an actuary who is a member of the Academy evaluated by the Academy’s Casualty Practice Council and determined to be a Qualified Actuary for particular lines of business and business activities. Should an actuary qualify under this alternate route, the actuary must attach a copy of the approval letter from the Academy to the Actuarial Opinion each year.”
Membership in the Academy is a current requirement for those actuaries who need to be evaluated by the Academy’s Casualty Practice Council.
3. In 1A Definitions, item ii, add language to emphasize there may be restrictions on acceptable designations:
“has (a) obtained an NAIC Accepted Actuarial Designation, subject to any noted restrictions, or (b) became a member in good standing of the Casualty Actuarial Society prior to January 1, 2021; and”
The noted restrictions are expected to come from the NAIC’s Educational Standards and Assessment Project.
4. Revise the grandfathering section (after paragraph 1A Definitions, items i-iv) to emphasize there may be restrictions on acceptable designations as follows:
Attachment Three-A Casualty Actuarial and Statistical (C) Task
4/6/19
© 2019 National Association of Insurance Commissioners 2
Replace “The listed actuarial designations earned prior to January 1, 2021 are grandfathered as accepted.” with the following: “The listed actuarial designations earned prior to January 1, 2021, subject to any noted restrictions, are grandfathered as accepted.”
CASTF believes there might be additional issues to discuss once we know the restrictions placed on the NAIC Accepted Actuarial Designations (resulting from the NAIC’s Educational Standards and Assessment Project). This might impact our proposal for grandfathering and/or cause other concerns. Once the NAIC’s Educational and Assessment Project is completed, we would appreciate discussing the instructions with the ad hoc group prior to the instructions being finalized.
The remainder of this letter explains why membership in the American Academy of Actuaries (Academy) should or should not be a requirement for satisfying the definition of Qualified Actuary in order to be a P/C Appointed Actuary.
Major Area of Difference Among CASTF Members: Whether to Require Membership in the Academy for Appointed Actuaries
The major difference expressed among the CASTF members on the calls of December 18, 2018, and January 8, 2019, pertained to whether or not membership in the Academy should be required for P/C Appointed Actuaries to meet the definition of a “Qualified Actuary”. In the pending draft, sub-item (iv) of the definition of a “Qualified Actuary” would require the actuary to be “a member of a professional actuarial association subject to the same Code of Professional Conduct promulgated by the Academy, the U.S. Qualification Standards, and the Actuarial Board for Counseling and Discipline when practicing in the U.S.” Some CASTF members wished to replace this with a requirement to be a Member of the American Academy of Actuaries (MAAA). Other CASTF members supported the above wording as currently drafted. The purpose of this portion of the letter is to identify the existence of this disagreement and to provide the key arguments for each of the positions on this issue.
CASTF has discussed whether membership in the Academy should be included in the definition of Qualified Actuary. The original definition proposed by the Executive (EX) Committee’s ad hoc group (“ad hoc group”) on December 29, 2017, included a membership requirement, but this was subsequently removed based on the ad hoc group’s review of comments on the proposal.
Following the latest exposure, some members of CASTF voiced support on its January 8, 2019, call for replacing the broad membership definition in item iv. with a more specific definition requiring membership in the Academy, as follows under Section 1A:
iv. is a Member of the American Academy of Actuaries (MAAA) member of a professional actuarial association subject to the same Code of Professional Conduct promulgated by the Academy, the U.S. Qualification Standards, and the Actuarial Board for Counseling and Discipline when practicing in the U.S..
Further discussion was held, with other members of CASTF voicing support for the currently proposed definition. Despite the lack of consensus on the definition, CASTF believes it is important for the Executive (EX) Committee’s ad hoc group to hear the arguments for and against requiring Academy membership.
Reasons Provided for Why Membership in the Academy Should Be a Requirement
1. The Academy is focused solely on the U.S. actuarial profession: Unlike other U.S. based actuarial organizations, the Academy serves members performing actuarial services in the U.S. only, with U.S. based membership criteria not duplicated by any other U.S. based organization. Non-residents and resident aliens of fewer than three years are required to describe their actuarial work experience and need for Academy
Attachment Three-A Casualty Actuarial and Statistical (C) Task
4/6/19
© 2019 National Association of Insurance Commissioners 3
membership and obtain a letter of reference attesting to their work in and knowledge of U.S. standards and practices. Academy membership would assure that non-resident members of the SOA and CAS would have sufficient U.S. based experience to issue actuarial opinions in the U.S. Specific requirements can be found in items 1 and 2C in the Academy’s membership form (attached). The Academy is currently the only U.S. based actuarial organization that performs this screening of non-residents and resident aliens of fewer than three years. Without requiring Academy membership, there is no such assurance for either a Board of Directors or the state regulators.
2. Academy membership has been established as the standard for qualifying actuaries: The Academy was formed in 1965 by the other existing actuarial organizations, including the Casualty Actuarial Society (CAS) and the Society of Actuaries (SOA), to be the credential recognized as indicating qualification for professional accreditation. As a result, nearly all NAIC model laws and regulations, and Federal laws and regulations require Academy membership when an actuarial opinion is required. Not including Academy membership in the definition of P/C Qualified Actuary would deviate from the historical standard implemented at state and federal levels.
3. Academy membership supports the continued professionalism of actuaries. The Academy is the home for the Committee on Qualifications, the Actuarial Standards Board, and the Actuarial Board for Counseling and Discipline. The Academy also promotes high level of professionalism of actuaries in their practice through its Council on Professionalism, which includes representatives from the above three organizations as well as the CAS and SOA.
4. The Academy serves the public interest. The Academy’s mission is to serve the public interest by providing non-partisan objective actuarial experience on U.S. public policy issues at both the state and federal level. The NAIC has benefited from the Academy’s expertise for years in drafting model laws, regulations, tables, and guidelines.
5. The Academy specifically supports P/C Appointed Actuaries with tools to improve the quality and reliability of P/C actuarial opinions. The Academy annually produces a P/C Loss Reserve Law Manual, which is a compilation of insurance laws on loss and loss expense reserves in all 50 states, DC, and Puerto Rico. Academy members receive a $400 discount on the purchase of the manual. The Academy also offers a two-day Seminar on Effective P/C Actuarial Opinions providing a deeper understanding of the laws, regulations, and qualifications for issuing actuarial opinions. Academy members receive a $100 discount on the registration fee. (Regulators may attend at no cost). Finally, the Academy produces an annual practice note providing common practices for issuing the P/C actuarial opinions and reports.
6. Other countries require membership in their national actuarial organization. Appointed actuaries in Canada must be members of the Canadian Institute of Actuaries (CIA), regardless of how they received their basic education. Many actuaries in Canada obtain their education from the CAS and SOA, which are recognized through mutual recognition to grant membership in the CIA. We understand Germany and Mexico have similar requirement for membership in their societies for actuaries who did not receive their education in those countries.
7. Academy membership is practical and understandable. Membership in the Academy is easier for a Board of Directors and state regulators to understand and verify.
8. Academy membership does not diminish the value of basic and continuing education provided by the CAS and SOA: Both the CAS and SOA have established syllabi of basic education that, if approved, will remain a vital component of an actuary’s qualifications.
Attachment Three-A Casualty Actuarial and Statistical (C) Task
4/6/19
© 2019 National Association of Insurance Commissioners 4
Reasons Provided for Why Membership in the Academy Should Not Be a Requirement
1. Membership in the Academy is not a current requirement in the P/C Statement of Actuarial Opinion Instructions. Not including such a membership requirement would preserve the status quo in this regard and would enable the CASTF to focus on the specific charges that motivated the proposed revisions to the Actuarial Opinion Instructions in the first place. Adding a membership requirement is another way to say all Appointed Actuaries must pay $675 Academy membership dues.
2. One purpose of redefining “Qualified Actuary” is to use an objective measure rather than to support membership in any particular organization. This appears to be the reason Sub-item (iv) in the pending draft’s definition of a “Qualified Actuary” requires the actuary to be “a member of a professional actuarial association subject to the same Code of Professional Conduct promulgated by the Academy, the U.S. Qualification Standards, and the Actuarial Board for Counseling and Discipline when practicing in the U.S.” This would, by itself, subject the actuary to the professionalism frameworks and disciplinary processes created by the Academy, whether or not the actuary is a member of the Academy. Regulators generally regard membership in the Academy positively. Membership in the Academy may entitle the actuary to take a more active role in the Academy’s work and decision-making processes, benefit from the Academy’s educational and networking opportunities, and participate in the policy discussions in which the Academy is engaged. In exchange, the actuary would pay Academy membership dues. However, the Academy’s membership dues and the benefits that they enable an actuary to access are not indispensable prerequisites to issuing a statutory Statement of Actuarial Opinion in connection with the loss and LAE reserves carried on an insurer’s P/C Annual Statement. That is, an individual is capable of being objectively qualified through the requisite combination of basic education, experience, and continuing education to issue such statutory Statements of Actuarial Opinion without paying Academy membership dues or taking advantage of the benefits offered by the Academy.
Beyond recognition of the professionalism frameworks and disciplinary processes created by the Academy, as well as the specific pathway offered through the Academy’s Casualty Practice Council, making Academy membership mandatory for all P/C Appointed Actuaries would create concerns about privileging one private actuarial organization over others. While supporters of such a mandate made the analogy between Academy membership for actuaries and State Medical Board licensure for medical practitioners, the more fitting analogy in the medical realm to Academy membership would be membership in the American Medical Association or a similar private professional (and policy advisory) group, which may have specific interests that are distinct from the interests of regulators or the public whom the regulators are explicitly tasked to protect. It is not the purpose of the Actuarial Opinion Instructions to make membership in a private interested-party organization a prerequisite for fulfilling an essentially public purpose – the preparation of statutory Statements of Actuarial Opinion – for which the intended users are regulators and the intended beneficiaries are members of the public.
3. Concerns about the qualifications of international actuaries should not be the basis for a new requirement that would apply to the vast majority of Appointed Actuaries who are based in the United States. If the Academy’s primary concern pertains to international actuaries who may have obtained basic education that may not have covered U.S. laws, whereas the Academy membership application requires such individuals to certify their familiarity with U.S. laws and practices in their actuarial practice area, then the way to resolve this would be the consensus change #2 agreed upon by the CASTF: to require the MAAA designation only from those actuaries who elect to be evaluated by the Academy’s Casualty Practice Council – since the actuaries who do not elect this option would be subject to the basic education requirements, which themselves include coverage of U.S. laws. However, to fulfill the above purpose, it is not necessary to require all or even the vast majority of P/C Appointed Actuaries to have Academy membership.
Attachment Three-A Casualty Actuarial and Statistical (C) Task
4/6/19
© 2019 National Association of Insurance Commissioners 5
4. Academy membership for Appointed Actuaries is not an Accreditation Standard and is only in a few states’ laws. The Actuarial Opinion Instructions are ultimately subordinate to State law and are not the proper vehicle for creating new law or new requirements. It is true that a few States’ laws already mandate Academy membership for P/C Appointed Actuaries – but this is not the case for all States, and the laws of specific NAIC-Accredited States were cited in which Academy membership was not a requirement. For those States where Academy membership is currently a requirement, the laws of those States would prevail over the Actuarial Opinion Instructions in any event. However, the Actuarial Opinion Instructions should not unilaterally set a more stringent bar than currently exists pursuant to the laws of any NAIC-Accredited States. The NAIC has a specific process for amending its model statutes and regulations, and even subsequent to such amendments, decisions would need to be made at the NAIC level as to whether to designate the amended model laws as Accreditation standards, and decisions would also need to be made at the level of a State’s Legislature as to whether to enact any given model statute and at the level of a State’s regulatory authority as to whether to enact any given model regulation. These processes need to be followed by those wishing to introduce any new mandates for membership in any specific actuarial organization.
5. While supporters of mandating Academy membership have cited similar mandates in the areas of practice of Life and Health Insurance, it remains the case that model laws and regulations pertaining to statutory Statements of Actuarial Opinion are distinct between P/C, Life, and Health areas of practice. Often the differences are material and lengthy; they reflect significant inherent differences in the lines of business subject to the statutory Statements of Actuarial Opinion. Accordingly, any specific reasons for requiring Academy membership for Appointed Actuaries in Life and/or Health areas of practice may not be germane to the P/C area of practice. Each area of practice should be considered from the standpoint of the issues, developments, and products specific to that area, and attempts to “harmonize” requirements across areas of practice have historically not succeeded and will often not produce a desirable result due to failure to recognize and respect area-specific differences, nuances, and needs.
6. The current proposal is a principle-based approach. The main purpose of the contemplated revisions to the Actuarial Opinion Instructions was to create a pathway for recognition of NAIC-Accepted Actuarial Designations that will be evaluated through the objective Educational Standards and Assessment process. This is a principles-based approach that would consider the content of an actuarial organization’s offerings instead of making membership in any specific organization a per se requirement. Mandating membership in the Academy (or any singular organization identified by name) would be in conflict with such a principles-based approach and would revert to the approach of “hard-coding” a specific organization into the Actuarial Opinion Instructions. Such “hard-coding” is insufficiently flexible to address evolving developments in the profession, for responding to which a principles-based approach offers more versatility.
7. Ultimately the Actuarial Opinion Instructions support a State-based system of insurance regulation, where regulatory authority rests in the States. Some individual States may favor mandating Academy membership for Appointed Actuaries and may choose to do so within the framework of their own laws. (Indeed, several States have done this.) However, in any situation where material differences exist among the States with regard to such an issue, the preferred approach is to “agree to disagree” – allowing individual States to pursue their own approaches in accordance with their laws, while the Actuarial Opinion Instructions should reflect the baseline requirements that are compatible with the laws and regulatory philosophies in all NAIC-Accredited member jurisdictions. The very existence of material differences of position on the issue of requiring Academy membership for Appointed Actuaries can be seen as an argument for not requiring it at this time.
Should you have questions, please contact Phil Vigliaturo, Chair of CASTF.
Cc: Kris DeFrain (NAIC)
1850 M Street NW Suite 300 Washington, DC 20036 Telephone 202 223 8196 Facsimile 202 872 1948 www.actuary.org
January 22, 2019
Kris DeFrain, FCAS, MAAA, CPCUDirector of Research and Actuarial Services National Association of Insurance
Commissioners (NAIC) Central OfficeVia Email
Re: CASTF Regulatory Review of Predictive Models White Paper
Dear Kris:
As the American Academy of Actuaries’1 senior property/casualty fellow, I appreciate this opportunity to comment on the white paper exposure draft of the Casualty Actuarial and Statistical Task Force (CASTF) discussing best practices for the Regulatory Review of Predictive Models (RRPM). To provide input, we have sought guidance from two Academy groups: the Automobile Insurance Committee of the Casualty Practice Council (CPC) and the Big Data Task Force.
I agree with the CASTF that “Insurers’ use of predictive analytics along with big data has significant potential benefits to both consumers and insurers.” Though predictive analytics is still in its early stages of use in insurance ratemaking, benefits are being realized. Along the way, the insurance industry has committed resources to fund and staff the development of predictive analytics projects. Actuaries have played a central role in this development.
In 2017, the CPC conducted a daylong seminar at the NAIC’s Insurance Summit to help familiarize regulators with predictive modeling including how it relates to public policy issues.In 2018, the Academy produced a monograph, Big Data and the Role of the Actuary, which includes extensive sections on regulatory and professionalism considerations.
The RRPM white paper is comprehensive in its scope. The need for a set of best practices in the review process is well noted. Additionally, there are 92 potential information items identified in the paper for use in reviewing model submission. I offer the following for CASTF’s consideration:
1 The American Academy of Actuaries is a 19,500+ member professional association whose mission is to serve the public and the U.S. actuarial profession. For more than 50 years, the Academy has assisted public policymakers on all levels by providing leadership, objective expertise, and actuarial advice on risk and financial security issues. The Academy also sets qualification, practice, and professionalism standards for actuaries in the United States.
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 1
1850 M Street NW Suite 300 Washington, DC 20036 Telephone 202 223 8196 Facsimile 202 872 1948 www.actuary.org
The CASTF’s charge to develop best practices for reviewing predictive rating models is an important goal. However, there is potential for the process to become unmanageable for both the modelers and the reviewers. Is there opportunity to explore whether the suggested RRPM process might be unwieldy for regulators? Would insurance consumers be better served by a more collaborative framework to assess issues?Confidentiality is addressed in two places. In the first instance, confidentiality is cited as a key regulatory principle, where appropriate. Later in the paper, it is addressed directly in that insurers are guided to know the specifics of state regulations regarding confidentiality of rate filings. Could there be a separate discussion of this issue that would seek to come up with alternative methods of model review that are acceptable to modelers and regulators?On page three of the RRPM, it states “GLM (generalized linear model) output is assumed, as part of the model design, to be 100% credible no matter the size of the underlying data set.” It should be noted that a GLM produces both a parameter estimate and a standard error around that parameter estimate. The standard error along with claim volume consideration is applied in the selection of proposed relativities.Information items A.1.a and A.1.d seem to be addressing the same concern. Could the differences between A.1.a and A.1.d be clarified? Information item A.5.a references the potential for submitting raw data. Could this prove problematic as to the security of insurance consumer data? Will regulators be attempting to recreate models? The sheer size of these data sets could make them difficult to provide. Would sample data set structures meet the needs of this information item? Finally, there might be issues regarding contractual relationships with third-party data providers and their resources.The actuarial standards of practice (ASOPs) are undoubtedly a valuable resource. The ASOPs establish standards for “appropriate” actuarial practice in the United States (ASOP No. 1, Introductory Actuarial Standard of Practice, Section 1). At a minimum, ASOP No. 12, Risk Classification (For All Practice Areas), and ASOP No. 41, Actuarial Communications, are applicable. Should those documents be explicitly considered in the best practices for RRPM?Item B.3.c addresses the intuitiveness of the predictor variables. Is it the intent of thisitem to prohibit variables that are not explicitly intuitive? Actuarial principles state that an intuitive argument is desirable, but not required. Item C.2.a appears to be addressing causal effect of the predictor variables. Could you please clarify this?In the introduction, there is a statement that “the insurer must anticipate … the reviewers’ interests because the reviewers will respond with unanticipated questions.” Anticipating the unanticipated can prove challenging. Likely, this comment is addressing a communication-related issue. Could you please clarify?
In closing, I wish to reiterate that the American Academy of Actuaries remains committed to working with CASTF as regulators strive to understand and monitor the growing role of predictive modeling in insurance rate development. We look forward to participating in the ongoing dialogue on the RRPM to help to achieve a thoughtful and effective review process.
If you have any questions about these comments, contact me ([email protected]) or Marc Rosenberg, senior casualty policy analyst, at 202-785-7865 or [email protected].
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 2
1850 M Street NW Suite 300 Washington, DC 20036 Telephone 202 223 8196 Facsimile 202 872 1948 www.actuary.org
Sincerely,
Richard Gibson, MAAA, FCASSenior Casualty FellowAmerican Academy of Actuaries
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 3
From: Ricker, Michael D (CED) <[email protected]> Sent: Tuesday, January 22, 2019 2:24 PM To: DeFrain, Kris <[email protected]> Subject: RE: Regulatory Review of Predictive Models White Paper - Comments Due Jan. 15
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Kris, a couple comments from AK regarding the 10/25/18 draft: 1. In section VII, it’s not clear how table C, “The Filed Rating Plan”, coordinates with tables A and
B. Are the items in A and B not expected to be included in rate/rule filings? Where/when/how are regulators expected to be accessing those pieces of information if not within rate/rule filings? Perhaps language can be added to make the difference between tables C and A&B more clear; the current intro language in section VII seems to imply that all of the information presented in section VII (i.e. all three tables) is for the purpose of providing supporting documentation within a rate/rule filing.
2. C.1.a appears to be referring to a state-specific SERFF Requirement. Alaska, for one, does not have an “Actuarial Memorandum section on the SERFF Supporting Documentation tab”.
Michael Ricker Property & Casualty Actuary Alaska Division of Insurance | P.O. Box 110805 | Juneau, AK 99811-0805 Phone: 907.465.2564 | Fax: 907.465.3422 | DOI Main phone: 907.465.2515 http://www.commerce.alaska.gov/web/ins [email protected]
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 4
Dear Mr. Piazza,
Thank you for the opportunity to comment on the draft white paper “Regulatory Review of Predictive Models” by the Casualty Actuarial and Statistical Task Force (CASTF).
Below are areas of concern that we would like to address:
The white paper asks companies to provide evidence of a causal relationship between risk characteristics and expected cost.
From CASTF white paper: B.3.c - Provide an intuitive argument for why an increase in each predictor variable should increase or decrease frequency, severity, loss costs, expenses, or whatever is being predicted. C.2.a - Provide an explanation how the characteristics/rating variables, included in the filed rating plan, logically and intuitively relate to the risk of insurance loss (or expense) for the type of insurance product being priced.
From “ASOP 12: Risk Classification” While the actuary should select risk characteristics that are related to expected outcomes, it is not necessary for the actuary to establish a cause and effect relationship between the risk characteristic and expected outcome in order to use a specific risk characteristic.
The white paper asks companies to provide information and data such that a regulator would be reproducing, rather than reviewing, the filed model.
From CASTF white paper: A.5.a - If the raw data selected to build the model is in a format that can be made available to the regulator, provide it. A.3.f - What adjustments were made to raw data, e.g., transformations, binning and/or categorizations? If so, name the characteristic/variable and describe the adjustment. B.6.b - What software was used? Provide the name of the software vender/developer, software product and a software version reference.
From “ASOP 41: Actuarial Communication” In the actuarial report, the actuary should state the actuarial findings, and identify the methods, procedures, assumptions, and data used by the actuary with sufficient clarity that another actuary qualified in the same practice area could make an objective appraisal of the reasonableness of the actuary’s work as presented in the actuarial report.
Additional Comments We also consider raw data to be proprietary and we do not wish to provide such data due its confidential nature. It is of the utmost importance that any sample raw data provided be protected to conceal the private information of insureds.
The white paper places a large emphasis on state-specific results.
From CASTF white paper:
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 5
C.6.a - Provide state-specific, book-of business-specific univariate historical experience data consisting of, at minimum, earned premiums, incurred losses, loss ratios and loss ratio relativities for each category of model output(s) proposed to be used within the rating plan. C.6.b - Provide an explanation of any material (especially directional) differences between model indications and state-specific univariate indications. B.4.b – Some states require state-only data to test the plan, especially for analysis where using the state-only data contradicts the countrywide results. State-only data might be more applicable but could also be impacted by low credibility for some segments of risk. B.4.j - Describe how the model was tested for geographic stability, e.g., across states or territories within state.
From “A Practitioners Guide to Generalized Linear Models”: Depending on the underlying claim frequencies and the number of factors being analyzed, credible results on personal lines portfolios can generally be achieved with around 100,000 exposures (which could for example be 50,000 in each of two years, etc). Meaningful results can sometimes be achieved with smaller volumes of data (particularly on claim types with adequate claims volume), but it is best to have many 100,000s of exposures.
Comments: In most cases, state-specific results will lack credibility for a multivariate model to achieve meaningful results and therefore not be useful for comparisons. The state-specific data may be misused and result in a less predictive model if countrywide patterns are overridden with low-volume state-specific data. If every difference between state-specific results and countrywide results were investigated, this would dramatically increase required filing time and speed-to-market would be greatly reduced.
The white paper places a large emphasis on univariate indications.
From CASTF white paper: C.6.a - Provide state-specific, book-of business-specific univariate historical experience data consisting of, at minimum, earned premiums, incurred losses, loss ratios and loss ratio relativities for each category of model output(s) proposed to be used within the rating plan. C.6.b - Provide an explanation of any material (especially directional) differences between model indications and state-specific univariate indications. Multivariate indications may be reasonable as refinements to univariate indications, but likely not for bringing about reversals of those indications.
Comments: It is generally accepted in the industry that multivariate indications are more accurate than univariate indications. Univariate indications alone should not be used to override multivariate model results. A clarification is also needed for how the paper uses the term “univariate indication”.
Modelers generally refer to “univariates” as a comparison of actual results vs predicted results by variable for the statistic being modeled, such as pure premium. However, a univariate indication can also be based on a loss ratio analysis.
It would be problematic to require a loss ratio for all model filings when loss ratio is not the target variable. For consistency and interpretability, loss ratio analyses require bringing all premiums to a common set of rating plan factors. This requires extensive work for a countrywide analysis and possibly significant work even within a state if the experience
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 6
period includes several rate filings. This extraneous effort is not justified when the loss ratio is not the target variable.
The white paper contains items that difficult or inappropriate to provide.
From CASTF white paper: B.6.a - Provide the names, contact emails, phone numbers and qualifications of the key persons who: a. Led the project b. Compiled the data c. Built the model d. Performed peer review
Comments Inappropriate to ask for contact information of employees in public documents. Filings should be considered the work product of the insurance company and a single point of contact should be used for questions. In traditional actuarial filings, the names, qualifications, and contacts of every person that worked on the filing is not provided. In some regulatory settings, one qualified actuary will sign off on the work.
From CASTF white paper: o C.2.a - Include a discussion of the relevance each characteristic/rating variable has on consumer
behavior that would lead to a difference in risk of loss. Comments
This does not provide information that is useful with respect to the strenuous nature of the ask. From CASTF white paper:
C.3.a – Provide a comparison between relativities indicated by the model to both current relativities and the insurer's selected relativities for each risk characteristic/variable in the rating plan. Each significant difference should be highlighted and explained.
Comments: Onerous to include explanation for every difference. Comparison is not possible when rating algorithms/structures are different. Focus of model comparisons should be on overall model performance rather than individual characteristics.
Once again, thank you for the opportunity to comment. Sincerely, Allstate Property & Casualty Actuarial Leadership For any questions, please contact: Mike Woods, FCAS, CSPA Allstate Insurance Company 2775 Sanders Rd Northbrook, IL 60062 [email protected]
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 7
Kris DeFrain, FCAS, MAAA, CPCU Director, Research and Actuarial Services National Association of Insurance Commissioners (NAIC)
Sent via e-mail at [email protected]
January 22, 2019
Dear Ms. DeFrain:
Thank you for the opportunity to comment on the NAIC Casualty Actuarial and Statistical Task
Force (CASTF) proposed exposure draft regarding the Regulatory Review of Predictive Models currently
under consideration before the CASTF. The following comments are submitted on behalf of the
American Property Casualty Insurance Association (APCIA)1.
The APCIA greatly appreciates the work of the CASTF on the exposure draft. The following remarks are
presented with intended support for regulatory guidance in the form of best practices to benefit
regulators and insurers in the review of predictive models and analytics utilized by the insurance
company to justify a filed rating plan for the private passenger auto or homeowners insurance market.
The APCIA believes that best practices could foster comprehensive upfront dialogue between the filing
company and regulator that supports an efficient and effective review appropriately focused on
ensuring compliance with the applicable statutory or regulatory rating standards. However, APCIA
strongly believes the creation of best practices should not be an initiative to create new rating standards
that extend the statutory scope of the rate review process.
The CASTF had previously surveyed state insurance departments for documented practices, guidelines,
and checklists developed to aid state insurance department staff that review and approve rate filings
that may rely in some part on a generalized linear or other type of predictive model. Further, the
nationally recognized Actuarial Standards Board has developed professional standards of practice
1 Representing nearly 60 percent of the U.S. property casualty insurance market, the American Property Casualty Insurance Association (APCIA) promotes and protects the viability of private competition for the benefit of consumers and insurers. APCIA represents the broadest cross-section of home, auto, and business insurers of any national trade association. APCIA members represent all sizes, structures, and regions, which protect families, communities, and businesses in the U.S. and across the globe. APCIA is the result of the merger between the American Insurance Association (AIA) and the Property Casualty Insurers Association of America (PCIAA).
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 8
Page 2 of 29
related to use of modeling in insurance applications that were established through a deliberative and
transparent process.
This current “best practices” draft includes numerous “essential” responses to questions that extend
well beyond what any insurance department currently requires as well as the best practices set forth in
established actuarial professional standards. The APCIA urges the CASTF to revert to identifying the
existing best practices currently employed by insurance regulators and to leverage the well-vetted best
practices established by the Actuarial Standards Board as the foundation for this effort. APCIA believes it
would be detrimental to the shared goals of regulators and industry to establish essential practices that
insurance departments in their current capacity have not and potentially cannot employ.
It is imperative that any best practice not create a one-size-fits-all prescriptive checklist that may unduly
restrict the use of advanced mathematical and actuarial techniques and the risk rating factors necessary
to most appropriately price companies’ insurance business. As the introduction section of the draft
reflects, insurers’ use of predictive analytics along with big data has significant potential benefits to both
consumers and insurers by transforming the insurer-consumer experience into a more meaningful
relationship. However, predictive analytics techniques are evolving rapidly, and their application is not
yet an industry norm. The extent to which insurers leverage predictive analytics in their insurance
application can be expected to vary for each company based on their own analysis and outlook of their
applicable book of business.
To establish true best practices, the APCIA recognizes the regulator’s essential role and duty over the
rating plans in their state in accordance with their specific statutory authority. Best practices can help
the regulator and insurance company establish a base understanding of the essential elements of a
model that may influence the regulatory review as to whether modeled rates are appropriately justified.
The expectation is that best practices will aid speed to market and competitiveness of the state
marketplace. With the adoption of such guiding best practices the state regulator and filing company
will be better able to identify the resources and areas of focus needed to assist in the review of
predictive models. To that end, the list of essential elements must be limited to those truly needed and
be deliberatively focused on areas of priority for what will amount to one component of the supporting
methodology for the company’s rating plan. So, while there will be cases where more information is
required, the table attached to this letter seeks to provide the task force with suggestions to cull and
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 9
Page 3 of 29
streamline the proposed ninety-two elements of regulatory review. APCIA believes the current number
of elements would add significant time-consuming demands on the company and regulator without
further benefit and give cause to dissuade the use of advanced analytics and the advancement of
product innovation.
In addition, the APCIA would urge the CASTF to initiate additional discussions amongst the member
states to garner further understanding of how generalized linear models have been reviewed and the
specific filing requirements for insurers that employ them. Various states already obtain significant
information on model input data, the structure of the model, and the model outputs, which becomes
the basis for a thorough discussion of the underlying assumptions and modeling methodology; the
reasons for the approaches selected and the mathematical formulas used; and, comparison and contrast
of indicated relativities and confidence intervals.
The APCIA lastly recommends that the “best practices” not create a new intuitive rating standard for the
relationship between selected risk factors and expected loss or expense. We urge the CASTF to adhere
to Actuarial Standard of Practice No.12 Risk Classification (for all Practice Areas) Section 3.2.2 states:
“Causality—While the actuary should select risk characteristics that are related to expected outcomes, it
is not necessary for the actuary to establish a cause and effect relationship between the risk
characteristic and expected outcome in order to use a specific risk characteristic.”
The APCIA submits these comments with the intent to support adopted best practices that attain the
goals of product innovation, education, efficiency, and compliance. We encourage the CASTF to consider
that regulator requests for raw data and details on the development of model algorithms could stifle
innovation and speed to market efforts and jeopardize the proprietary nature of the data and privacy
concerns associated with sharing it. Requiring this level of detail from companies, as well as requiring
regulators to find time to review this level of detail within every filing, can severely slow down the
approval timeline. This can unduly create more burden on companies with greater levels of
segmentation and innovation in their products. In the very competitive marketplace that exists for
personal lines insurance, we believe consumers would be best served through more effective discussion
of the essential qualitative information about the rating variables, combined with more general support
for the predictive nature of the variables used.
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 10
Page 4 of 29
The following table includes the comments of the APCIA related to the specific 92 elements of the
proposed regulatory review of predictive models. Input is additionally provided on other sections of the
exposed draft document.
Thank you again for the opportunity to comment. We look forward to working with the Task Force to
achieve a solution that benefits regulators, insurers and ultimately our consumers.
Sincerely,
David Kodama Lisa Brown
Assistant Vice President, APCIA Assistant General Counsel & Director, APCIA
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 11
Page
5 o
f 29
A. S
elec
ting
Mod
el In
put
Info
rmat
ion
Impo
rtan
ce to
R
egul
ator
’s R
evie
w
"Ess
entia
l" o
r "M
ay B
e R
eque
sted
"
Com
men
tsA
PCIA
1.
Avai
labl
e D
ata
Sour
ces
A.1
.a
Prov
ide
deta
ils o
f all
data
sour
ces i
nclu
ding
th
e ex
perie
nce
perio
d fo
r in
sura
nce
data
and
whe
n th
e da
ta w
as la
st
reco
rded
or u
pdat
ed.
Esse
ntia
l
This
info
rmat
ion
can
be u
sed
to e
valu
ate
the
com
plet
enes
s of t
he d
ata
sour
ce, i
nteg
rity
of th
e da
ta so
urce
, rel
evan
ce o
f the
dat
a to
the
pred
ictiv
e tim
efra
me,
the
pote
ntia
l for
his
toric
al b
ias,
trans
pare
ncy
to
insu
red
of th
e da
ta so
urce
, and
the
abili
ty o
f the
insu
red
to m
ake
corr
ectio
ns to
the
data
sour
ce.
A.1
.bSp
ecify
the
com
pani
es
who
se d
ata
is in
clud
ed in
th
e da
tase
ts.
May
Be
Req
uest
ed
If th
e fil
er is
par
t of a
gro
up, d
o th
e da
tase
ts in
clud
e da
ta fr
om a
ffili
ated
co
mpa
nies
? If
so, w
hich
com
pani
es?
If th
e fil
er is
an
advi
sory
or
gani
zatio
n, w
hat c
ompa
nies
are
use
d? A
re th
e co
mpa
nies
incl
uded
in
the
data
rele
vant
and
com
patib
le to
the
com
pany
that
file
d th
era
ting
plan
?
Whe
n/w
hyw
ould
this
be
rele
vant
for p
urpo
ses o
f rev
iew
ing
how
the
mod
el is
use
d in
de
velo
pmen
t of t
he ra
ting
plan
?
A.1
.c
Prov
ide
the
geog
raph
ical
sc
ope
and
geog
raph
ic
expo
sure
dis
tribu
tion
of
the
data
.
Esse
ntia
lEv
alua
te w
heth
er th
e da
ta is
rele
vant
to th
e lo
ss p
oten
tial f
or w
hich
it is
be
ing
used
. For
exa
mpl
e, v
erify
that
hur
rican
e da
ta is
onl
y us
ed w
here
hu
rric
anes
can
occ
ur.
Que
stio
n sh
ould
focu
s atte
ntio
n on
just
ifica
tion
that
the
data
use
d is
app
ropr
iate
for
the
mod
eled
resp
onse
A.1
.d
List
eac
h da
ta so
urce
. Fo
r eac
h so
urce
, lis
t all
data
ele
men
ts u
sed
as
inpu
t to
the
mod
el th
at
cam
e fr
om th
at so
urce
.
Esse
ntia
lC
omm
enta
ry sh
ould
cla
rify
that
the
scop
e is
rega
rdin
gan
yva
riabl
e th
at e
nded
up
in
the
final
mod
el(s
ame
stan
dard
hel
d fo
r any
oth
er a
ctua
rial t
echn
ique
/met
hodo
logy
);cl
arify
the
“sou
rce”
is re
lativ
e to
the
insu
ranc
e co
mpa
ny.
A.1
.e
Spec
ify th
e ty
pe o
f dat
a (e
.g.,
acci
dent
yea
r or
polic
y ye
ar, t
ext,
num
eric
).
Esse
ntia
lC
ombi
ne u
nder
A.1
.a –
basi
c la
yout
, for
m, a
nd so
urce
of t
he d
ata
utili
zed
(sam
e st
anda
rd h
eld
for a
ny o
ther
act
uaria
l tec
hniq
ue/m
etho
dolo
gy)
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 12
Page
6 o
f 29
A.1
.f
Expl
ain
if in
tern
al o
r ex
tern
al d
ata
was
use
d an
d if
exte
rnal
dat
a w
as
used
, dis
clos
e re
lianc
e on
dat
a su
pplie
d by
ot
hers
.
Esse
ntia
lIn
clud
e un
der A
.1.a
A.1
.g
Prov
ide
deta
ils o
f any
no
n-in
sura
nce
data
use
d (c
usto
mer
-pro
vide
d or
ot
her)
, inc
ludi
ng w
ho
owns
this
dat
a, h
ow
cons
umer
s can
ver
ify
thei
r dat
a an
d co
rrec
t er
rors
, whe
ther
the
data
w
as c
olle
cted
by
use
of a
qu
estio
nnai
re/c
heck
list,
whe
ther
it w
as
volu
ntar
ily re
porte
d by
th
e ap
plic
ant,
and
whe
ther
any
of t
he
varia
bles
are
subj
ect t
o th
e Fa
ir C
redi
t Rep
ortin
g A
ct. I
f the
dat
a is
from
an
out
side
sour
ce, w
hat
step
s wer
e ta
ken
to
verif
y th
e da
ta w
as
accu
rate
?
Esse
ntia
lIf
the
data
is fr
om a
third
-par
ty so
urce
, the
com
pany
shou
ld p
rovi
de
info
rmat
ion
on h
ow th
e so
urce
add
ress
es th
e qu
estio
ns in
this
co
nsid
erat
ion.
The
insu
ranc
e co
mpa
ny m
ay n
ot b
e ab
le to
dete
rmin
e ho
w c
onsu
mer
s ver
ify a
nd
corr
ect 3
rdpa
rty d
ata.
Thi
shas
nev
er b
een
a pr
ereq
uisi
te fo
r rat
e ap
prov
al.F
or
exam
ple,
insu
rers
are
not
requ
ired
to d
etai
l how
vehi
cle
owne
rs c
an a
nd d
o ve
rify
and
corr
ect e
rror
s in
thei
r MV
Rs.
Prov
idin
g cu
stom
ers a
n ab
ility
to c
halle
nge
and/
or
corr
ect d
ata
inpu
ts m
ay p
rove
diff
icul
t if t
he in
sure
r doe
sn’t
own
the
data
. Im
posi
ng
this
as a
n “e
ssen
tial”
elem
ent o
f the
revi
ew m
ay re
sult
in a
disi
ncen
tive
to u
se o
r eve
n co
nsid
er n
ew so
urce
s of i
nfor
mat
ion,
and
ther
efor
e, a
dis
ince
ntiv
e to
inno
vate
.
2.
Sub-
Mod
els
A.2
.a
Dis
clos
e re
lianc
e on
sub-
mod
el o
utpu
t use
d as
inpu
t to
this
mod
el. I
f a su
b-m
odel
was
relie
d up
on,
prov
ide
the
vend
or n
ame,
an
d th
e na
me
and
vers
ion
of
the
sub-
mod
el. I
f the
sub-
mod
el w
as b
uilt/
crea
ted
in-
hous
e, p
rovi
de c
onta
ct
info
rmat
ion
for t
he p
erso
n re
spon
sibl
e fo
r the
sub-
mod
el.
Esse
ntia
l
Exam
ples
of s
uch
sub-
mod
els i
nclu
de c
redi
t/fin
anci
al sc
orin
g al
gorit
hms a
nd h
ouse
hold
com
posi
te sc
ore
mod
els.
Sub-
mod
els c
an
be e
valu
ated
sepa
rate
ly a
nd in
the
sam
e m
anne
r as t
he p
rimar
y m
odel
un
der e
valu
atio
n.
The
“ess
entia
l” a
ttent
ion
shou
ld fo
cust
he fi
ling
com
pany
todi
sclo
se re
lianc
e an
d ex
plan
atio
n of
the
rele
vant
data
sour
cesa
nd/o
r mod
el/s
ub-m
odel
s,an
d ho
w th
ey a
reus
ed to
supp
ort t
he se
lect
ed ra
te fi
ling.
Whi
le w
ho b
uilt
the
sub-
mod
el a
nd h
ow it
was
built
may
be
of in
tere
st, i
t sho
uld
not b
e th
e “e
ssen
tial”
focu
s ove
r the
insu
ranc
e ap
plic
atio
n of
the
sub-
mod
el–
this
shou
ld b
e th
e st
anda
rd n
o di
ffere
nt fo
r any
oth
er
actu
aria
l or s
tatis
tical
tool
use
d in
the
rate
mak
ing.
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 13
Page
7 o
f 29
A.2
.b
If u
sing
cat
astro
phe
mod
el
outp
ut, i
dent
ify th
e ve
ndor
an
d th
e m
odel
se
tting
s/as
sum
ptio
ns u
sed
whe
n th
e m
odel
was
run.
Esse
ntia
lFo
r exa
mpl
e, it
is im
porta
nt to
kno
w h
urric
ane
mod
el se
tting
s for
st
orm
surg
e, d
eman
d su
rge,
long
/sho
rt-te
rm v
iew
s.
A.2
.c
If u
sing
cat
astro
phe
mod
el
outp
ut (a
sub-
mod
el) a
s in
put t
o th
e G
LM u
nder
re
view
, dis
clos
e w
heth
er
loss
ass
ocia
ted
with
the
mod
eled
out
put w
as
rem
oved
from
the
loss
ex
perie
nce
data
sets
.
Esse
ntia
l
If a
wea
ther
-bas
ed su
b-m
odel
is in
put t
o th
e G
LM u
nder
revi
ew, l
oss
data
use
d to
dev
elop
the
mod
el sh
ould
not
incl
ude
loss
exp
erie
nce
asso
ciat
ed w
ith th
e w
eath
er-b
ased
sub-
mod
el. D
oing
so c
ould
cau
se
dist
ortio
ns in
the
mod
eled
resu
lts b
y do
uble
cou
ntin
g su
ch lo
sses
w
hen
dete
rmin
ing
rela
tiviti
es o
r los
s loa
ds in
the
filed
ratin
g pl
an.
For e
xam
ple,
redu
ndan
t los
ses i
n th
e da
ta m
ay o
ccur
whe
n no
n-hu
rric
ane
win
d lo
sses
are
incl
uded
in th
e da
ta w
hile
als
o us
ing
a se
vere
con
vect
ive
stor
m m
odel
in th
e ac
tuar
ial i
ndic
atio
n. S
uch
redu
ndan
cy m
ay a
lso
occu
r with
the
incl
usio
n of
fluv
ial o
r plu
vial
flo
od lo
sses
whe
n us
ing
a flo
od m
odel
, inc
lusi
on o
f fre
eze
loss
es
whe
n us
ing
a w
inte
r sto
rm m
odel
or i
nclu
ding
dem
and
surg
e ca
used
by
any
cat
astro
phic
eve
nt.
A.2
.d
If u
sing
out
put o
f any
sc
orin
g al
gorit
hms,
prov
ide
a lis
t of t
he v
aria
bles
use
d to
de
term
ine
the
scor
e an
d pr
ovid
e th
e so
urce
of t
he
data
use
d to
cal
cula
te th
e sc
ore.
Esse
ntia
lA
ny su
b-m
odel
shou
ld b
e re
view
ed in
the
sam
e m
anne
r as t
he
prim
ary
mod
el th
at u
ses t
he su
b-m
odel
’s o
utpu
t as i
nput
.
A.2
.e
Was
the
sub-
mod
el
prev
ious
ly a
ppro
ved
(or
acce
pted
) by
the
regu
lato
ry
agen
cy?
Esse
ntia
lIf
the
sub-
mod
el w
as p
revi
ousl
y ap
prov
ed, t
hat m
ay c
hang
e th
e ex
tent
of t
he su
b-m
odel
’s re
view
.
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 14
Page
8 o
f 29
3.
Adju
stm
ents
and
Scr
ubbi
ng
A.3
.aPr
ovid
e pr
e-sc
rubb
ed d
ata
dist
ribut
ions
for e
ach
inpu
t.M
ay B
e R
eque
sted
Com
pare
thes
e di
strib
utio
ns to
A.3
.g
The
sugg
estio
n th
at th
is is
“es
sent
ial”
to th
e re
gula
tor’
s rev
iew
of a
rate
filin
g ra
ises
co
ncer
n. T
his w
illlik
ely
be a
sign
ifica
nt a
dmin
istra
tivel
y bu
rden
som
e in
quiry
that
w
ill re
quire
a h
igh
leve
l of e
xper
tise,
tim
e an
d ot
her r
esou
rces
for t
he re
gula
tort
o m
ake
use
of. I
twou
ld b
e m
ore
appr
opria
te to
ask
wha
t var
iabl
es h
ad m
eani
ngfu
l ad
just
men
ts a
nd th
en d
escr
ibe
thos
e ad
just
men
ts.A
dditi
onal
ly, t
hist
ype
of
info
rmat
ion
coul
dbe
con
side
red
high
ly c
onfid
entia
l and
shou
ld o
nly
be re
ques
ted
ifit
can
be p
rovi
ded
in a
pro
prie
tary
and
con
fiden
tial m
anne
r. Th
e “e
ssen
tial”
det
ail o
f the
da
tadi
strib
utio
ns fo
r the
rele
vant
inpu
ts a
nd o
utpu
ts c
an b
e pr
ovid
ed u
nder
A.1
.
A.3
.bH
ow w
as m
issi
ng d
ata
hand
led?
Esse
ntia
lC
ombi
ne A
.3.b
. –d.
Dat
a qu
ality
“sc
rubb
ing”
is a
n “e
ssen
tial”
cons
ider
atio
n fo
r any
da
tase
t. Is
the
stan
dard
the
sam
e fo
r mod
elle
d vs
non
-mod
elle
d da
ta?
A.3
.cIf
dup
licat
e re
cord
s exi
st,
how
wer
e th
ey h
andl
ed?
Esse
ntia
lSe
e co
mm
ent u
nder
A.3
.b.
A.3
.d
Wer
e an
y da
ta o
utlie
rs
iden
tifie
d an
d su
bseq
uent
ly
adju
sted
? N
ame
the
outli
ers
and
expl
ain
the
adju
stm
ents
m
ade
to th
ese
outli
ers.
Esse
ntia
lSe
e co
mm
ent u
nder
A.3
.b
A.3
.e
Wer
e pr
emiu
m, e
xpos
ure,
lo
ss o
r exp
ense
dat
a ad
just
ed (e
.g.,
deve
lope
d,
trend
ed, a
djus
ted
for
cata
stro
phe
expe
rienc
e or
ca
pped
) and
, if s
o, h
ow?
Do
the
adju
stm
ents
var
y fo
r di
ffere
nt se
gmen
ts o
f the
da
ta a
nd, i
f so,
wha
t are
the
segm
ents
and
how
was
the
data
adj
uste
d?
Esse
ntia
l
Look
for a
nom
alie
s in
the
data
that
shou
ld b
e ad
dres
sed.
For
ex
ampl
e, is
ther
e an
ext
rem
e lo
ss e
vent
in th
e da
ta?
If o
ther
pro
cess
es
wer
e us
ed to
load
rate
s for
spec
ific
loss
eve
nts,
how
is th
e im
pact
of
thos
e lo
sses
shou
ld b
e re
mov
ed c
onsi
dere
d?fr
om th
e in
put d
ata,
e.g
., la
rge
loss
es, f
lood
, hur
rican
e or
seve
re c
onve
ctiv
e st
orm
mod
els f
or
PPA
com
preh
ensi
ve o
r hom
eow
ners
’ los
s.
A.3
.f
Wha
t adj
ustm
ents
wer
e m
ade
to ra
w d
ata,
e.g
., tra
nsfo
rmat
ions
, bin
ning
an
d/or
cat
egor
izat
ions
? If
so
, nam
e th
e ch
arac
teris
tic/v
aria
ble
and
desc
ribe
the
adju
stm
ent.
Esse
ntia
lSe
e co
mm
ent u
nder
A.3
.a.C
ombi
ne w
ith C
.5.a
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 15
Page
9 o
f 29
A.3
.gPr
ovid
e po
st-s
crub
bed
data
di
strib
utio
ns fo
r eac
h in
put.
May
Be
Req
uest
edC
ompa
re th
ese
dist
ribut
ions
to A
.3.a
See
com
men
t und
er A
.3.a
.
4.
Dat
a O
rgan
izat
ion
–RE
NAM
E D
ata
Revi
ew a
nd R
econ
cilia
tion
A.4
.a
Doc
umen
t the
met
hod
of
orga
niza
tion
for c
ompi
ling
data
, inc
ludi
ng p
roce
dure
s to
mer
ge d
ata
from
diff
eren
t so
urce
s and
a de
scrip
tion
of
any
prel
imin
ary
anal
yses
,da
ta c
heck
s, an
d lo
gica
l te
sts p
erfo
rmed
on
the
data
an
d th
e re
sults
of t
hose
test
s.
May
Be
Req
uest
edEs
sent
ial
This
shou
ld e
xpla
in h
ow d
ata
from
sepa
rate
sour
ces w
as m
erge
d.
The
“ess
entia
l” a
ttent
ion
to d
ata
orga
niza
tion
and
prep
arat
ion
can
dem
and
sign
ifica
nt
time
and
reso
urce
s with
out a
ttent
ion
to h
ow th
is su
ppor
ts th
e ra
te re
view
pro
cess
.B
est P
ract
ices
shou
ld ra
ther
focu
s on
how
the
data
supp
orts
the
ratin
g va
riabl
es in
the
filed
insu
ranc
e ap
plic
atio
n.
A.4
.b
Doc
umen
t the
pro
cess
for
revi
ewin
g th
e ap
prop
riate
ness
, re
ason
able
ness
, con
sist
ency
an
d co
mpr
ehen
sive
ness
of
the
data
, inc
ludi
ng a
ju
stifi
catio
n of
why
the
data
m
akes
sens
e.
Esse
ntia
l
For e
xam
ple,
if b
y-pe
ril m
odel
ing
is p
erfo
rmed
, the
doc
umen
tatio
n sh
ould
be
for e
ach
peril
and
mak
e in
tuiti
ve se
nse.
For
exa
mpl
e, if
“m
urde
r” o
r “th
eft”
rate
s are
use
d to
pre
dict
the
win
d pe
ril, p
rovi
de
supp
orta
nd a
logi
cal e
xpla
natio
n.
Cla
rific
atio
n ne
eded
–It
is n
ot u
nder
stoo
d w
hat i
s mea
nt b
y th
e es
sent
ial n
eed
for
“jus
tific
atio
n of
why
the
data
mak
e se
nse”
rela
tive
to th
e ov
erar
chin
g go
al to
just
ify
the
rate
filin
g.Th
e re
fere
nce
to a
n “i
ntui
tive
sens
e” fo
r any
by-
peril
mod
elin
g is
not
de
fined
and
unc
lear
.Reg
ulat
ory
requ
ests
for i
ntui
tive
argu
men
ts a
nd e
xpla
natio
nsof
in
tuiti
ve re
latio
nshi
ps a
re in
cons
iste
nt w
ith A
SOP
12 R
isk
Cla
ssifi
catio
n-3
.2.2
.and
sh
ould
not
be
incl
uded
as a
n es
sent
ial b
est p
ract
ice
for r
egul
ator
y re
view
.
A.4
.c
Dis
clos
e m
ater
ial f
indi
ngs
from
the
data
revi
ew a
nd
iden
tify
any
pote
ntia
l m
ater
ial l
imita
tions
, def
ects
, bi
as o
r unr
esol
ved
conc
erns
fo
und
or b
elie
ved
to e
xist
in
the
data
.
Esse
ntia
lC
ombi
ne A
.4.c
and
A.4
.d w
ith “
esse
ntia
l” fo
cus o
n ho
w th
e co
mpa
ny a
ddre
sses
any
m
ater
ial l
imita
tions
, err
ors,
bias
es in
the
mod
elin
g.
A.4
.d
For a
ny e
rror
s or m
ater
ial
limita
tions
in th
e da
ta,
expl
ain
how
they
wer
e co
rrec
ted.
Esse
ntia
l
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 16
Page
10
of 2
9 5.
Fi
nal D
ata
Info
rmat
ion
A.5
.a
If th
e ra
w d
ata
sele
cted
to
build
the
mod
el is
in a
fo
rmat
that
can
be
mad
e av
aila
ble
to th
e re
gula
tor,
prov
ide
it.
May
Be
Req
uest
ed
Mul
tiple
con
cern
s tha
t “be
st p
ract
ices
” do
not
add
ress
the
expe
rtise
and
reso
urce
s re
quire
d to
revi
ew, a
sses
s and
util
ize
“raw
dat
a” fo
r pur
pose
of r
ate
revi
ew. P
rovi
ding
ra
w d
ata
will
rais
eco
ncer
n ab
out d
ata
secu
rity,
reco
rd re
tent
ion,
con
fiden
tialit
y an
d pr
oprie
tary
pro
tect
ions
.
B. B
uild
ing
the
Mod
el
Info
rmat
ion
Impo
rtan
ce to
Reg
ulat
or’s
R
evie
w "
Esse
ntia
l" o
r "M
ay
Be
Req
uest
ed"
Com
men
tsA
PCIA
1.
Hig
h-Le
vel N
arra
tive
for B
uild
ing
the
Mod
el
B.1
.a
Iden
tify
the
type
of m
odel
(e
.g. G
ener
aliz
ed L
inea
r M
odel
–G
LM, d
ecis
ion
tree,
Bay
esia
n G
ener
aliz
ed
Line
ar M
odel
, Gra
dien
t-B
oost
ing
Mac
hine
, neu
ral
netw
ork,
etc
.), d
escr
ibe
its
role
in th
e ra
ting
syst
em a
nd
prov
ide
the
reas
ons w
hy
that
type
of m
odel
is a
n ap
prop
riate
cho
ice
for t
hat
role
.
Esse
ntia
lIf
by-
peril
or b
y-co
vera
ge m
odel
ing
is u
sed,
the
expl
anat
ion
shou
ld b
e by
-per
il/co
vera
ge.
For S
ectio
n B
–C
onsi
dert
he w
orki
ng d
raft
of th
e A
ctua
rial S
tand
ards
Boa
rd o
n M
odel
ing
sect
ions
3.5
.1 M
itiga
tion
of M
odel
Ris
k -V
alid
atio
nan
d 3.
6 Pr
esen
tatio
n of
Res
ults
, inc
ludi
ng E
xpla
natio
n of
Lim
itatio
ns o
f Mod
els;
Dis
cuss
ion
of M
odel
s;C
ompa
rison
to P
rior R
epor
ts; a
nd D
escr
iptio
n of
Con
serv
atis
m o
r Opt
imis
m
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 17
Page
11
of 2
9 B
.1.b
A d
escr
iptio
n of
why
the
mod
el (u
sing
the
varia
bles
in
clud
ed in
it) i
s app
ropr
iate
fo
r the
line
of b
usin
ess.
Esse
ntia
lIf
by-
peril
, by-
form
or b
y-co
vera
ge m
odel
ing
is u
sed,
the
expl
anat
ion
shou
ld b
e by
-per
il/co
vera
ge/fo
rm.
Com
bine
with
B.1
.a
B.1
.c
Des
crib
e th
e m
odel
revi
ew
proc
ess,
from
initi
al c
once
pt
to fi
nal m
odel
. Kee
p th
is in
ov
ervi
ew n
arra
tive
mod
e,
less
than
3 p
ages
.
Esse
ntia
lMay
Be
Req
uest
ed
Prov
ide
as p
art o
f B.1
.a. a
nd B
.1.b
.Thi
s sho
uld
be th
e ov
er-a
rchi
ng “
esse
ntia
l”
cons
ider
atio
n fo
r B.1
, for
whi
ch th
e na
rrat
ive
shou
ld c
aptu
re a
ll th
ese
sub-
elem
ents
.A
ttent
ion
shou
ld fo
cus n
ot o
n co
ncep
t dev
elop
men
t but
impl
emen
tatio
n an
d in
corp
orat
ion
of th
e fin
al m
odel
into
the
ratin
g pl
an.
B.1
.d
Des
crib
e w
heth
er lo
ss ra
tio,
pure
pre
miu
m o
r fr
eque
ncy/
seve
rity
anal
yses
w
as p
erfo
rmed
and
, if
sepa
rate
freq
uenc
y/se
verit
y m
odel
ing
was
per
form
ed,
how
pur
e pr
emiu
ms
wer
e de
term
ined
.
Esse
ntia
l
B.1
.eW
hat i
s the
mod
el’s
targ
et
varia
ble?
Esse
ntia
lA
cle
ar d
escr
iptio
n of
the
targ
et v
aria
ble
is k
ey to
un
ders
tand
ing
the
purp
ose
of th
e m
odel
.
B.1
.fPr
ovid
e a
det
aile
d
desc
riptio
n o
f th
e va
riabl
e se
lect
ion
proc
ess.
Esse
ntia
l
B.1
.g
Was
inpu
t dat
a se
gmen
ted
in a
ny w
ay, e
.g.,
was
m
odel
ing
perf
orm
ed o
n a
by-c
over
age
or b
y-pe
ril
basi
s or b
y-fo
rm?
Expl
ain
the
form
of d
ata
segm
enta
tion
and
the
reas
ons f
or d
ata
segm
enta
tion.
Esse
ntia
lTh
e re
gula
tor w
ould
use
this
to fo
llow
the
logi
c of
the
mod
elin
g pr
oces
s.
B.1
.h
Des
crib
e an
y lim
itatio
ns o
r co
ncer
ns in
the
anal
ysis
re
sulti
ng fr
om d
ata
issu
es
and
disc
uss t
he re
sulti
ng
impa
ct o
n th
e m
odel
ing
resu
lts.
Esse
ntia
lTh
is c
anbe
cap
ture
d in
the
info
rmat
ion
prov
ided
und
erA
.4.c
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 18
Page
12
of 2
9 B
.1.i
Expl
ain
Hho
w d
ata
cred
ibili
ty (o
r lac
k th
ereo
f)
was
acc
ount
ed fo
r in
the
mod
el b
uild
ing?
Esse
ntia
l
Adj
ustm
ents
may
be
need
ed g
iven
mod
els d
o no
t exp
licitl
y co
nsid
er th
e cr
edib
ility
of t
he in
put d
ata
or th
e m
odel
’s
resu
lting
out
put;
mod
els t
ake
inpu
t dat
a at
face
val
ue a
nd
assu
me
100%
cre
dibi
lity
whe
n pr
oduc
ing
mod
eled
out
put.
Re-
stat
e: ‘E
xpla
in th
e cr
edib
ility
adj
ustm
ents
mad
e af
ter m
odel
ing’
2.
Med
ium
-Lev
el N
arra
tive
for B
uild
ing
the
Mod
el
B.2
.a
Des
crib
e an
y ju
dgm
ent u
sed
thro
ugho
ut th
e m
odel
ing
proc
ess.
Dis
clos
e as
sum
ptio
ns u
sed
in
cons
truct
ing
the
mod
el a
nd
prov
ide
supp
ort f
or th
ese
assu
mpt
ions
.
Esse
ntia
l
The
“des
crib
e an
y ju
dgm
ent”
is c
over
ed a
cros
s all
the
actu
aria
l sup
port
filed
, of
whi
ch th
e m
odel
ing
is b
ut a
par
t of t
he a
ctua
rial j
udge
men
t bei
ng a
sses
sed
in th
e ra
te
revi
ew.
It w
ould
be
nons
ensi
cal t
o ex
tract
a d
escr
iptio
n of
all
judg
emen
tsus
edou
t of
thei
r res
pect
ive
cont
ext.
Ref
er to
act
uaria
l sta
ndar
ds o
f pra
ctic
e on
dis
clos
ures
and
use
of
pro
fess
iona
l jud
gmen
t in
use
of m
odel
s
B.2
.b
If p
ost-m
odel
adj
ustm
ents
w
ere
mad
e to
the
data
and
th
e m
odel
was
reru
n,
expl
ain
the
deta
ils a
nd th
e ra
tiona
le. I
t is n
ot n
eces
sary
to
dis
cuss
eac
h ite
ratio
n of
ad
ding
and
subt
ract
ing
varia
bles
, but
the
regu
lato
r sh
ould
be
prov
ided
with
a
gene
ral d
escr
iptio
n of
how
th
at w
as d
one,
incl
udin
g an
y m
easu
res r
elie
d up
on.
Esse
ntia
lEv
alua
te th
e ad
ditio
n or
rem
oval
of v
aria
bles
and
the
mod
el
fittin
g.
B.2
.b –
B.2
.c: C
allin
g fo
r the
“es
sent
ial”
cap
ture
of d
etai
ls o
n th
e ite
rativ
e m
odel
runs
, tes
ts a
nd a
djus
tmen
ts c
an d
eman
d si
gnifi
cant
tim
e an
d re
sour
ce c
omm
itmen
t for
th
e co
mpa
ny a
nd re
gula
tor.
The
“ess
entia
l” p
riorit
y sh
ould
be
on th
e ke
y da
ta
adju
stm
ents
, var
iabl
ew
eigh
ting,
and
the
stat
istic
al (v
alid
atio
n, g
oodn
ess o
f fit,
R-
squa
re m
easu
res)
testi
ng.
B.2
.c
Des
crib
e th
e un
ivar
iate
te
stin
g an
d ba
lanc
ing
that
w
as p
erfo
rmed
dur
ing
the
mod
el-b
uild
ing
proc
ess,
in
clud
ing
a ve
rbal
su
mm
arya
n ex
plan
atio
nof
th
e th
ough
t pro
cess
es
invo
lved
.
Esse
ntia
lFu
rther
ela
bora
tion
from
B.2
.b.
See
com
men
t und
er B
.2.b
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 19
Page
13
of 2
9 B
.2.d
Des
crib
e th
e 2-
way
test
ing
and
bala
ncin
g th
at w
as
perf
orm
ed d
urin
g th
e m
odel
-bui
ldin
g pr
oces
s,
incl
udin
g a
verb
al
sum
mar
yexp
lana
tion
of th
e th
ough
t pro
cess
es o
f in
clud
ing
(or n
ot in
clud
ing)
in
tera
ctio
n te
rms.
Esse
ntia
lFu
rther
ela
bora
tion
from
B.2
.a a
nd B
.2.b
.
B.2
.e
For t
he G
LM, w
hat w
as th
e lin
k fu
nctio
n us
ed?
Wha
t di
strib
utio
n w
as u
sed
for t
he
mod
el (e
.g.,
Pois
son,
G
auss
ian,
log-
norm
al,
Twee
die)
? Ex
plai
n w
hy th
e lin
k fu
nctio
n di
strib
utio
n w
as c
hose
n. P
rovi
de th
e fo
rmul
as fo
r the
dis
tribu
tion
and
link
func
tions
, in
clud
ing
spec
ific
num
eric
al
para
met
ers o
f the
di
strib
utio
n.
Esse
ntia
l
B.2
.fW
ere
ther
e da
ta si
tuat
ions
G
LM w
eigh
ts w
ere
used
? D
escr
ibe
thes
e.M
ay B
e R
eque
sted
Inve
stig
ate
whe
ther
id
entic
al
reco
rds
wer
e co
mbi
ned
to
build
the
mod
el.
3.
Pred
icto
r Var
iabl
es
B.3
.a
Prov
ide
the
nam
es,
desc
riptio
ns a
nd u
ses o
f ea
ch p
redi
ctor
var
iabl
e,
offs
et v
aria
ble,
con
trol
varia
ble,
pro
xy v
aria
ble,
ge
ogra
phic
var
iabl
e,
geod
emog
raph
ic v
aria
ble
and
all o
ther
var
iabl
es in
th
e m
odel
; exp
lana
tions
sh
ould
not
use
pr
ogra
mm
ing
lang
uage
or
cod
e.
Esse
ntia
l
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 20
Page
14
of 2
9 B
.3.b
For e
ach
pred
icto
r va
riabl
e, st
ate
whe
ther
th
e va
riabl
e is
co
ntin
uous
, dis
cret
e or
B
oole
an.
Esse
ntia
lTh
is sh
ould
be
prov
ided
unde
r the
gen
eral
des
crip
tion
of th
e va
riabl
e da
ta.
B.3
.c
Prov
ide
an in
tuiti
ve
argu
men
tsup
port
for
why
how
an in
crea
se in
ea
ch p
redi
ctor
var
iabl
e sh
ould
may
incr
ease
or
decr
ease
freq
uenc
y,
seve
rity,
loss
cos
ts,
expe
nses
, or w
hate
ver i
s be
ing
pred
icte
d.
Esse
ntia
l
See
A.4
.b. R
equi
ring
an “
esse
ntia
l” in
tuiti
ve a
rgum
ent f
or a
pre
dict
or v
aria
ble
is n
ot
unde
rsto
od re
lativ
e to
act
uaria
l sta
ndar
ds o
f pra
ctic
e. R
efer
to A
SOP
12 R
isk
Cla
ssifi
catio
n-3
.2.2
.The
re n
eeds
to b
e fu
rther
com
men
tary
/gui
danc
e or
del
ete
the
“int
uitiv
e” st
anda
rd to
be
a m
ore
gene
ral d
escr
iptio
n, e
xpla
natio
n an
d su
ppor
t for
the
impa
ct e
ach
pred
icto
r var
iabl
e ha
s on
the
mod
elle
d ou
tput
.
B.3
.d
If th
e m
odel
er u
sed
a Pr
inci
pal C
ompo
nent
A
naly
sis (
PCA
) ap
proa
ch, p
rovi
de a
na
rrat
ive
abou
t tha
t pr
oces
s, ex
plai
n w
hy
PCA
was
use
d, a
nd
desc
ribe
the
step
-by-
step
pr
oces
s use
d to
tra
nsfo
rm o
bser
vatio
ns
(usu
ally
cor
rela
ted)
into
a
set o
f lin
early
un
corr
elat
ed v
aria
bles
. In
clud
e a
listin
g of
the
PCA
var
iabl
e an
d its
pr
inci
pal c
ompo
nent
s.
Esse
ntia
lR
enam
e th
is a
s a m
ore
gene
ric d
imen
sion
ality
redu
ctio
n an
d th
en li
st P
CA
as o
ne
exam
ple.
4.
Mas
sagi
ng D
ata,
Mod
el V
alid
atio
n an
d G
oodn
ess-
of-F
it M
easu
res
B.4
.a
Prov
ide
a de
scrip
tion
of
how
the
avai
labl
e ra
w
data
was
div
ided
be
twee
n m
odel
de
velo
pmen
t, te
st a
nd
valid
atio
n da
tase
ts.
Des
crib
e al
l ci
rcum
stan
ces u
nder
w
hich
the
test
ing
and
Esse
ntia
l
For S
ub-s
ectio
n B
.4 -
Con
side
r ASO
P 38
3.5.
2 M
odel
Out
put—
In v
iew
of t
he
inte
nded
use
of t
he m
odel
, the
act
uary
shou
ldex
amin
e th
e m
odel
out
put f
or
reas
onab
lene
ss, c
onsi
derin
g fa
ctor
s suc
h as
the
follo
win
g:a.
the
resu
lts d
eriv
ed fr
om
alte
rnat
e m
odel
s or m
etho
ds, w
here
ava
ilabl
e an
dap
prop
riate
;b. h
ow h
isto
rical
ob
serv
atio
ns, i
f app
licab
le, c
ompa
re to
resu
lts p
rodu
ced
byth
e m
odel
;c. t
he
cons
iste
ncy
and
reas
onab
lene
ss o
f rel
atio
nshi
ps a
mon
g va
rious
out
putr
esul
ts; a
nd,d
. th
e se
nsiti
vity
of t
he m
odel
out
put t
o va
riatio
ns in
the
user
inpu
t and
mod
el
assu
mpt
ions
.
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 21
Page
15
of 2
9
valid
atio
n da
tase
ts w
ere
acce
ssed
.
B.4
.b
Des
crib
e th
e m
etho
ds
used
to a
sses
s the
st
atis
tical
si
gnifi
canc
e/go
odne
ss o
f th
e fit
of t
he m
odel
, suc
h as
lift
char
ts a
nd
stat
istic
al te
sts.
Dis
clos
e w
heth
er th
e re
sults
are
ba
sed
on te
stin
g da
ta,
valid
atio
n da
ta a
nd
hold
out s
ampl
es. E
nsur
e th
at th
e as
sess
men
t in
clud
es m
odel
pr
ojec
tion
resu
lts
com
pare
d to
his
toric
al
actu
al re
sults
to v
erify
th
at m
odel
ed re
sults
bea
ra
reas
onab
le re
latio
nshi
p to
act
ual r
esul
ts. D
iscu
ss
the
resu
lts.
Esse
ntia
l
Som
e st
ates
requ
ire st
ate-
only
dat
a to
test
the
plan
, esp
ecia
lly
for a
naly
sis w
here
usi
ng th
e st
ate-
only
dat
a co
ntra
dict
s the
co
untry
wid
e re
sults
. Sta
te-o
nly
data
mig
ht b
e m
ore
appl
icab
le
but c
ould
als
o be
impa
cted
by
low
cre
dibi
lity
for s
ome
segm
ents
of ri
sk.
Com
men
tary
shou
ld c
autio
n th
e re
gula
tor t
hat s
tate
-onl
y da
ta m
ay n
ot b
e st
atis
tical
ly
cred
ible
or a
ppro
pria
te.
B.4
.c
Des
crib
e an
y ad
just
men
ts th
at w
ere
mad
e in
the
data
with
re
spec
t to
scal
ing
for
disc
rete
var
iabl
es o
r bi
nnin
g th
e da
ta.
Esse
ntia
lR
efer
toA
.3.f.
It is
mor
e ap
prop
riate
to se
eka
“des
crip
tion”
of h
ow v
aria
bles
wer
e sc
aled
, bin
ned,
or t
rans
form
ed a
s opp
osed
to v
ery
time-
inte
nsiv
e ex
hibi
ts o
f pre
/pos
t-sc
rub
dist
ribut
ions
.
B.4
.dD
escr
ibe
any
trans
form
atio
ns m
ade
for
cont
inuo
us v
aria
bles
.Es
sent
ial
Com
bine
with
B.4
.c.a
nd p
ossi
bly
A.3
.f.
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 22
Page
16
of 2
9 B
.4.e
Esse
ntia
l
Typi
cal p
-val
ues g
reat
er th
an 5
% a
re la
rge
and
shou
ld b
e qu
estio
ned.
Rea
sona
ble
busi
ness
judg
men
t can
som
etim
es
prov
ide
legi
timat
e su
ppor
t for
hig
h p-
valu
es. R
easo
nabl
enes
s of
the
p-va
lue
thre
shol
d co
uld
also
var
y de
pend
ing
on th
e co
ntex
t of t
he m
odel
, e.g
., th
e th
resh
old
mig
ht b
e lo
wer
whe
n m
any
cand
idat
e va
riabl
es w
ere
eval
uate
d fo
r inc
lusi
on in
the
mod
el.
Com
men
tary
shou
ld c
larif
yth
at fo
r var
iabl
es th
at a
re m
odel
ed c
ontin
uous
ly o
nly
the
stat
istic
s aro
und
the
mod
eled
par
amet
ers w
ould
be
prov
ided
. e.g
., w
ould
not
exp
ect t
o pr
ovid
e co
nfid
ence
inte
rval
s aro
und
each
leve
l of a
n A
OI c
urve
. 4.e
-4.n
Com
men
tary
sh
ould
con
side
r tha
tpro
vidi
ng p-values
for e
very
var
iabl
e m
ayno
t be
nece
ssar
y, a
nd
that
pro
vidi
ng th
e ov
eral
l lift
and
acc
urac
yte
sts m
aybe
suff
icie
nt fo
r rev
iew
ing
pred
ictiv
e m
odel
s.
Reg
ulat
ors s
houl
d be
ale
rted
that
cer
tain
pre
dict
ive
mod
els m
ay n
ot g
ener
ate
p-va
lues
or
F te
sts.
B.4
.f
Iden
tify
the
thre
shol
d fo
r st
atis
tical
sign
ifica
nce
and
expl
ain
why
it w
as
sele
cted
. Pro
vide
ave
rbal
def
ense
an
expl
anat
ion
for k
eepi
ng
the
varia
ble
for e
ach
disc
rete
var
iabl
e le
vel
whe
re th
e p-
valu
es w
ere
not l
ess t
han
the
chos
en
thre
shol
d.
Esse
ntia
lSe
e C
omm
ent f
or B
.4.e
.Se
e co
mm
ent u
nder
B.4
.e.
B.4
.g
For o
vera
ll di
scre
te
varia
bles
, pro
vide
type
3
chi-s
quar
e te
sts,
p-va
lues
, F te
sts a
nd a
ny
othe
r rel
evan
t and
m
ater
ial t
est.
Wer
e m
odel
dev
elop
men
t dat
a,
valid
atio
n da
ta, t
est d
ata
or o
ther
dat
a us
ed fo
r th
ese
test
s?
Esse
ntia
lSe
e C
omm
ent f
or B
.4.e
.Se
e co
mm
ent u
nder
B.4
.e.
B.4
.h
For c
ontin
uous
var
iabl
es,
prov
ide
conf
iden
ce
inte
rval
s, ch
i-squ
are
test
s, p-
valu
es a
nd a
ny
othe
r rel
evan
t and
m
ater
ial t
est.
Wer
e m
odel
dev
elop
men
t dat
a,
valid
atio
n da
ta, t
est d
ata
or o
ther
dat
a us
ed fo
r th
ese
test
s?
Esse
ntia
lSe
e C
omm
ent f
or B
.4.e
.Se
e co
mm
ent u
nder
B.4
.e.
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 23
Page
17
of 2
9 B
.4.i
Des
crib
e ho
w th
e m
odel
w
as te
sted
for s
tabi
lity
over
tim
e.Es
sent
ial
Eval
uate
the
build
/test
/val
idat
ion
data
sets
for p
oten
tial m
odel
di
stor
tions
(e.g
., a
win
ter s
torm
in y
ear 3
of 5
can
dis
tort
the
mod
el in
bot
h th
e te
stin
g an
d va
lidat
ion
data
sets
).
Furth
er c
omm
enta
ry is
nee
ded
to c
larif
yw
hat“
esse
ntia
l” su
ppor
t is n
eede
d to
de
mon
stra
te th
e te
stin
g of
“st
abili
tyov
er ti
me”
.
B.4
.j
Des
crib
e ho
w th
e m
odel
w
as te
sted
for
geog
raph
ic st
abili
ty, e
.g.,
acro
ss st
ates
or
terr
itorie
s with
in st
ate.
Esse
ntia
lEv
alua
te th
e ge
ogra
phic
split
s for
pot
entia
l mod
el d
isto
rtion
s.Fu
rther
com
men
tary
is n
eede
d to
cla
rify
wha
t is m
eant
by
“geo
grap
hic
stab
ility
” or
w
hat t
ypes
of e
xhib
its w
ould
pro
vide
info
rmat
ion
rega
rdin
g th
is co
ncer
n.
B.4
.k
Des
crib
e ho
w o
verf
ittin
g w
as a
ddre
ssed
and
the
resu
lts o
f cor
rela
tion
test
s.
Esse
ntia
l
B.4
.l
Prov
ide
supp
ort
dem
onst
ratin
g th
at th
e G
LM a
ssum
ptio
ns a
re
appr
opria
te (f
or
exam
ple,
the
choi
ce o
f er
ror d
istri
butio
n).
Esse
ntia
lV
isua
l rev
iew
of p
lots
of a
ctua
l err
ors i
s usu
ally
suff
icie
nt.
B.4
.m
Prov
ide
the
form
ula
rela
tions
hip
betw
een
the
data
and
the
mod
el
outp
uts,
with
a d
efin
ition
of
eac
h m
odel
inpu
t and
ou
tput
. Pro
vide
all
nece
ssar
y co
effic
ient
s to
eval
uate
the
pred
icte
d va
lue
for
any
real
or
hypo
thet
ical
set o
f in
puts
.
Esse
ntia
lB
.4.l
and
B.4
.m w
ill sh
ow th
e m
athe
mat
ical
func
tions
in
volv
ed a
nd c
ould
be
used
to re
prod
uce
som
e m
odel
pr
edic
tions
.
Com
men
tary
shou
ld a
dvis
e th
e re
gula
tor t
hatp
redi
cted
val
ue m
ay n
ot b
e a
part
of th
em
odel
out
put;
only
rela
tiviti
es. A
n ou
tput
that
repr
esen
ts p
redi
cted
pur
e pr
emiu
m
from
a p
redi
ctiv
e m
odel
shou
ld n
ot b
e co
nfus
ed w
ith a
n ac
tuar
ial e
stim
ate
of
pros
pect
ive
pure
pre
miu
m. P
rovi
ding
this
cou
ld in
clud
e ad
ditio
nal d
etai
l abo
ut
mod
elin
g pr
actic
e in
exc
ess o
f wha
t has
bee
n hi
stor
ical
ly p
rovi
ded
or w
hat i
s nee
ded
to a
sses
s goo
dnes
s of f
it.
B.4
.n
Prov
ide
5-10
sam
ple
reco
rds a
nd th
e ou
tput
of
the
mod
el fo
r tho
se
reco
rds.
Esse
ntia
l
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 24
Page
18
of 2
9 5.
“O
ld M
odel
” Ve
rsus
“N
ew M
odel
”
B.5
.a
An
expl
anat
ion
of w
hy
this
mod
el is
bet
ter t
han
the
one
it is
repl
acin
g.
How
was
that
con
clus
ion
form
ed?
Wha
t met
rics
wer
e re
lied
on fo
r m
easu
rem
ent?
Esse
ntia
lR
egul
ator
s sho
uld
expe
ct to
see
impr
ovem
ent i
n th
e ne
w c
lass
pl
an’s
pre
dict
ive
abili
ty o
r oth
er su
ffic
ient
reas
on fo
r the
ch
ange
.
B.5
.b
Wer
e 2
Gin
i coe
ffici
ents
co
mpa
red?
Wha
t was
the
conc
lusi
on d
raw
n fr
om
this
com
paris
on?
May
Be
Req
uest
edO
ne e
xam
ple
of a
com
paris
on m
ight
be
suff
icie
nt.
B.5
.c
Wer
e do
uble
lift
char
ts
anal
yzed
? W
hat w
as th
e co
nclu
sion
dra
wn
from
th
is a
naly
sis?
Esse
ntia
lO
ne e
xam
ple
of a
com
paris
on m
ight
be
suff
icie
nt.
Whe
n w
ould
this
be
rele
vant
? ->
“May
be
Req
uest
ed”
B.5
.d
Prov
ide
a lis
t of a
ll ne
w
pred
icto
r var
iabl
es in
the
mod
el th
at w
ere
not i
n th
e pr
ior m
odel
.
Esse
ntia
lU
sefu
l to
diff
eren
tiate
bet
wee
n ol
d an
d ne
w v
aria
bles
so th
e re
gula
tor c
an p
riorit
ize
mor
e tim
e on
fact
ors n
ot y
et re
view
ed.
B.5
.e
Prov
ide
a lis
t of
pred
icto
r var
iabl
es u
sed
in th
e ol
d m
odel
that
are
no
t use
d in
the
new
m
odel
. Why
wer
e th
ey
drop
ped
from
the
new
m
odel
?
Esse
ntia
l
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 25
Page
19
of 2
9 6.
M
odel
er/S
oftw
are
B.6
.a
Prov
ide
the
nam
es,
cont
act e
mai
ls, p
hone
nu
mbe
rs a
nd
qual
ifica
tions
of t
he k
ey
pers
ons w
ho:
a.
Led
the
proj
ect
b.
Com
pile
d th
e da
tac.
B
uilt
the
mod
eld.
Pe
rfor
med
pee
r re
view
Esse
ntia
l
The
is n
ot “
esse
ntia
l” o
r rel
evan
t for
pur
pose
s of r
evie
win
g ho
w th
e m
odel
is u
sed
in
deve
lopm
ent o
f the
ratin
g pl
an.W
here
els
e is
such
info
rmat
ion
soug
ht?
For e
ach
filin
g, g
ener
ally
ther
e is
aqu
alifi
edac
tuar
y pr
ovid
ing
the
filin
g an
d si
gnin
g on
beh
alf
of th
e co
mpa
ny. W
here
as th
e co
mpa
ny m
ay o
ffer
this
info
rmat
ion,
it is
the
filin
g ac
tuar
y th
at m
ust d
eter
min
e if
the
mod
el/s
oftw
are
deve
lope
r mus
t be
enga
ged
in o
rder
to
resp
ond
suff
icie
ntly
to th
e re
gula
tor’
s inq
uirie
s.
B.6
.b
Wha
t sof
twar
e w
as
used
? Pr
ovid
e th
e na
me
of th
e so
ftwar
e ve
nder
/dev
elop
er,
softw
are
prod
uct a
nd a
so
ftwar
e ve
rsio
n re
fere
nce.
Esse
ntia
lTh
is sh
ould
be
capt
ured
in th
e su
mm
ary
narr
ativ
e fo
r the
mod
el.
B.6
.c
Whe
n di
d w
ork
to b
uild
th
e m
odel
beg
in a
nd
whe
n w
as th
e m
odel
bu
ild fi
naliz
ed?
May
Be
Req
uest
edEs
sent
ial
The
“ess
entia
l”fo
cus s
houl
d be
mor
e on
the
desc
riptio
n of
the
data
(e.g
.yea
rs o
f dat
a;as
of d
ate)
. If t
he m
odel
is a
com
mer
cial
pro
duct
, the
n th
e co
mpa
ny m
ay w
ant t
o of
fer
the
prod
uctio
n ye
ar, v
ersi
on/m
odel
nam
e.N
ever
thel
ess,
this
shou
ld n
ot b
e es
sent
ial t
o th
e re
view
and
exp
lana
tion
of h
ow th
e m
odel
func
tions
and
impa
cts t
he ra
ting
plan
.
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 26
Page
20
of 2
9 C
. The
File
d Ra
ting
Plan
Info
rmat
ion
Impo
rtan
ce to
R
egul
ator
’s R
evie
w
"Ess
entia
l" o
r "M
ay
Be
Req
uest
ed"
Com
men
tsA
PCIA
1.
Gen
eral
Impa
ct o
f Mod
el o
n Ra
ting
Algo
rith
mTh
e Pr
edic
tive
Mod
el is
not
the
Pric
ing
Mod
el.T
he p
redi
ctiv
e m
odel
impa
cts t
he ra
te
indi
catio
n w
hich
influ
ence
s the
rate
sele
ctio
n w
hich
pro
mpt
s the
rate
just
ifica
tion.
C.1
.a
In th
e A
ctua
rial
Mem
oran
dum
sect
ion
on
the
SER
FF S
uppo
rting
D
ocum
enta
tion
tab,
for
each
mod
el re
lied
upon
, in
clud
e a
docu
men
t tha
t ex
plai
ns th
e m
odel
and
its
role
in th
e ra
ting
syst
em.
Esse
ntia
l
This
item
bec
omes
“Es
sent
ial”
if th
e ro
le o
f the
mod
el c
anno
t be
imm
edia
tely
dis
cern
ed b
y th
e re
view
er fr
om a
qui
ck re
view
of
the
rate
and
/or r
ule
page
s. (I
mpo
rtanc
e is
dep
ende
nton
stat
e re
quire
men
ts a
nd e
ase
of id
entif
icat
ion
by th
e fir
st la
yer o
f re
view
and
esc
alat
ion
to th
e ap
prop
riate
revi
ew st
aff.)
C.1
.b
Prov
ide
an e
xpla
natio
n of
how
the
mod
el w
as
used
to a
djus
t the
ratin
g al
gorit
hm.
Esse
ntia
lC
omm
enta
ry sh
ould
advi
se th
e re
gula
tor t
hat m
odel
sare
gen
eral
ly u
sed
for f
acto
r-ba
sed
indi
catio
ns, w
hich
are
then
use
d as
the
basi
s for
sele
cted
cha
nges
to th
e ra
ting
plan
. It i
s the
cha
nges
to th
e ra
ting
plan
that
cre
ate
impa
cts.
C.1
.c
Prov
ide
a co
mpl
ete
list
of a
ll ch
arac
teris
tics/
varia
bles
us
ed in
the
prop
osed
ra
ting
plan
, inc
ludi
ng
thos
e us
ed a
s inp
ut to
the
mod
el (i
nclu
ding
sub-
mod
els a
nd c
ompo
site
va
riabl
es) a
nd a
ll ot
her
char
acte
ristic
s/va
riabl
es
used
to c
alcu
late
a
prem
ium
. For
eac
h ch
arac
teris
tic/v
aria
ble,
in
dica
te if
it is
onl
y in
put
to th
e m
odel
, whe
ther
it
Esse
ntia
lEx
ampl
es o
f var
iabl
es u
sed
as in
puts
to th
e m
odel
and
use
d a
s se
para
te u
niva
riate
ratin
g ch
arac
teris
tics m
ight
be
crite
ria u
sed
to
dete
rmin
e a
ratin
g tie
r or h
ouse
hold
com
posi
te c
hara
cter
istic
.
This
shou
ld b
e co
vere
d al
read
y in
the
gene
ral d
escr
iptio
n of
the
inpu
t and
mod
el d
ata,
whi
ch is
sepa
rate
from
the
othe
r ris
k ch
arac
teris
tics a
nd v
aria
bles
that
may
be
filed
as
part
of th
e ra
ting
plan
.
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 27
Page
21
of 2
9
is o
nly
a se
para
te
univ
aria
te ra
ting
char
acte
ristic
, or w
heth
er
it is
bot
h in
put t
o th
e m
odel
and
a se
para
te
univ
aria
te ra
ting
char
acte
ristic
. The
list
sh
ould
pro
vide
tra
nspa
rent
des
crip
tions
of
eac
h lis
ted
char
acte
ristic
/var
iabl
e.
C.1
.d
For e
ach
char
acte
ristic
/var
iabl
e us
ed a
s bot
h in
put t
o th
e m
odel
(inc
ludi
ng su
b-m
odel
s and
com
posi
te
varia
bles
) and
as a
se
para
te u
niva
riate
ratin
g ch
arac
teris
tic, e
xpla
in
how
thes
e ar
e te
mpe
red
or a
djus
ted
to a
ccou
nt
for p
ossi
ble
over
lap
or
redu
ndan
cy in
wha
t the
ch
arac
teris
tic/v
aria
ble
mea
sure
s.
Esse
ntia
l
Mod
elin
g lo
ss ra
tio w
ith th
ese
char
acte
ristic
s/va
riabl
es a
s con
trol
varia
bles
wou
ld a
ccou
nt fo
r pos
sibl
e ov
erla
p. T
he in
sure
r sho
uld
addr
ess t
his p
ossi
bilit
y or
oth
er c
onsi
dera
tions
, e.g
., tie
r pl
acem
ent m
odel
s ofte
n us
e ris
k ch
arac
teris
tics/
varia
bles
that
are
al
so u
sed
else
whe
re in
the
ratin
g pl
an.
C.1
.e
If th
e fil
ing
supp
ort
incl
udes
an
upda
te o
r re
plac
emen
t of a
n ex
istin
g m
odel
, ide
ntify
an
d ex
plai
n th
e ch
ange
s in
cal
cula
tions
, as
sum
ptio
ns, p
aram
eter
s an
d da
ta u
sed
to b
uild
th
e m
odel
s. Pr
ovid
e an
ex
plan
atio
n of
why
the
upda
ted/
repl
acem
ent
mod
el is
bet
ter t
han
the
one
it is
repl
acin
g,
incl
udin
g, h
ow th
at
conc
lusi
on w
as re
ache
d,
and
the
met
rics r
elie
d up
on to
reac
h th
at
conc
lusi
on.
Esse
ntia
lC
over
ed u
nder
sect
ion
B.5
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 28
Page
22
of 2
9 2.
Re
leva
nce
of V
aria
bles
/ Re
latio
nshi
p to
Ris
k of
Los
s
C.2
.a
Prov
ide
an e
xpla
natio
n ho
w th
e ch
arac
teris
tics/
ratin
g va
riabl
es, i
nclu
ded
in th
e fil
ed ra
ting
plan
, lo
gica
lly a
nd in
tuiti
vely
re
late
to th
e ris
k of
in
sura
nce
loss
(or
expe
nse)
for t
he ty
pe o
f in
sura
nce
prod
uct b
eing
pr
iced
. Inc
lude
a
disc
ussio
n of
the
rele
vanc
e ea
ch
char
acte
ristic
/ratin
g va
riabl
e ha
s on
cons
umer
beh
avio
r tha
t w
ould
lead
to a
di
ffere
nce
in ri
sk o
f los
s (o
r exp
ense
).
Esse
ntia
lTh
is e
xpla
natio
n w
ould
not
be
need
ed if
the
conn
ectio
n be
twee
n va
riabl
es a
nd ri
sk o
f los
s (or
exp
ense
) has
alre
ady
been
ill
ustra
ted.
Cov
ered
und
erA
.4.b
and
B.3
.c. R
efer
toex
istin
g st
anda
rdsi
n A
SOP
12 R
isk
Cla
ssifi
catio
n-3
.2.2
This
shou
ld b
e th
e sa
me
stan
dard
for m
odel
ed c
hara
cter
istic
s/ra
ting
varia
bles
as f
or
non-
mod
eled
.
3.
Com
pari
son
of M
odel
Out
puts
to C
urre
nt a
nd S
elec
ted
Ratin
g Fa
ctor
s
C.3
.a
Prov
ide
a co
mpa
rison
be
twee
n re
lativ
ities
in
dica
ted
by th
e m
odel
to b
oth
curr
ent
rela
tiviti
es a
nd th
e in
sure
r's se
lect
ed
rela
tiviti
es fo
r eac
h ris
k ch
arac
teris
tic/v
aria
ble
in th
e ra
ting
plan
. Ea
ch si
gnifi
cant
di
ffere
nce
shou
ld b
e hi
ghlig
hted
and
ex
plai
ned.
Esse
ntia
l
“Sig
nific
ant d
iffer
ence
” m
ay v
ary
base
d on
the
risk
char
acte
ristic
/var
iabl
e an
d co
ntex
t. H
owev
er, t
he m
ovem
ent o
f a
sele
cted
rela
tivity
shou
ld b
e in
the
dire
ctio
n of
the
indi
cate
d re
lativ
ity; i
f not
, an
expl
anat
ion
is n
eces
sary
as t
o w
hy th
e m
ovem
ent i
s log
ical
.
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 29
Page
23
of 2
9 C
.3.b
Wha
t cal
cula
tions
, ju
dgm
ents
and
ad
just
men
ts, i
f any
, w
ere
mad
e be
fore
us
ing
the
mod
el
outp
ut in
the
ratin
g sy
stem
? Id
entif
y an
y ad
just
men
ts th
at w
ere
mad
e to
the
indi
cate
d m
odel
to d
eriv
e th
e se
lect
ed m
odel
.
Esse
ntia
lC
over
edun
der C
.3.a
C.3
.c
If th
e m
odel
resu
lts in
sc
ores
, tie
rs, o
r ra
nges
of v
alue
s for
w
hich
indi
catio
ns a
re
then
der
ived
for e
ach
such
resu
lting
ca
tego
ry, p
rovi
de
expl
anat
ions
for f
iled
ratin
g va
lues
that
de
viat
e fr
om th
ese
indi
catio
ns a
nd
supp
ortin
g in
form
atio
n/
anal
yses
. For
ex
ampl
e, id
entif
y an
y ad
just
men
ts th
at w
ere
mad
e to
the
fact
ors
indi
cate
d fo
r eac
h ca
tego
ry o
f the
mod
el
outp
uts t
o de
rive
the
fact
ors s
elec
ted
for
the
ratin
g pl
an.
Esse
ntia
lTh
is is
esp
ecia
lly im
porta
nt if
dev
iatio
ns a
re m
ater
ial a
nd/o
r im
pact
one
con
sum
er p
opul
atio
n m
ore
than
ano
ther
.C
over
und
er C
.3.a
4.
Resp
onse
s to
Dat
a, C
redi
bilit
y an
d G
ranu
lari
ty Is
sues
C.4
.a
Wha
t con
side
ratio
n w
as g
iven
to th
e cr
edib
ility
of t
he
outp
ut d
ata?
Esse
ntia
l
At w
hat l
evel
of g
ranu
larit
y is
cre
dibi
lity
appl
ied.
If m
odel
ing
was
by-
cove
rage
, by-
form
or b
y-pe
ril, e
xpla
in h
ow th
ese
wer
e ha
ndle
d w
hen
ther
e w
as n
ot e
noug
h cr
edib
le d
ata
by c
over
age,
fo
rm o
r per
il to
mod
el.
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 30
Page
24
of 2
9 C
.4.b
If ap
plic
able
, dis
cuss
th
e ra
tiona
le fo
r usi
ng
a m
odel
that
is m
ore
gran
ular
than
the
ratin
g pl
an.
Esse
ntia
lTh
is is
app
licab
le if
the
insu
rer h
ad to
com
bine
mod
eled
out
put
in o
rder
to re
duce
the
gran
ular
ity o
f the
ratin
g pl
an.
Com
men
tary
shou
ld c
larif
y ho
w g
ranu
larit
y is
an
“ess
entia
l” re
gula
tory
con
cern
for
the
ratin
g pl
an.T
his s
houl
d be
cov
ered
und
er th
e ge
nera
l nar
rativ
e of
how
the
mod
el
is im
plem
ente
d an
d in
corp
orat
ed in
to th
e ra
ting
plan
.
C.4
.c
If ap
plic
able
, dis
cuss
th
e ra
tiona
le fo
r usi
ng
a ra
ting
plan
that
is
mor
e gr
anul
ar th
an
mod
eled
out
put.
Esse
ntia
l
A m
ore
gran
ular
ratin
g pl
an im
plie
s tha
t the
insu
rer h
ad to
ex
trapo
late
cer
tain
ratin
g tre
atm
ents
, esp
ecia
lly a
t the
tails
of a
di
strib
utio
n of
attr
ibut
es, i
n a
man
ner n
ot sp
ecifi
ed b
y th
e m
odel
in
dica
tions
.
Com
men
tary
shou
ld c
larif
y ho
w g
ranu
larit
y is
an
“ess
entia
l” re
gula
tory
con
cern
for
the
ratin
g pl
an.T
his s
houl
d be
cov
ered
und
er th
e ge
nera
l nar
rativ
e of
how
the
mod
el
is im
plem
ente
d an
d in
corp
orat
ed in
to th
e ra
ting
plan
.
5.
Def
initi
ons o
f Rat
ing
Varia
bles
C.5
.a
Prov
ide
a tra
nspa
rent
pr
esen
tatio
n an
d ex
plan
atio
n of
bi
nnin
g de
cisi
ons t
hat
assi
gn ra
nges
of
mod
el o
utpu
ts to
pa
rticu
lar r
atin
g ca
tego
ries.
Esse
ntia
lC
ombi
ne w
ith A
.3.f
C.5
.b
Prov
ide
com
plet
e de
finiti
ons o
f any
ra
ting
tiers
or o
ther
in
term
edia
te ra
ting
cate
gorie
s tha
t tra
nsla
te th
e m
odel
ou
tput
s int
o so
me
othe
r stru
ctur
e th
at is
th
en p
rese
nted
with
in
the
rate
and
/or r
ule
page
s.
Esse
ntia
l
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 31
Page
25
of 2
9 6.
Su
ppor
ting
Dat
a
C.6
.a
Prov
ide
stat
e-sp
ecifi
c, b
ook-
of-
busi
ness
-spe
cific
un
ivar
iate
his
toric
al
expe
rienc
e da
ta
cons
istin
g of
, at
min
imum
, ear
ned
prem
ium
s, in
curr
ed
loss
es, l
oss r
atio
s and
lo
ss ra
tio re
lativ
ities
fo
r eac
h ca
tego
ry o
f m
odel
out
put(s
) pr
opos
ed to
be
used
w
ithin
the
ratin
g pl
an.
May
Be
Req
uest
edEs
sent
ial
Com
men
tary
shou
ld c
autio
n re
gula
tor t
hat a
lterin
g un
ivar
iate
resu
lts fo
r mod
els b
uilt
with
cou
ntry
wid
e da
ta a
t a st
ate
leve
l may
not b
e st
atis
tical
ly c
redi
ble
or a
ppro
pria
te.
Loss
ratio
resu
lts m
ay n
otbe
com
para
ble
to re
sults
from
a m
odel
sinc
e th
ese
resu
lts
refle
ct th
e ex
istin
g ra
ting
plan
as w
ell a
s los
ses.
Prov
idin
g st
ate-speci
fic d
ata
for e
very
va
riabl
e m
ayno
t mak
e se
nse
beca
use
such
dat
a w
ould
not
be
cred
ible
at t
he st
ate
leve
l in
man
y in
stan
ces.
The
leve
l of i
nfor
mat
ion
may
als
o be
con
side
red
prop
rieta
ry.
C.6
.b
Prov
ide
an
expl
anat
ion
of a
ny
mat
eria
l (es
peci
ally
di
rect
iona
l) di
ffere
nces
bet
wee
n m
odel
indi
catio
ns
and
stat
e-sp
ecifi
c un
ivar
iate
in
dica
tions
.
May
Be
Req
uest
edEs
sent
ial
Mul
tivar
iate
indi
catio
ns m
ay b
e re
ason
able
as r
efin
emen
ts to
un
ivar
iate
indi
catio
ns, b
ut li
kely
not
for b
ringi
ng a
bout
reve
rsal
s of
thos
e in
dica
tions
. For
inst
ance
, if t
he u
niva
riate
indi
cate
d re
lativ
ity fo
r an
attri
bute
is 1
.5 a
nd th
e m
ultiv
aria
te in
dica
ted
rela
tivity
is 1
.25,
this
is p
oten
tially
a p
laus
ible
app
licat
ion
of th
e m
ultiv
aria
te te
chni
ques
. If,
how
ever
, the
uni
varia
te in
dica
ted
rela
tivity
is 0
.7 a
nd th
e m
ultiv
aria
te in
dica
ted
rela
tivity
is 1
.25,
a
regu
lato
r may
que
stio
n w
heth
er th
e at
tribu
te in
que
stio
n is
ne
gativ
ely
corr
elat
ed w
ith o
ther
det
erm
inan
ts o
f ris
k. C
redi
bilit
y of
stat
e da
ta sh
ould
be
cons
ider
ed w
hen
stat
e in
dica
tions
diff
er
from
mod
eled
resu
lts b
ased
on
a br
oade
r dat
a se
t. H
owev
er, t
he
rele
vanc
e of
the
broa
der d
ata
set t
o th
e ris
ks b
eing
pric
ed sh
ould
al
so b
e co
nsid
ered
.
Com
men
tary
shou
ld c
autio
n th
e re
gula
tor t
hatm
akin
g m
odel
mod
ifica
tions
at a
stat
e le
vel m
ay n
ot b
e ap
prop
riate
usi
ng st
ate
spec
ific
data
resu
lts. A
lterin
g un
ivar
iate
re
sults
for m
odel
s bui
lt w
ith c
ount
ryw
ide
data
at a
stat
e le
vel m
ayno
t be
stat
istic
ally
cr
edib
le o
r app
ropr
iate
. The
regu
lato
r sho
uld
be a
dvis
ed th
at a
lthou
gh th
ere
is
pote
ntia
l for
thes
e ty
pe o
f rev
ersa
ls to
reve
al a
con
cern
in th
e m
odel
, it i
s not
at a
ll un
com
mon
or u
nexp
ecte
d to
see
this
eff
ect.
e.g.
, a u
niva
riate
revi
ew in
dica
ting
high
er
pure
pre
miu
ms f
or h
omes
with
a so
phis
ticat
edal
arm
syst
em;a
n ef
fect
refle
ctin
gal
arm
s in
hom
es o
f hig
her t
han
aver
age
valu
e an
d a
“rev
ersa
l” e
xpec
ted
unde
r a
mul
tivar
iate
revi
ew. H
owev
er, r
elat
ions
hips
can
be
very
com
plic
ated
and
diff
icul
t to
teas
e ou
t in
prac
tice.
The
filin
g ac
tuar
y sh
ould
be
prep
ared
to p
rovi
deth
ese
expl
anat
ions
,cor
rela
tions
and
inte
ract
ions
in d
ata.
7.
Con
sum
er Im
pact
sH
ow d
oes s
ectio
n C
.7 a
pply
in o
ther
com
pone
nts o
f the
rate
filin
g? T
he “
esse
ntia
l”
focu
s sho
uld
be o
n ho
w th
e m
odel
in q
uest
ion
impa
cts t
he ra
ting
plan
.
C.7
.a
Iden
tify
mod
el
chan
ges a
nd ra
ting
varia
bles
that
will
ca
use
larg
e pr
emiu
m
disr
uptio
ns.
Esse
ntia
lC
omm
enta
ry sh
ould
focu
s reg
ulat
or’s
atte
ntio
n on
the
key
ratin
g fa
ctor
s tha
t cou
ld b
e th
e ca
use
ofa
larg
e pr
emiu
m sw
ing.
Ove
rall
rate
cha
nge
hist
ogra
ms c
an sh
ow
chan
ges i
n th
e ke
y se
lect
ed fa
ctor
s.
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 32
Page
26
of 2
9 C
.7.b
Did
the
insu
rer
perf
orm
sens
itivi
ty
test
ing
to id
entif
y si
gnifi
cant
cha
nges
in
prem
ium
due
to sm
all
or in
crem
enta
l ch
ange
in a
sing
le
risk
char
acte
ristic
? If
so
, dis
cuss
and
pr
ovid
e th
e re
sults
of
that
test
ing.
May
Be
Req
uest
ed
One
way
to se
e se
nsiti
vity
is to
ana
lyze
a g
raph
of e
ach
risk
char
acte
ristic
’s/v
aria
ble’
s pos
sibl
e re
lativ
ities
. Loo
k fo
r si
gnifi
cant
var
iatio
n be
twee
n ad
jace
nt re
lativ
ities
and
eva
luat
e if
such
var
iatio
n is
reas
onab
le.
C.7
.c
Mea
sure
and
des
crib
e th
e im
pact
s on
expi
ring
polic
ies a
nd
desc
ribe
the
proc
ess
used
by
man
agem
ent
to m
itiga
te o
r get
co
mfo
rtabl
e w
ith
thos
e im
pact
s.
Esse
ntia
lTh
is is
not
rele
vant
to th
e re
view
of h
ow th
e m
odel
is im
plem
ente
d an
d in
corp
orat
ed
into
the
ratin
g pl
an
C.7
.d
Prov
ide
a ra
te
disr
uptio
n an
alys
is,
dem
onst
ratin
g th
e di
strib
utio
n of
pe
rcen
tage
impa
cts
on re
new
al b
usin
ess
(cre
ate
by re
ratin
g th
e cu
rren
t boo
k of
bu
sine
ss).
Incl
ude
the
larg
est d
olla
r and
pe
rcen
tage
impa
cts
aris
ing
from
the
filin
g, in
clud
ing
(des
irabl
y) th
e im
pact
s aris
ing
spec
ifica
lly fr
om th
e ad
optio
n of
the
mod
el
or c
hang
es to
the
mod
el a
s the
y tra
nsla
te in
to th
e pr
opos
ed ra
ting
plan
.
Esse
ntia
l
Whi
le th
e de
faul
t req
uest
wou
ld ty
pica
lly b
e fo
r the
dis
tribu
tion
of im
pact
s at t
he o
vera
ll fil
ing
leve
l, th
e re
gula
tor m
ay n
eed
to
delv
e in
to th
e m
ore
gran
ular
var
iabl
e-sp
ecifi
c ef
fect
s of r
ate
chan
ges i
f the
re is
con
cern
abo
ut p
artic
ular
var
iabl
es h
avin
g ex
trem
e or
dis
prop
ortio
nate
impa
cts,
or si
gnifi
cant
impa
cts t
hat
have
oth
erw
ise
yet t
o be
subs
tant
iate
d.Se
e A
ppen
dix
C fo
r an
exam
ple
of a
dis
rupt
ion
anal
ysis
.
This
is n
ot re
leva
nt to
the
revi
ew o
f how
the
mod
el is
impl
emen
ted
and
inco
rpor
ated
in
to th
e ra
ting
plan
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 33
Page
27
of 2
9 C
.7.e
Prov
ide
expo
sure
di
strib
utio
ns fo
r ou
tput
var
iabl
es a
nd
show
the
effe
cts o
f ra
te c
hang
es a
t gr
anul
ar a
nd
sum
mar
y le
vels
.
May
Be
Req
uest
edEs
sent
ial
See
App
endi
x C
for a
n ex
ampl
e of
an
expo
sure
dis
tribu
tion.
The
regu
lato
r sho
uld
cons
ider
that
this
info
rmat
ion
may
be c
onsid
ered
pro
prie
tary
C.7
.f
Expl
ain
how
the
insu
rer
will
hel
p ed
ucat
e co
nsum
ers t
o m
itiga
te th
eir r
isk.
May
Be
Req
uest
edEs
sent
ial
This
is n
ot re
leva
nt to
the
revi
ew o
f how
the
mod
el is
impl
emen
ted
and
inco
rpor
ated
in
to th
e ra
ting
plan
C.7
.g
Iden
tify
sour
ces t
o be
us
ed a
t "po
int o
f sal
e"
to p
lace
indi
vidu
al
risks
with
in th
e m
atrix
of r
atin
g sy
stem
cl
assi
ficat
ions
. How
ca
n a
cons
umer
ve
rify
thei
r ow
n "p
oint
-of-
sale
" da
ta
and
corr
ect a
ny
erro
rs?
May
Be
Req
uest
edEs
sent
ial
Cou
ld b
e "E
ssen
tial"
if th
e va
riabl
es/ c
hara
cter
istic
s use
d co
uld
1) h
ave
publ
ic-p
olic
y im
plic
atio
ns, 2
) res
ult i
n er
rone
ous
info
rmat
ion
bein
g us
ed, o
r 3) r
esul
t in
man
y la
rge,
dis
rupt
ive
prem
ium
cha
nges
at r
enew
al. A
noth
er c
onsi
dera
tion
to ju
dge
“im
porta
nce”
is w
heth
er c
onsu
mer
s are
pro
activ
ely
invo
lved
(e
.g.,
use
of c
onsu
mer
cre
dit i
nfor
mat
ion
and
cred
it-re
port
accu
racy
issu
es).
This
is n
ot re
leva
nt to
the
revi
ew o
f how
the
mod
el is
impl
emen
ted
and
inco
rpor
ated
in
to th
e ra
ting
plan
C.7
.h
Iden
tify
ratin
g va
riabl
es th
at re
mai
n st
atic
ove
r a
cons
umer
’s li
fetim
e ve
rsus
thos
e th
at w
ill
be u
pdat
ed
perio
dica
lly.
Doc
umen
t gui
delin
es
for v
aria
bles
that
are
lis
ted
as st
atic
yet
for
whi
ch th
e un
derly
ing
cons
umer
attr
ibut
es
may
cha
nge
over
tim
e.
May
Be
Req
uest
edEs
sent
ial
This
is n
ot re
leva
nt to
the
revi
ew o
f how
the
mod
el is
impl
emen
ted
and
inco
rpor
ated
in
to th
e ra
ting
plan
C.7
.i
Prov
ide
the
regu
lato
r w
ith a
des
crip
tion
of
how
the
com
pany
w
ill re
spon
d to
co
nsum
ers’
inqu
iries
May
Be
Req
uest
edEs
sent
ial
This
is n
ot re
leva
nt to
the
revi
ew o
f how
the
mod
el is
impl
emen
ted
and
inco
rpor
ated
in
to th
e ra
ting
plan
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 34
Page
28
of 2
9
abou
t how
thei
r pr
emiu
m w
as
calc
ulat
ed.
C.7
.j
Prov
ide
the
regu
lato
r w
ith a
mea
ns to
ca
lcul
ate
the
rate
ch
arge
d a
cons
umer
.
May
Be
Req
uest
ed
Espe
cial
ly fo
r a c
ompl
ex m
odel
or r
atin
g pl
an, a
scor
e or
pr
emiu
m c
alcu
lato
r via
Exc
el o
r sim
ilar m
eans
wou
ld b
e id
eal,
but t
his c
ould
be
elic
ited
on a
cas
e-by
-cas
e ba
sis.
Abi
lity
to
calc
ulat
e th
e ra
te c
harg
ed c
an a
llow
the
regu
lato
r to
perfo
rm
sens
itivi
ty te
stin
g w
hen
ther
e ar
e sm
all c
hang
es to
a ri
sk
char
acte
ristic
/var
iabl
e.
The
regu
lato
r sho
uld
cons
ider
that
the
com
pany
may
not
be
able
toha
nd o
ver a
prop
rieta
ry ra
ting
mod
el a
s par
t oft
he re
view
of t
he p
redi
ctiv
e m
odel
.In
othe
r cas
es,
insu
rers
may
not
be
able
topr
ovid
e re
gula
tors
with
the
mea
ns to
recr
eate
the
outp
ut o
f a
prop
rieta
ry m
odel
.
Reg
ulat
ors s
houl
d be
cau
tione
d th
at if
insu
ranc
e co
mpa
nies
are
requ
ired
to b
uild
an
Exce
l-type too
l for
regu
lato
rs to
be
able
to c
alcu
late
the
prem
ium
cha
rged
, it c
ould
hi
nder
the
filin
g re
view
pro
cess
and
det
er a
ny in
vest
men
t in
inno
vativ
e ra
ting/
mod
elin
g te
chni
ques
.
8.
Accu
rate
Tra
nsla
tion
of M
odel
into
a R
atin
g Pl
an
C.8
.a
Prov
ide
suff
icie
nt
info
rmat
ion
for t
he
revi
ewer
to b
e ab
le to
un
ders
tand
how
the
mod
el o
utpu
ts a
re
used
with
in th
e ra
ting
syst
em a
nd to
ver
ify
that
the
ratin
g pl
an, i
n fa
ct, r
efle
cts t
he
mod
el o
utpu
t and
any
ad
just
men
ts m
ade
to
the
mod
el o
utpu
t.
Esse
ntia
l
III. D
O R
EGU
LATO
RS N
EED
BEST
PRA
CTIC
ES T
O R
EVIE
W P
REDI
CTIV
E M
ODE
LS?
A G
LM c
onsi
sts o
f thr
ee e
lem
ents6
:• A
pro
babi
lity
dist
ribu
tion
from
the
expo
nent
ial f
amily
.
1
APCI
A: S
tric
tly sp
eaki
ng th
is is
not t
rue
as G
LM u
ses q
uasi-
likel
ihoo
d ra
ther
than
act
ual p
roba
bilit
y di
strib
utio
ns.
The
first
ele
men
t in
the
bulle
t poi
nt li
st sh
ould
be
a “v
aria
nce
func
tion”
, not
a p
roba
bilit
y di
strib
utio
n.
In a
dditi
on, G
LM o
utpu
t is a
ssum
ed, a
s par
t of t
he m
odel
des
ign,
to b
e 10
0% c
redi
ble
no m
atte
r the
size
of t
he u
nder
lyin
g da
ta se
t.
APCI
A: T
his s
tate
men
t is f
alse
. An
adv
anta
ge o
f GLM
is th
at w
e ca
n as
sign
stan
dard
err
ors a
nd c
onfid
ence
inte
rval
s to
the
estim
ator
s. I
thin
k th
e in
tent
ion
was
to c
autio
n th
at G
LM p
oint
-est
imat
e re
sults
are
not
go
spel
trut
h an
d ne
ed to
be
eval
uate
d fo
r cre
dibi
lity
of v
olum
e an
d di
agno
stic
s on
the
mod
el a
ssum
ptio
ns.
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 35
Page
29
of 2
9 IV
. SC
OPE
Rec
omm
ende
d ed
its a
s fol
low
s:
The
focu
s of t
his p
aper
will
be
on G
LM
s use
d to
cre
ate
priv
ate
pass
enge
r au
tom
obile
and
hom
eow
ners
’ ins
uran
ce r
atin
g pl
ans.
Gui
danc
e an
d th
e kn
owle
dge
need
ed to
revi
ew p
redi
ctiv
e m
odel
s ide
ntifi
ed in
this
pap
er a
re, i
n- la
rge
part
,may
not
be
dire
ctly
tran
sfer
rabl
e to
oth
er ty
pes o
f pre
dict
ive
mod
els,
to o
ther
line
s of b
usin
ess,
or o
ther
insu
ranc
e en
deav
ors,
e.g
., co
mm
erci
al a
utom
obile
, wor
kers
’ com
pens
atio
n, m
arke
ting,
und
erw
ritin
g, o
r cla
ims.
Whe
n tr
ansf
errin
g gu
idan
ce to
oth
er li
nes o
f bus
ines
s or o
ther
insu
ranc
e en
deav
ors,
uni
que
cons
ider
atio
ns n
eed
to b
e gi
ven
to
the
cred
ibili
ty o
f the
dat
a an
d le
ss h
omog
eneo
us n
atur
e of
com
mer
cial
bus
ines
s,m
ay a
rise
depe
ndin
g on
the
cont
ext i
n w
hich
how
a p
redi
ctiv
e m
odel
is p
ropo
sed
to b
e de
ploy
ed, t
he u
ses t
o w
hich
it is
pro
pose
d to
be
put,
and
the
pote
ntia
l con
sequ
ence
s of a
n in
sure
r act
ing
on th
e ou
tput
of a
ny g
iven
pre
dict
ive
mod
el. T
his p
aper
doe
s not
del
ve in
to th
ese
poss
ible
con
side
ratio
ns b
ut re
gula
tors
shou
ld b
e pr
epar
ed to
add
ress
them
as t
hey
aris
e.
APCI
A: T
rans
fera
bilit
y - T
he c
onsid
erat
ions
for c
omm
erci
al li
nes m
ay d
iffer
from
per
sona
l lin
es, a
nd c
erta
inly
the
guid
ance
her
e re
late
d to
pre
dict
ive
mod
els i
ncor
pora
ted
into
file
d ra
ting
plan
s is e
ven
mor
e di
stan
ced
from
mod
els u
sed
in th
e co
ntex
t of m
arke
ting
or c
laim
s. A
dditi
onal
ly, h
avin
g a
one-
size-
fits a
ll ap
proa
ch to
eva
luat
ing
mod
els
acro
ss a
ll re
gula
tory
func
tions
, ins
uran
ce a
pplic
atio
ns, a
nd li
nes o
f bus
ines
s wou
ld b
e ov
er-c
onsu
min
g to
the
detr
imen
t of t
he sh
ared
goa
l for
this
effo
rt.
V. C
ON
FIDE
NTI
ALIT
Y
APCI
A: G
reat
er a
tten
tion
need
s to
be g
iven
to th
is se
ctio
n. B
efor
e su
ch “
best
pra
ctic
es”
can
be e
mpl
oyed
, the
regu
lato
r mus
t con
sider
that
ext
ensio
n or
enh
ance
men
t of c
onfid
entia
lity
prot
ectio
ns m
ay b
e ne
cess
ary
to e
xcha
nge
such
mod
elin
g in
telle
ctua
l pro
pert
y.
VI. G
UID
ANCE
FO
R RE
GULA
TORY
REV
IEW
OF
PRED
ICTI
VE M
ODE
LS (B
EST
PRAC
TICE
S)
APCI
A: R
egul
ator
y gu
idan
ce n
eede
d to
cla
rify
dete
rmin
atio
n of
unf
airly
disc
rimin
ator
y of
‘inp
uts’
and
‘out
puts
’. Ho
w d
o ‘o
utpu
ts’ v
ersu
s ‘in
puts
’ get
eva
luat
ed a
s fai
r/un
fair.
VIII.
PRO
POSE
D CH
ANG
ES T
O T
HE P
RODU
CT F
ILIN
G R
EVIE
W H
ANDB
OO
K
XIII.
APP
ENDI
X B
- - G
LOSS
ARY
OF
TERM
S
PCA
appr
oach
(Prin
cipa
l Com
pone
nt A
naly
sis) –
The
PCA
app
roac
h is
als
o kn
own
as fa
ctor
ana
lysi
s. T
hese
met
hods
cre
ate
mul
tiple
new
var
iabl
es fr
om c
orre
late
d gr
oups
of p
redi
ctor
s. T
hose
new
var
iabl
es e
xhib
it lit
tle o
r no
corr
elat
ion
betw
een
them
—th
ereb
y m
akin
g th
em p
oten
tially
mor
e us
eful
in a
GLM
. A P
CA in
a fi
ling
can
be d
escr
ibed
as “
a G
LM w
ithin
a G
LM.”
One
of t
he m
ore
com
mon
app
licat
ions
of P
CA is
geo
dem
ogra
phic
ana
lysi
s, w
here
man
y at
trib
utes
are
use
d to
mod
ify te
rrito
rial d
iffer
entia
ls o
n, fo
r exa
mpl
e, a
cen
sus b
lock
leve
l.
APCI
A: S
tric
tly sp
eaki
ng, f
acto
r ana
lysis
is a
diff
eren
t tec
hniq
ue th
at tr
ies t
o se
para
te o
ut ra
ndom
var
ianc
e fr
om la
tent
var
iabl
e va
rianc
e. R
e-ph
rase
to sa
y th
at th
ey a
re si
mila
r or r
elat
ed a
ppro
ache
s.
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 36
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 37
California DOI CommentsDraft: 10/25/18As adopted by the Casualty Actuarial and Statistical (C) Task Force on XX/XX/XX
© 2018 National Association of Insurance Commissioners 1
Casualty Actuarial and Statistical (C) Task Force
Regulatory Review of Predictive Models
Table of Contents
I. Introduction ...............................................................................................................................................................2 II. What is a “Best Practice?”..........................................................................................................................................2 III. Do Regulators Need Best Practices to Review Predictive Models? ..............................................................................3 IV. Scope.........................................................................................................................................................................4 V. Confidentiality...........................................................................................................................................................5 VI. Guidance for Regulatory Review of Predictive Models (Best Practices) ......................................................................5 VII. Predictive Models – Information for Regulatory Review.............................................................................................6 VIII. Proposed Changes to the Product Filing Review Handbook ...................................................................................... 22 IX. Proposed State Guidance.......................................................................................................................................... 22 X. Other Considerations................................................................................................................................................ 22 XI. Recommendations Going Forward ........................................................................................................................... 22 XII. Appendix A – Best Practice Development ................................................................................................................ 23 XIII. Appendix B - - Glossary of Terms............................................................................................................................ 24 XIV. Appendix C – Sample Rate-Disruption Template...................................................................................................... 25 XV. Appendix D – Information Needed by Regulator Mapped into Best Practices............................................................ 28 XVI. Appendix E – References ......................................................................................................................................... 28
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 38
California DOI CommentsDraft: 10/25/18As adopted by the Casualty Actuarial and Statistical (C) Task Force on XX/XX/XX
© 2018 National Association of Insurance Commissioners 2
I. INTRODUCTION
Insurers’ use of predictive analytics along with big data has significant potential benefits to both consumers and insurers.by transforming the insurer-consumer experience into a more meaningful relationship. Predictive analytics can reveal insights into the relationship between consumer behavior and the cost of insurance, lower the cost of insurance for many, and provide incentives tools for the consumers to better control and mitigate loss. However, predictive analytic techniques are evolving rapidly and leaving many regulators without the necessary tools to effectively review insurers’ use of predictive models in insurance applications.
When a rate plan is truly innovative, the insurer must anticipate or imagine the reviewers’ interests because reviewers will respond with unanticipated questions and have unique educational needs. Insurers can learn from the questions, teach the reviewers, and so forth. When that back-and-forth learning is history, filing requirements and insurer presentations can be routinely organized to meet or exceed reviewers’ needs and expectations. Hopefully, this paperhelps bring more consistency and even uniformity to the art of reviewing predictive models within a rate filing.
The Casualty Actuarial and Statistical (C) Task Force (CASTF) has been charged with identifying best practices to serve as a guide to state insurance departments in their review of predictive models1 underlying rating plans. There were two charges given to CASTF by the Property and Casualty Insurance (C) Committee at the request of the Big Data (EX) Working Group:
A. Draft and propose changes to the Product Filing Review Handbook to include best practices for review of predictive models and analytics filed by insurers to justify rates.
B. Draft and propose state guidance (e.g., information, data) for rate filings that are based on complex predictive models.
This paper will identify best practices when reviewing predictive models and analytics filed by insurers with regulators to justify rates and provide state guidance for review of rate filings based on predictive models. Upon adoption of this paper by the Executive (EX) Committee and Plenary, the Task Force will evaluate how to incorporate these best practices into the Product Filing Review Handbook and will recommend such changes to the Speed to Market (EX) Working Group.
II. WHAT IS A “BEST PRACTICE?”
A best practice is a form of program evaluation in public policy. At its most basic level, a practice is a “tangible and visible behavior… [based on] an idea about how the actions…will solve a problem or achieve a goal” 2. Best practices are used to maintain quality as an alternative to mandatory legislated standards and can be based on self-assessment or benchmarking.3 Therefore, a best practice represents an effective method of problem solving.
A. Key Regulatory Principles
In this paper, best practices are based on the following principles that promote a comprehensive and coordinated review of predictive models across states:
1 In this paper, reference to “model” or “predictive model” are the same as “complex predictive model” unless qualified.2 Bardach, E. and Patashnik, E. (2016.) A Practical Guide for Policy Analysis, The Eightfold Path to More Effective Problem Solving.
Thousand Oaks, CA: CQ Press. See Appendix A for an overview of Bardach’s best-practice analysis. 3 Bogan, C.E. and English, M.J. (1994). Benchmarking for Best Practices: Winning Through Innovative Adaptation. New York: McGraw-
Hill.
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 39
California DOI CommentsDraft: 10/25/18As adopted by the Casualty Actuarial and Statistical (C) Task Force on XX/XX/XX
© 2018 National Association of Insurance Commissioners 3
1. State insurance regulators will maintain their current rate regulatory authority.
2. State insurance regulators will be able to share information to aid companies in getting insurance products to market more quickly.
3. State insurance regulators will share expertise and discuss technical issues regarding predictive models.
4. State insurance regulators will maintain confidentiality, where appropriate, regarding predictive models.
In this paper, best practices are presented in the form of guidance to regulators who review predictive models and to insurance companies filing rating plans that incorporate predictive models. Guidance will identify specific information useful to a regulator in the review of a predictive model, comment on what might be important about that information and, where appropriate, provide insight as to when the information might identify an issue the regulator needs to be aware of or explore further.
III. DO REGULATORS NEED BEST PRACTICES TO REVIEW PREDICTIVE MODELS?
The term “predictive model” refers to a set of models that use statistics to predict outcomes4. When applied to insurance, the model is chosen to estimate the probability or expected value of an outcome given a set amount of input data; for example, models can predict the frequency of loss, the severity of loss, or the pure premium. The generalized linear model (GLM)5 is a commonly used predictive model in insurance applications, particularly in building an insurance product’s rating plan.
Depending on definitional boundaries, predictive modeling can sometimes overlap with the field of machine learning. In this modeling space, predictive modeling is often referred to as predictive analytics.
Before GLMs became vogue, rating plans were built using univariate methods. Univariate methods were considered intuitive and easy to demonstrate the relationship to costs (loss and/or expense). Today, many insurers consider univariate methods too simplistic since they do not take into account the interaction (or dependencies) of the selected input variables and they may imply an assumption of a constant variance across the range of a target variable.
According to many in the insurance industry, GLMs introduce significant improvements over univariate-based rating plans by automatically adjusting for correlations among input variables. Today, the majority of predictive models used in private passenger automobile and homeowners’ rating plans are GLMs. However, GLM results are not always intuitive,and the relationship to costs may be difficult to explain. This is one of the primary reasons regulators can benefit from best practices.
A GLM consists of three elements6:
A probability distribution from the exponential family.
1
As can be seen in the description of the three GLM components above, it may take more than a casual introduction to statistics to comprehend the construction of a GLM. As stated earlier, a downside to GLMs is that it is more challenging to interpret the GLMs output than with univariate models. In addition, GLM output is assumed, as part of the model design, to be 100% credible no matter the size of the underlying data set. Because of this presumption in credibility, which may or may not be valid in practice, the modeler and the regulator reviewing the model would need to engage inthoughtful consideration when incorporating GLM output into a rating plan to ensure that model predictiveness is not compromised by any lack of actual credibility.
4 A more thorough exploration of different predictive models will be found in many statistics’ books, including Geisser,
Seymour(September 2016). Predictive Inference: An Introduction. New York: Chapman & Hall.5 The generalized linear model (GLM) is a flexible family of models that are unified under a single method. Types of GLM include logistic
regression, Poisson regression, gamma regression and multinomial regression. 6 More information on model elements can be found in most statistics’ books.
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 40
California DOI CommentsDraft: 10/25/18As adopted by the Casualty Actuarial and Statistical (C) Task Force on XX/XX/XX
© 2018 National Association of Insurance Commissioners 4
To further complicate regulatory review of models in the future, modeling methods are evolving rapidly and not limited to GLMs. As computing power grows exponentially, it is opening up the modeling world to more sophisticated forms of data acquisition and data analysis. Insurance actuaries and data scientists are seeking increased predictiveness by using even more complex predictive modeling methods. Examples of these are predictive models utilizing random forests, decision trees, neural networks, or combinations of available modeling methods (often referred to as ensembles). These evolving techniques will make the regulators’ understanding and oversight of filed rating plans incorporating predictive models even more challenging.
In addition to the growing complexity of predictive models, many state insurance departments do not have in-house actuarial support or have limited resources to contract out for support when reviewing rate filings that include use of predictive models. The Big Data (EX) Working Group identified the need to provide states with guidance and assistance when reviewing predictive models underlying filed rating plans.7 The Working Group circulated a proposal addressing aid to state insurance regulators in the review of predictive models as used in private passenger automobile and homeowners’ insurance rate filings. This proposal was circulated to all of the Working Group members and interested parties on December 19, 2017, for a public comment period ending January 12, 2018.8 The Big Data Working Groupeffort resulted in the new CASTF charges (see the Introduction section) with identifying best practices that provide guidance to states in the review of predictive models.
So, to get to the question asked by the title of this section: Do regulators need best practices to review predictive models? It might be better to ask this question another way: Are best practices in the review of predictive models of value to regulators and insurance companies? The answer is “yes” to both questions. Best practices will aid regulatory reviewers by raising their level of model understanding. However, best practices are not intended to create standards for filings that include predictive models. Rather, best practices will assist the states in identifying the model elements they should be looking for in a filing that will aid the regulator in understanding why the company believes that the filed predictive model improves the company’s rating plan, making that rating plan fairer to all consumers in the marketplacemaking the company more competitive in the marketplace. To make this work, both regulators and industry need to recognize that:
Best practices merely provide guidance to regulators in their essential and authoritative role over the rating plans in their state.
All states may have a need to review predictive models whether that occurs with approval of rating plans or in amarket conduct exam. Best practices help the regulator identify elements of a model that may influence the regulatory review as to whether modeled rates are appropriately justified. Each regulator needs to decide if the insurer’s proposed rates are compliant with state laws and regulations and whether to act on that information.
Best practices will lead to improved quality in predictive model reviews across states, aiding speed to market and competitiveness of the state marketplace.
Best practices provide a framework for states to share knowledge and resources to facilitate the technical review of predictive models.
Best practices aid training of new regulators and/or regulators new to reviewing predictive models. (This is especially useful for those regulators who do not actively participate in NAIC discussions related to the subject of predictive models.)
Each regulator adopting best practices will be better able to identify the resources needed to assist their state in the review of predictive models.
Lastly, from this point on in this paper, best practices will be referred to as “guidance.” This reference is in line with the intent of this paper to support individual state autonomy in the review of predictive models.
IV. SCOPE
The focus of this paper will be on GLMs used to create private passenger automobile and homeowners’ insurance rating plans.
7 Minutes of the Big Data (EX) Working Group, March 9, 2018: https://secure.naic.org/secure/minutes/2018_spring/ex_it_tf.pdf?598 All comments received by the end of January were posted to the NAIC website March 12 for review.
Commented [LKW1]: The regulator’s role is not to improve the competitiveness of a particular company in the marketplace, but rather to ensure that rates are fair and the marketplace as a whole is competitive.
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 41
California DOI CommentsDraft: 10/25/18As adopted by the Casualty Actuarial and Statistical (C) Task Force on XX/XX/XX
© 2018 National Association of Insurance Commissioners 5
Guidance and the knowledge needed to review predictive models identified in this paper are, in large part, transferrable to other types of predictive models, to other lines of business, or other insurance endeavors, e.g., commercial automobile, workers’ compensation, marketing, underwriting, or claims. When transferring guidance to other lines of business or other insurance endeavors, unique considerations may arise depending on the context in which a predictive model is proposed to be deployed, the uses to which it is proposed to be put, and the potential consequences of an insurer acting on the output of any given predictive model. This paper does not delve into these possible considerations but regulators should be prepared to address them as they arise.
V. CONFIDENTIALITY
Insurers and regulators should be aware that a rate filing might become part of the public record. Each state determines the confidentiality of a rate filing, supplemental material to the filing, when filing information might become public, the procedure to request that filing information be held confidentially, and the procedure by which a public records request is made. It is incumbent on an insurer to be familiar with each state’s laws regarding the confidentiality of information submitted with their rate filing.
VI. GUIDANCE FOR REGULATORY REVIEW OF PREDICTIVE MODELS (BEST PRACTICES)
Encourage and maintain a competition competitive among insurersmarketplace.
Protect the confidentiality of filed predictive models and supporting information (according to state law).
Review a predictive model efficiently.
Obtain a clear understanding of the characteristics that are input to a predictive model (and its sub-models), their relationship to each other and their relationship to non-modeled characteristics/variables used to calculate a risk’s premium.
Determine that individual input characteristics to a predictive model are related to the expected loss or expense differences in risk. Each input characteristic should have an intuitive or demonstrable actual relationship to expected loss or expense.
Determine that the data used as input to the predictive model is accurate, including a clear understanding how missing values, erroneous values and outliers are handled.
Determine that any adjustments to the raw data are handled appropriately, including but not limited to, trending, development, capping, removal of catastrophes.
Determine that individual input characteristics to a predictive model (and its sub-models) are not unfairly discriminatory and do not reflect proxies for prohibited characteristics.
Obtain a clear understanding of how the selected predictive model was built and why the insurer believes this type of model works in a private passenger automobile or homeowner’s insurance risk application.
Determine that individual output characteristics from a predictive model are related to expected loss or expense differences in risk. Each output characteristic should have a demonstrable actual relationship to expected loss or expense.
Obtain a clear understanding of how model output interacts with non-modeled characteristics/variables used to calculate a risk’s premium.
Determine that individual outputs from a predictive model and their associated selected relativities are not unfairly discriminatory or otherwise inappropriate.
Obtain a clear understanding of how the predictive model was integrated into the insurer’s state rating plan and how it improves the state ratingthat plan, (this latter element is only applicable when a new or revised model is introduced into an existing rating plan).
For predictive model refreshes, determine whether sufficient validation was performed to ensure the model is still a good fit.
Formatted: Font: (Default) Times New Roman, 10 pt,
Formatted: Normal, Left, No bullets or numbering
Formatted: Font: (Default) Times New Roman, 10 pt, Formatted: Font: (Default) Times New Roman, 10 pt,
Formatted: Font: Italic
Commented [WL2]: This should reflect model refreshes as well. Always applicable.
Formatted: Normal, Left, No bullets or numbering
Formatted: Font: (Default) Times New Roman, 10 pt, Formatted: Font: (Default) Times New Roman, 10 pt,
Formatted: Font: Italic
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 42
California DOI CommentsDraft: 10/25/18As adopted by the Casualty Actuarial and Statistical (C) Task Force on XX/XX/XX
© 2018 National Association of Insurance Commissioners 6
Determine the extent the model causes premium disruption for individual policyholders, and how the insurer will explain the disruption to individual consumers that inquire about it.
Determine the means available to a consumer to correct or contest individual data input values that may be in error.
Obtain a clear understanding how often each risk characteristics used as input to the model is updated and whether the model is periodically rerun to reflect changes to non-static characteristics.
Given an insurer’s rating plan relies on a predictive model and knowing all characteristics of a risk, a regulator should be able to audit/calculate the risk’s premium without consultation with the insurer.
VII.PREDICTIVE MODELS – INFORMATION FOR REGULATORY REVIEW
This section of the paper identifies the information a regulator may need to review a predictive model used by an insurer to support a filed P/C insurance rating plan. The list is lengthy but not exhaustive. It is not intended to limit the authority of a regulator to request additional information in support of the model or filed rating plan. Nor is every item on the list intended to be a required for every filing. However, the items listed should help guide a regulator to obtain sufficient information to determine if the rating plan meets state specific filing and legal requirements.
Though the list seems long, the insurer should already have internal documentation on the model for more than half of the information listed. The remaining items on the list require either minimal analysis (approximately 25%) or deeper analysis to generate the information for a regulator (approximately 25%).
Commented [LKW3]: This is more of a general requirement for any rating variable, and not specific to predictive model reviews.
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 43
California DOI CommentsDraft: 10/25/18As adopted by the Casualty Actuarial and Statistical (C) Task Force on XX/XX/XX
© 2018 National Association of Insurance Commissioners 7
A. Selecting Model Input
Information
Importance to Regulator’s
Review"Essential" or
"May Be Requested"
Comments
1. Available Data Sources
A.1.a
Provide details of all data sources including the experience period for insurance data and when the data was last recorded or updated.
Essential
This information can be used to evaluate the completeness of the data source, integrity of the data source, relevance of the data to the predictive timeframe, the potential for historical bias,transparency to insured of the data source, and the ability of the insured to make corrections to the data source.
A.1.a.i
Provide reconciliation between raw insurance data and external insurance reports.
EssentialAccuracy of insurance data should be reviewed as well.
A.1.a.ii
Provide the evaluation date of both insurance data, non-insurance or external data.
Essential
This information can be used to identify the staleness of data, especially external/non-insurance data, which may limit its relevance. The external/non-insurance data may not currently undergo the same scrutiny as insurance data and as the use of big data becomes more prevalent in insurance rating, this aspect of review will become more critical.
A.1.b Specify the companies whose data is included in the datasets.
May Be RequestedEssential
If the filer is part of a group, do the datasets include data from affiliated companies? If so, which companies? If the filer is an advisory organization, what companies are used? Are the companies included in the data relevant and compatible to the company that filed the rating plan?
A.1.cProvide the geographical scope and geographic exposure distribution of the data.
Essential
Evaluate whether the data is relevant to the loss potential for which it is being used. For example, verify that hurricane data is only used where hurricanes can occur.
A.1.d
List each data source. For each source, list all data elements used as input to the model that came from that source.
Essential
A.1.eSpecify the type of data (e.g., accident year or policy year, text, numeric).
Essential
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 44
California DOI CommentsDraft: 10/25/18As adopted by the Casualty Actuarial and Statistical (C) Task Force on XX/XX/XX
© 2018 National Association of Insurance Commissioners 8
A.1.f
Explain if internal or external data was used and if external data was used, disclose reliance on data supplied by others.
Essential
A.1.g
Provide details of any non-insurance data used (customer-provided or other), including who owns this data, how consumers can verify their data and correct errors, whether the data was collected by use of a questionnaire/checklist, whether it was voluntarily reported by the applicant, and whether any of the variables are subject to the Fair Credit Reporting Act. If the data is from an outside source, what steps were taken to verify the data was accurate?
EssentialIf the data is from a third-party source, the company should provide information on how the sourceaddresses the questions in this consideration.
2. Sub-Models
A.2.a
Disclose reliance on sub-modeloutput used as input to this model. If a sub-model was relied upon, provide the vendor name, and the name and version of the sub-model. If the sub-model was built/created in-house, provide contact information for the person responsible for the sub-model.
Essential
Examples of such sub-models include credit/financial scoring algorithms and household composite score models. Sub-models can be evaluated separately and in the same manner as the primary model under evaluation.
A.2.b
If using catastrophe model output, identify the vendor and the model settings/assumptions used when the model was run.
EssentialFor example, it is important to know hurricane model settings for storm surge, demand surge, long/short-term views.
A.2.c
If using catastrophe model output (a sub-model) as input to the GLMunder review, disclose whether loss associated with the modeled output was removed from the lossexperience datasets.
Essential
If a weather-based sub-model is input to the GLM under review, loss data used to develop the model should not include loss experience associated withthe weather-based sub-model. Doing so could cause distortions in the modeled results by double counting such losses when determining relativities or loss loads in the filed rating plan. For example, redundant losses in the data may occur when non-hurricane wind losses are included in the data whilealso using a severe convective storm model in the actuarial indication. Such redundancy may also occur with the inclusion of fluvial or pluvial floodlosses when using a flood model, inclusion of freeze losses when using a winter storm model orincluding demand surge caused by any catastrophic event.
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 45
California DOI CommentsDraft: 10/25/18As adopted by the Casualty Actuarial and Statistical (C) Task Force on XX/XX/XX
© 2018 National Association of Insurance Commissioners 9
A.2.d
If using output of any scoring algorithms, provide a list of the variables used to determine the score and provide the source of the data used to calculate the score.
EssentialAny sub-model should be reviewed in the same manner as the primary model that uses the sub-model’s output as input.
A.2.eWas the sub-model previously approved (or accepted) by the regulatory agency?
Essential If the sub-model was previously approved, that may change the extent of the sub-model’s review.
3. Adjustments and Scrubbing
A.3.a Provide pre-scrubbed data distributions for each input. May Be Requested Compare these distributions to A.3.g
A.3.b
Provide the percentage of exposures and premium for missing information from the model data by category.How was missing data handled?
Essential
A.3.c If duplicate records exist, how were they handled? Essential
A.3.d
Were any data outliers identified and subsequently adjusted? Name the outliers and explain the adjustments made to these outliers.
Essential
A.3.e
Were premium, exposure, loss or expense data adjusted (e.g., developed, trended, adjusted for catastrophe experience or capped) and, if so, how? Do the adjustments vary for different segments of the data and, if so, what are the segments and how was the data adjusted?
Essential
Look for anomalies in the data that should be addressed. For example, is there an extreme loss event in the data? If other processes were used to load rates for specific loss events, those losses should be removed from the input data, e.g., large losses, flood, hurricane or severe convective storm models for PPA comprehensive or homeowners’ loss.
A.3.f
What adjustments were made to raw data, e.g., transformations, binning and/or categorizations? If so, name the characteristic/variable and describe the adjustment.
Essential
A.3.g Provide post-scrubbed data distributions for each input. May Be Requested Compare these distributions to A.3.a
4. Data Organization
A.4.a
Document the method of organization for compiling data, including procedures to merge data from different sources and a description of any preliminary analyses, data checks, and logical tests performed on the data and the results of those tests.
Essential This should explain how data from separate sources was merged.
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 46
California DOI CommentsDraft: 10/25/18As adopted by the Casualty Actuarial and Statistical (C) Task Force on XX/XX/XX
© 2018 National Association of Insurance Commissioners 10
A.4.b
Document the process for reviewing the appropriateness, reasonableness, consistency and comprehensiveness of the data, including a justification of why the data makes sense.
Essential
For example, if by-peril modeling is performed, the documentation should be for each peril and make intuitive sense. For example, if “murder” or “theft” rates are used to predict the wind peril, provide support and a logical explanation.
A.4.c
Disclose material findings from the data review and identify any potential material limitations, defects, bias or unresolved concerns found or believed to exist in the data.
Essential
A.4.dFor any errors or material limitations in the data, explain how they were corrected.
Essential
5. Final Data Information
A.5.aIf the raw data selected to build the model is in a format that can be made available to the regulator, provide it.
May Be Requested
Commented [LKW4]: Please provide more explanation as to what is meant by these overarching terms.
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 47
California DOI CommentsDraft: 10/25/18As adopted by the Casualty Actuarial and Statistical (C) Task Force on XX/XX/XX
© 2018 National Association of Insurance Commissioners 11
B. Building the Model
Information
Importance to Regulator’s
Review"Essential" or
"May Be Requested"
Comments
1. High-Level Narrative for Building the Model
B.1.a
Identify the type of model (e.g. Generalized Linear Model – GLM, decision tree, Bayesian Generalized Linear Model, Gradient-Boosting Machine, neural network, etc.), describe its role in the rating systemand provide the reasons why that type of model is an appropriate choice for that role.
Essential If by-peril or by-coverage modeling is used, the explanation should be by-peril/coverage.
B.1.bA description of why the model (using the variables included in it) is appropriate for the line of business.
EssentialIf by-peril, by-form or by-coverage modeling is
used, the explanation should be by-peril/coverage/form.
B.1.c
Describe the model review process, from initial concept to final model. Keep this in overview narrative mode, less than 3 pages.
Essential
B.1.d
Describe whether loss ratio, pure premium or frequency/severity analyses was performed and, if separate frequency/severity modeling was performed, how pure premiums were determined.
Essential
B.1.e What is the model’s target variable? Essential A clear description of the target variable is key to understanding the purpose of the model.
B.1.f Provide a detailed description of the variable selection process. Essential
B.1.g
Was input data segmented in any way, e.g., was modeling performed on a by-coverage or by-peril basis or by-form? Explain the form of data segmentation and the reasons for data segmentation.
Essential The regulator would use this to follow the logic of the modeling process.
B.1.h
Describe any limitations or concerns in the analysis resulting from data issues and discuss the resulting impact on the modeling results.
Essential
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 48
California DOI CommentsDraft: 10/25/18As adopted by the Casualty Actuarial and Statistical (C) Task Force on XX/XX/XX
© 2018 National Association of Insurance Commissioners 12
B.1.i
How was data credibility (or lack thereof) was accounted for in the model building? How did the company determine the granularity of the rating variables?
Essential
Adjustments may be needed given models do not explicitly consider the credibility of the input data or the model’s resulting output; models take input data at face value and assume 100% credibility when producing modeled output.
2. Medium-Level Narrative for Building the Model
B.2.a
Describe any judgment used throughout the modeling process.Disclose assumptions used in constructing the model and provide support for these assumptions.
Essential
B.2.b
If post-model adjustments were made to the data and the model was rerun, explain the details and the rationale. It is not necessary to discuss each iteration of adding and subtracting variables, but the regulator should be provided with a general description of how that was done, including any measures relied upon.
Essential Evaluate the addition or removal of variables and the model fitting.
B.2.c
Describe the univariate testing and balancing that was performed during the model-building process, including a verbal summary of the thought processes involved.
Essential Further elaboration from B.2.b.
B.2.d
Describe the 2-way testing and balancing that was performed during the model-building process, including a verbal summary of the thought processes of including (or not including) interaction terms.
Essential Further elaboration from B.2.a and B.2.b.
B.2.e
For the GLM, what was the link function used? What distribution was used for the model (e.g., Poisson, Gaussian, log-normal, Tweedie)? Explain why the particular link function and distribution was were chosen. Provide the formulas for the distribution and link functions, including specific numerical parameters of the distribution.
Essential
B.2.f Were there data situations where GLM weights were used? Describe these. May Be Requested Investigate whether identical records were
combined to build the model.
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 49
California DOI CommentsDraft: 10/25/18As adopted by the Casualty Actuarial and Statistical (C) Task Force on XX/XX/XX
© 2018 National Association of Insurance Commissioners 13
3. Predictor Variables
B.3.a
Provide a complete data dictionary, including the names, descriptions and uses of each predictor variable, offset variable, control variable, proxy variable, geographic variable, geodemographic variable and all other variables in the model(including sub-models and external models); explanations should not use programming language or code.
Essential
B.3.a.i
Provide a list of predictor variables considered but not used in the final model, and the rationale for their removal.
May Be Requested
The rationale for this requirement is to identify variables that the company finds to be predictive but ultimately may reject for reasons other than loss-cost considerations (e.g., price optimization)
B.3.b
For each predictor variable within each model or sub-model, state whether the variable is continuous, discrete or Boolean.
Essential
B.3.b.iProvide a correlation matrix for all predictor variables included in the model and sub-model(s).
Essential
While GLMs accommodate collinearity, the correlation matrix provides more information about the magnitude of correlation between variables.
B.3.c
Provide an intuitive argument for why an increase in each predictor variable should increase or decrease frequency, severity, loss costs, expenses, or any element or characteristic whatever is being predicted.
Essential
B.3.d
If the modeler used a Principal Component Analysis (PCA) approach, provide a narrative about that process, explain why PCA was used, and describe the step-by-step process used to transform observations (usually correlated) into a set of linearly uncorrelated variables. Include a listing of the PCA variable and its principal components.
Essential
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 50
California DOI CommentsDraft: 10/25/18As adopted by the Casualty Actuarial and Statistical (C) Task Force on XX/XX/XX
© 2018 National Association of Insurance Commissioners 14
4. Massaging Data, Model Validation and Goodness-of-Fit Measures
B.4.a
Provide a description of how the available raw data was divided between model development(training), test, and validation datasets. Describe all circumstances under which the testing and validation datasets were accessed.
Essential
B.4.b
Describe the methods used to assess the statistical significance/goodness of the fit of the model, such as lift charts and statistical tests. Disclose whether the results are based on testing data, validation data and holdout samples. Ensure that the assessment includes model projection results compared to historical actual results to verify that modeled results bear a reasonable relationship to actual results. Discuss the results.
Essential
Some states require state-only data to test the plan, especially for analysis where using the state-only data contradicts the countrywide results. State-only data might be more applicable but could also be impacted by low credibility for some segments of risk.
B.4.c
Describe any adjustments that were made in the data with respect to scaling for discrete variables or binning the data.
Essential
B.4.d Describe any transformations made for continuous variables. Essential
B.4.e
For each discrete variable level, provide the parameter value, confidence intervals, chi-square tests, p-values and any other relevant and material tests. Were model development data, validation data, test data or other data used for these tests?
Essential
Typical p-values greater than 5% are large and should be questioned. Reasonable business judgment can sometimes provide legitimate support for high p-values. Reasonableness of the p-value threshold could also vary depending on the context of the model, e.g., the threshold might be lower when many candidate variables were evaluated for inclusion in the model.
B.4.f
Identify the threshold for statistical significance and explain why it was selected. Provide a verbal defense for keeping the variable for each discrete variable level where the p-values were not less than the chosen threshold.
Essential See Comment for B.4.e.
B.4.g
For overall discrete variables, provide type 3 chi-square tests, p-values, F tests and any other relevant and material test. Were model development data, validation data, test data or other data used for these tests?
Essential See Comment for B.4.e.
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 51
California DOI CommentsDraft: 10/25/18As adopted by the Casualty Actuarial and Statistical (C) Task Force on XX/XX/XX
© 2018 National Association of Insurance Commissioners 15
B.4.h
For continuous variables, provide confidence intervals, chi-square tests, p-values and any other relevant and material test. Were model development data, validation data, test data or other data used for these tests?
Essential See Comment for B.4.e.
B.4.i Describe how the model was tested for stability over time. Essential
Evaluate the build/test/validation datasets for potential model distortions (e.g., a winter storm in year 3 of 5 can distort the model in both thetesting and validation datasets).
B.4.jDescribe how the model was tested for geographic stability, e.g., across states or territories within state.
Essential Evaluate the geographic splits for potential model distortions.
B.4.kDescribe how overfitting was addressed and the results of correlation tests.
Essential
B.4.l
Provide support demonstrating that the GLM assumptions are appropriate (for example, the choice of error distribution).
Essential Visual review of plots of actual errors is usually sufficient.
B.4.m
Provide the formula relationship between the data and the model outputs, with a definition of each model input and output. Provide all necessary coefficients to evaluate the predicted value for any real or hypothetical set of inputs.
EssentialB.4.l and B.4.m will show the mathematical functions involved and could be used to reproduce some model predictions.
B.4.nProvide 5-10 sample records and the output of the model for those records.
Essential
5. “Old Model” Versus “New Model”
B.5.a
Provide aAn explanation of why this model is better than the one it is replacing. How was that conclusion formed? What metrics were relied on for measurement?
EssentialRegulators should expect to see improvement in the new class plan’s predictive ability or other sufficient reason for the change.
B.5.bWere 2 two Gini coefficients compared? What was the conclusion drawn from this comparison?
May Be Requested One example of a comparison might be sufficient.
B.5.cWere double lift charts analyzed? What was the conclusion drawn from this analysis?
Essential One example of a comparison might be sufficient.
B.5.dProvide a list of all new predictor variables in the model that were not in the prior model.
EssentialUseful to differentiate between old and new variables so the regulator can prioritize more time on factors not yet reviewed.
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 52
California DOI CommentsDraft: 10/25/18As adopted by the Casualty Actuarial and Statistical (C) Task Force on XX/XX/XX
© 2018 National Association of Insurance Commissioners 16
B.5.e
Provide a list of predictor variables used in the old model that are not used in the new model. Provide a detailed explanation of Why why were they were dropped from the new model?.
Essential
6. Modeler/Software
B.6.a
Provide the names, contact emails, phone numbers and qualifications of the key persons who:
a. Led the projectb. Compiled the datac. Built the modeld. Performed peer review
Essential
B.6.b
What software was used? Provide the name of the software vendervendor/developer, software product and a software version reference.
Essential
B.6.cWhen did work to build the model begin and when was the model build finalized?
Essential
Commented [WL5]: What is the rationale of this request? Suggest adding that rationale in the Comments section.
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 53
California DOI CommentsDraft: 10/25/18As adopted by the Casualty Actuarial and Statistical (C) Task Force on XX/XX/XX
© 2018 National Association of Insurance Commissioners 17
C. The Filed Rating Plan
Information
Importance to Regulator’s
Review"Essential" or
"May Be Requested"
Comments
1. General Impact of Model on Rating Algorithm
C.1.a
In the Actuarial Memorandum section on the SERFF Supporting Documentation tab, for each model and sub-model (including external models) relied upon, include a document that explains the model and its role in the rating system.
Essential
This item becomes “Essential” if the role of the model cannot be immediately discerned by the reviewer from a quick review of the rate and/or rule pages. (Importance is dependent on state requirements and ease of identification by the first layer of review and escalation to the appropriate review staff.)
C.1.bProvide an explanation of how the model was used to adjust the rating algorithm.
Essential
C.1.c
Provide a complete list of all characteristics/variables used in the proposed rating plan, including those used as input to the model (including sub-models and composite variables) and all other characteristics/variables used to calculate a premium. For each characteristic/variable, indicate if it is only input to the model, whether it is only a separate univariate rating characteristic, or whether it is both input to the model and a separate univariate rating characteristic. The list should provide transparent descriptions of each listed characteristic/variable.
Essential
Examples of variables used as inputs to the model and used as separate univariate rating characteristics might be criteria used to determine a rating tier or household composite characteristic.
C.1.d
For each characteristic/variable used as both input to the model (including sub-models and composite variables) and as a separate univariate rating characteristic, explain how these are tempered or adjusted to account for possible overlap or redundancy in what the characteristic/variable measures.
Essential
Modeling loss ratio with these characteristics/variables as control variables would account for possible overlap. The insurer should address this possibility or other considerations, e.g., tier placement models often use risk characteristics/variables that are also used elsewhere in the rating plan.
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 54
California DOI CommentsDraft: 10/25/18As adopted by the Casualty Actuarial and Statistical (C) Task Force on XX/XX/XX
© 2018 National Association of Insurance Commissioners 18
C.1.e
If the filing support includes an update or replacement of an existing model, identify and explain the changes in calculations, assumptions, parameters and data used to build the models. Provide an explanation of why the updated/replacement model is better than the one it is replacing, including, how that conclusion was reached, and the metrics relied upon to reach that conclusion.
Essential
2. Relevance of Variables / Relationship to Risk of Loss
C.2.a
Provide an explanation how the characteristics/rating variables,included in the filed rating plan,logically and intuitively relate to the risk of insurance loss (or expense) for the type of insurance product being priced. Include a discussion of the relevance each characteristic/rating variable has on consumer behavior that would lead to a difference in risk of loss (or expense).
EssentialThis explanation would not be needed if the connection between variables and risk of loss (or expense) has already been illustrated.
3. Comparison of Model Outputs to Current and Selected Rating Factors
C.3.a
Provide a comparison between relativities indicated by the model toboth current relativities and the insurer's selected relativities foreach risk characteristic/variable inthe rating plan. Each significant difference should be highlighted and explained.
Essential
“Significant difference” may vary based on the risk characteristic/variable and context. However, the movement of a selected relativity should be in the direction of the indicated relativity; if not, an explanation is necessary as to why the movement is logical.
C.3.b
What calculations, judgments and adjustments, if any, were made before using the model output in the rating system? Identify any adjustments that were made to the indicated indications produced by the model to derive the selected selections, modeland provide an explanation for the necessity of such adjustments.
Essential
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 55
California DOI CommentsDraft: 10/25/18As adopted by the Casualty Actuarial and Statistical (C) Task Force on XX/XX/XX
© 2018 National Association of Insurance Commissioners 19
C.3.c
If the model results in scores, tiers, or ranges of values for which indications are then derived for each such resulting category, provide explanations for filed rating values that deviate from these indicationsand supporting information/analyses. For example, identify any adjustments that were made to the factors indicated for each category of the model outputs to derive the factors selected for the rating plan.
EssentialThis is especially important if deviations are material and/or impact one consumer population more than another.
4. Responses to Data, Credibility and Granularity Issues
C.4.a What consideration was given to the credibility of the output data? Essential
At what level of granularity is credibility applied. If modeling was by-coverage, by-form or by-peril, explain how these were handled when there was not enough credible data by coverage, formor peril to model.
C.4.bIf applicable, discuss the rationale for using a model that is more granular than the rating plan.
EssentialThis is applicable if the insurer had to combine modeled output in order to reduce the granularity of the rating plan.
C.4.cIf applicable, discuss the rationale for using a rating plan that is more granular than modeled output.
Essential
A more granular rating plan implies that the insurer had to extrapolate certain rating treatments, especially at the tails of a distribution of attributes, in a manner not specified by the model indications.
5. Definitions of Rating Variables
C.5.a
Provide a transparent presentation and explanation of binning decisions that assign ranges of model outputs to particular rating categories.
Essential
C.5.b
Provide complete definitions of any rating tiers or other intermediate rating categories that translate the model outputs into some other structure that is then presented within the rate and/or rule pages.
Essential
Commented [WL6]: Confusing and difficult to read. Suggest rewrite.
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 56
California DOI CommentsDraft: 10/25/18As adopted by the Casualty Actuarial and Statistical (C) Task Force on XX/XX/XX
© 2018 National Association of Insurance Commissioners 20
6. Supporting Data
C.6.a
Provide state-specific, book-of-business-specific univariate historical experience data,separately for each year included in the model, consisting of, at minimum, earned exposures, earned premiums, incurred losses, loss ratios and loss ratio relativities for each category of model output(s) proposed to be used within the rating plan. For each data element, explain whether it is raw or adjusted, and if the latter, provide detailed explanation for the adjustments.
EssentialFor example, were losses developed/undeveloped, trended/untrended,capped/uncapped, etc.
C.6.b
Provide an explanation of any material (especially directional) differences between model indications and state-specific univariate indications.
Essential
Multivariate indications may be reasonable as refinements to univariate indications, but likely not for bringing about reversals of those indications. For instance, if the univariate indicated relativity for an attribute is 1.5 and the multivariate indicated relativity is 1.25, this is potentially a plausible application of the multivariate techniques. If, however, the univariate indicated relativity is 0.7 and the multivariate indicated relativity is 1.25, a regulator may question whether the attribute in question is negatively correlated with other determinants of risk. Credibility of state data should be considered when state indications differ from modeled results based on a broader data set. However, the relevance of the broader data set to the risks being priced should also be considered.
7. Consumer Impacts
C.7.aIdentify model changes and rating variables that will cause large premium disruptions.
Essential
C.7.b
Did the insurer perform sensitivity testing to identify significant changes in premium due to small or incremental change in a single risk characteristic? If so, discuss and provide the results of that testing.
May Be Requested
One way to see sensitivity is to analyze a graph of each risk characteristic’s/variable’s possible relativities. Look for significant variation between adjacent relativities and evaluate if such variation is reasonable and credible.
C.7.c
Measure and describe the impacts on expiring policies and describe the process used by management to mitigate or get become comfortable with those impacts.
Essential
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 57
California DOI CommentsDraft: 10/25/18As adopted by the Casualty Actuarial and Statistical (C) Task Force on XX/XX/XX
© 2018 National Association of Insurance Commissioners 21
C.7.d
Provide a rate disruption/dislocationanalysis, demonstrating the distribution of percentage impacts on renewal business (create by rerating the current book of business). Include the largest dollar and percentage impacts arising from the filing, including (desirably) the impacts arising specifically from the adoption of the model or changes to the model as they translate into the proposed rating plan.
Essential
While the default request would typically be for the distribution of impacts at the overall filing level, the regulator may need to delve into the more granular variable-specific effects of rate changes if there is concern about particular variables having extreme or disproportionate impacts, or significant impacts that have otherwise yet to be substantiated.See Appendix C for an example of a disruption analysis.
C.7.e
Provide exposure distributions for output variables and show the effects of rate changes at granular and summary levels.
Essential See Appendix C for an example of an exposure distribution.
C.7.fExplain how the insurer will help educate consumers to mitigate their risk.
Essential
C.7.g
Identify sources to be used at "point of sale" to place individual risks within the matrix of rating system classifications. How can a consumer verify their own "point-of-sale" data and correct any errors?
Essential
Could be "Essential" if the variables/ characteristics used could 1) have public-policy implications, 2) result in erroneous information being used, or 3) result in many large, disruptive premium changes at renewal. Another consideration to judge “importance” is whether consumers are proactively involved (e.g., use of consumer credit information and credit-report accuracy issues).
C.7.h
Identify rating variables that remain static over a consumer’s lifetime versus those that will be updated periodically. Document guidelines for variables that are listed as static,yet for which the underlying consumer attributes may change over time.
Essential
C.7.i
Provide the regulator with a description of how the company will respond to consumers’ inquiries about how their premium was calculated.
Essential
C.7.jProvide the regulator with a means to calculate the rate charged a consumer.
May Be RequestedEssential
Especially for a complex model or rating plan, a score or premium calculator via Excel or similar means would be ideal, but this could be elicited on a case-by-case basis. Ability to calculate the rate charged can allow the regulator to perform sensitivity testing when there are small changes to a risk characteristic/variable.
Commented [WL7]: Our concern is if this is not Essential, filers may be led to believe they can create “black boxes” creating rates that regulators would not be able to validate against filed rating plans.
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 58
California DOI CommentsDraft: 10/25/18As adopted by the Casualty Actuarial and Statistical (C) Task Force on XX/XX/XX
© 2018 National Association of Insurance Commissioners 22
8. Accurate Translation of Model into a Rating Plan
C.8.a
Provide sufficient information for the reviewer to be able to understand how the model outputs are used within the rating systemand to verify that the rating plan, in fact, reflects the model output and any adjustments made to the model output.
Essential
VIII. PROPOSED CHANGES TO THE PRODUCT FILING REVIEW HANDBOOK
TBD – placeholder to include best practices for review of predictive models and analytics filed by insurers to justify rates
IX. PROPOSED STATE GUIDANCE
TBD –placeholder for guidance for rate filings that are based on predictive model
X. OTHER CONSIDERATIONS
During the development of this paperguidance, a topics arose that are not addressed in this paper. These topics may need addressing during the regulator’s review of a predictive model. A few of these issues may be discussed elsewhere within NAIC. All of these issues, if addressed, will be addressed by each state on a case-by-case basis. The topics for considerationinclude:
TBD
XI. RECOMMENDATIONS GOING FORWARD
TBD
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 59
California DOI CommentsDraft: 10/25/18As adopted by the Casualty Actuarial and Statistical (C) Task Force on XX/XX/XX
© 2018 National Association of Insurance Commissioners 23
XII.APPENDIX A – BEST PRACTICE DEVELOPMENT
Best-practices development is a method for reviewing public policy processes that have been effective in addressing particular issues and could be applied to a current problem. This process relies on the assumptions that top performance is aresult of good practices and these practices may be adapted and emulated by others to improve results9.
The term “best practice” can be a misleading one due to the slippery nature of the word “best”. When proceeding with policy research of this kind, it may be more helpful to frame the project as a way of identifying practices or processes that have worked exceptionally well and the underlying reasons for their success. This allows for a mix-and-match approach for making recommendations that might encompass pieces of many good practices10.
Researchers have found that successful best-practice analysis projects share five common phases:
A. Scope
The focus of an effective analysis is narrow, precise and clearly articulated to stakeholders. A project with a broader focus becomes unwieldy and impractical. Furthermore, Bardach urges the importance of realistic expectations in order to avoid improperly attributing results to a best practice without taking into account internal validity problems.
B. Identify Top Performers
Identify outstanding performers in this area to partner with and learn from. In this phase, it is key to recall that a best practice is a tangible behavior or process designed to solve a problem or achieve a goal (i.e. reviewing predictive models contributes to insurance rates that are not unfairly discriminatory). Therefore, top performers are those who are particularly effective at solving a specific problem or regularly achieve desired results in the area of focus.
C. Analyze Best Practices
Once successful practices are identified, analysts will begin to observe, gather information and identify the distinctive elements that contribute to their superior performance. Bardach suggests it is important at this stage to distill the successful elements of the process down to their most essential idea. This allows for flexibility once the practice is adapted for a new organization or location.
D. Adapt
Analyze and adapt the core elements of the practice for application in a new environment. This may require changing some aspects to account for organizational or environmental differences while retaining the foundational concept or idea. This is also the time to identify potential vulnerabilities of the new practice and build in safeguards to minimize risk.
E. Implementation and evaluation
The final step is to implement the new process and carefully monitor the results. It may be necessary to make adjustments, so it is likely prudent to allow time and resources for this. Once implementation is complete, continued evaluation is important to ensure the practice remains effective.
9 Ammons, D. N. and Roenigk, D. J. 2014. Benchmarking and Interorganizational Learning in Local Government. Journal of Public Administration Research and Theory, Volume 25, Issue 1. P 309-335. https://doi.org/10.1093/jopart/muu01410 Bardach, E. and Patashnik, E. 2016. A Practical Guide for Policy Analysis: The Eightfold Path to More Effective Problem Solving. Thousand Oaks, CA. CQ Press.
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 60
California DOI CommentsDraft: 10/25/18As adopted by the Casualty Actuarial and Statistical (C) Task Force on XX/XX/XX
© 2018 National Association of Insurance Commissioners 24
XIII. APPENDIX B - - GLOSSARY OF TERMS
Offset vs. control factors - TBD
Probability Distribution - TBD
Exponential Family - TBD
Linear Predictor - TBD
Link Function - TBD
Univariate Model - TBD
Generalized Linear Model - TBD
Private Passenger Automobile Insurance – TBD
Homeowners Insurance – TBD
Rating algorithm – TBD
Rating plan – TBD
Rating system – TBD
PCA approach (Principal Component Analysis) – The PCA approach is also known as factor analysis. These methods create multiple new variables from correlated groups of predictors. Those new variables exhibit little or no correlation between them—thereby making them potentially more useful in a GLM. A PCA in a filing can be described as “a GLM within a GLM.” One of the more common applications of PCA is geodemographic analysis, where many attributes are used to modify territorial differentials on, for example, a census block level.
Fair Credit Reporting Act – The Fair Credit Reporting Act (FCRA), 15 U.S.C. § 1681 (FCRA) is U.S. Federal Government legislation enacted to promote the accuracy, fairness and privacy of consumer information contained in the files of consumer reporting agencies. It was intended to protect consumers from the willful and/or negligent inclusion of inaccurate informationin their credit reports. To that end, the FCRA regulates the collection, dissemination and use of consumer information, including consumer credit information.11 Together with the Fair Debt Collection Practices Act (FDCPA), the FCRA forms the foundation of consumer rights law in the United States. It was originally passed in 1970 and is enforced by the US Federal Trade Commission, the Consumer Financial Protection Bureau and private litigants.
Overfitting - TBD
Geodemographic - Geodemographic segmentation (or analysis) is a multivariate statistical classification technique for discovering whether the individuals of a population fall into different groups by making quantitative comparisons of multiplecharacteristics with the assumption that the differences within any group should be less than the differences between groups. Geodemographic segmentation is based on two principles:
1. People who live in the same neighborhood are more likely to have similar characteristics than are two people chosen at random.
2. Neighborhoods can be categorized in terms of the characteristics of the population that they contain. Any two neighborhoods can be placed in the same category, i.e., they contain similar types of people, even though they are widely separated.
Etc.
11 Dlabay, Les R.; Burrow, James L.; Brad, Brad (2009). Intro to Business. Mason, Ohio: South-Western Cengage Learning. p. 471. ISBN 978-0-538-44561-
0.
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 61
California DOI CommentsDraft: 10/25/18As adopted by the Casualty Actuarial and Statistical (C) Task Force on XX/XX/XX
© 2018 National Association of Insurance Commissioners 25
XIV. APPENDIX C – SAMPLE RATE-DISRUPTION TEMPLATE
NOTE: Uncapped Capped (If Applicable)
Minimum % Change Minimum % ChangeMaxmium % Change Maxmium % ChangeTotal Number of Insureds (Auto-Calculated)
1994Total Number of Insureds (Auto-Calculated)
1994
Uncapped Rate DisruptionPercent-Change Range Number of Insureds in Range Percent-Change Range Number of Insureds in Range
19
Capped Rate Disruption (If Applicable)
Template Updated October 2018State Division of Insurance - EXAMPLE for Rate Disruption
19
EXAMPLE Uncapped Rate Disruption
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 62
California DOI CommentsDraft: 10/25/18As adopted by the Casualty Actuarial and Statistical (C) Task Force on XX/XX/XX
© 2018 National Association of Insurance Commissioners 26
EXAMPLE Capped Rate Disruption
State Division of Insurance - EXAMPLE for Largest Percentage Increase Template Updated October 2018
Uncapped Change Uncapped Dollar Change Current Premium
Capped Change (If Applicable) Capped $ Change (If Applicable) Proposed Premium
Characteristics of Policy (Fill in Below)
Vehicle: BI Limits: PD Limits: UM/UIM Limits: MED Limits:
Attribute% Impact
(Uncapped)Dollar Impact (Uncapped)
TOTAL 15.00% $82.50
Corresponding Dollar Increase (for Insured Receiving Largest Percentage Increase)
For Auto Insurance:
At minimum, identi fy age and gender of each named insured, amount of insurance, terri tory, construction type, protection class , any prior loss his tory, and any other key attributes whose treatments are affected by this fi l ing.
Most Significant Impacts to This Policy
Largest Percentage Increase
Automobile policy: Territory:
COMP Deductible: COLL Deductible:
NOTE: as needed. Tota l percent and dol lar impacts should reconci le to the va lues presented above in this exhibi t.
What lengths of pol icy terms does the insurer offer in this book of bus iness?
Check a l l options that apply below.
12-Month Policies121212--Month PoliciesPPMonth PoliciesMonth Policies
6-Month Policies3-Month Policies
Other (SPECIFY)
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 63
California DOI CommentsDraft: 10/25/18As adopted by the Casualty Actuarial and Statistical (C) Task Force on XX/XX/XX
© 2018 National Association of Insurance Commissioners 27
State Division of Insurance - EXAMPLE for Largest Dollar Increase Template Updated October 2018
Uncapped Change Current Premium Uncapped Percent Change
Capped Change (If Applicable) Proposed Premium Capped % Change (If Applicable)Characteristics of Policy (Fill in Below)
Vehicle: BI Limits: PD Limits: UM/UIM Limits: MED Limits:
Attribute% Impact
(Uncapped)Dollar Impact (Uncapped)
for PD
TOTAL 12.00% $306.60
For Auto Insurance:
At minimum, identi fy age and gender of each named insured, amount of insurance, terri tory, construction type, protection class , any prior loss his tory, and any other key attributes whose treatments are affected by this fi l ing.
Corresponding Percentage Increase (for Insured Receiving Largest Dollar Increase)Largest Dollar Increase
Most Significant Impacts to This Policy
NOTE: as needed. Tota l percent and dol lar impacts should reconci le to the va lues presented above in this exhibi t.
Automobile policy: Territory:
COMP Deductible: COLL Deductible:
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 64
California DOI CommentsDraft: 10/25/18As adopted by the Casualty Actuarial and Statistical (C) Task Force on XX/XX/XX
© 2018 National Association of Insurance Commissioners 28
XV.APPENDIX D – INFORMATION NEEDED BY REGULATOR MAPPED INTO BEST PRACTICES
XVI. APPENDIX E – REFERENCES
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 65
To: NAIC Casualty Actuarial and Statistical Task Force From: Casualty Actuarial Society Machine Learning Task Force Members Date: January 15, 2019 Subject: Concerns Related to Regulatory Review of Predictive Models White Paper This purpose of this document is to raise concerns voiced by members of the Casualty Actuarial Society’s (CAS) Machine Learning Task Force (MLTF) with respect to the 10/25/2018 draft Regulatory Review of Predictive Models white paper of the NAIC’s Casualty Actuarial and Statistical Task Force. It should be noted that the MLTF does not speak on behalf of the CAS or its leadership, and within the MLTF there have been a variety of responses to the draft. This document collects some of those perspectives, which may at times be in conflict, with the intent of fully expressing our group’s concerns and considerations with respect to statistical modeling practice. Our primary concerns can be categorized as follows:
1. Scope (GLMs vs. machine learning) 2. Confidentiality of models 3. Focal point of regulation (rates vs. models) and depth of clarifying questions
Scope of White Paper The SCOPE paragraph the white paper may be excessively broad. While it first states that the scope will “focus on GLMs used to create private passenger automobile and homeowners’ insurance rating plans,” it then expands the scope universally with the statement that “Guidance and knowledge needed to review predictive models identified in this paper are, in large part, transferrable to other types of predictive models, to other lines of business, or other insurance endeavors.” The intended application to other types of models is confirmed in the High-Level Narrative for Building the Model, as item B.1.a asks to “Identify the type of model (e.g. Generalized Linear Model - GLM, decision tree, Bayesian Generalized Linear Model, Gradient-Boosting Machine, neural network, etc.)” Given that the paper addresses model review as it relates specifically to GLMs and private passenger auto with the hope that the guidance should be generalizable, it is somewhat concerning that more language in the paper was not devoted to some of the potentially significant ways that other types of models and other lines of business can deviate from the conventions of “GLM / PPA.” In some instances, these deviations can be such that questions marked as “essential” in this document may be extremely onerous, extremely difficult if not impossible to answer, misleading, or lacking in usefulness in other lines and for other types of
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 66
models. For this reason, we would suggest adding additional cautionary language (or revise the SCOPE section) to avoid potential misunderstandings. As an example of these issues, consider private passenger automobile coverage. Coverage is widely available and actuaries are accessing increasingly personal data to develop more refined GLM rating plans. The regulatory issues of personal privacy, fair credit reporting, and discrimination rise to the surface. Contrast this with the homeowners’ line, which faces a climate-driven coverage crisis. Measuring contagion risk has become as important as measuring individual risk. New data sources include vegetative fuel loads, topography, and weather data. Homeowners actuaries have to deal with extremely high severities and poorly fitting existing models. Innovation will be needed to maintain a functioning voluntary market. For these reasons, we recommend that the Task Force should limit the scope of the recommendations, provide up-front caveats regarding the use of the term “essential” to describe various data request items, and provide some additional discussion regarding the ways that rate filing review may differ depending on the model and line of business.
Confidentiality of Models The creation of innovative pricing models requires a major investment on the part of insurance companies. They make these investments in the hope of achieving a sustainable competitive advantage and consider these models as intellectual property. Public access to the inner workings of proprietary pricing models and company data acts as a serious deterrent to the investment in new models. The Task Force white paper acknowledges the importance of confidentiality, but offers no protections beyond what currently exists in state laws. It puts the burden on the insurers “to be familiar with each state’s laws regarding the confidentiality of information submitted with their rate filings.” We are concerned with the fact that the white paper offers no new protections to balance against the additional information requirements that will result from the proposed best practices. We would recommend that the Task Force add a column to the best practices table to indicate which items are likely to require confidentiality.
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 67
Focus of Regulation and Depth of Questions Members of our group expressed concern about the focus of the white paper in a number of complementary ways, in part based on their own experiences with the rate filing process. Our members have been involved in preparing rate filings and responding to questions from regulatory officials, providing reviews of models on behalf of insurers, and participating directly in the rate filing review process on behalf of regulators. Several members expressed concern that the best practices recommended in the white paper had the implied intent of allowing the regulator to build the model, themselves, rather than being designed for the purpose of reviewing the models. For these members, the statistical and data reporting required for a model review should be much less onerous than the requirements for creating a model from scratch. To that end, they voiced concern as follows:
“State insurance regulators are tasked with enforcing laws governing insurance rates, not the models that company management may have consulted during the pricing process. Internal decisions about credibility, binning, granularity, and limitation of rate disruption have long been part of the pricing process, even during the historical period of univariate pricing. The internal company pricing process has not typically been required in routine rate filings, but is typically requested only on an exception basis in the case of large rate changes. Please do not create best practices where the internal decision making process becomes “essential” in each filing. While regulators may feel that they need to understand the underlying models in order to understand the rating plans, it is possible to consider a change rating plan without an understanding of the technical elements of the underlying model. We submit that a rate change or rating plan could be judged to be acceptable if it is more efficient than the current rates, non-discriminatory, limits disruption, and is certified by an actuary. This approach puts the focus of regulation of the rating plan rather than the model.”
Others of our group took the view that often, significant technical detail is required for a thorough vetting of a model, especially in cases where particular variables, modeling decisions, or techniques used by the modeler are new or unusual in that line of business. In such cases the questions suggested by the white paper may, in fact, be inadequate to fully address the question of whether a decision produces rates that are not unfairly discriminatory. However, in the majority of models, such depth of questioning is not warranted. Questions of significant depth are only requested if a high-level review indicates that such questions are required. So, on the one hand, it is important that filers should be aware that such difficult questions are possibilities, particularly if there are unusual modeling decisions, because this
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 68
knowledge may allow them to prepare filings with forethought that makes answering such questions much less onerous. On the other hand, we are concerned that it is not necessary to ask such onerous questions in every instance. More to the point, there is some concern listing the majority of questions as “essential” and asking many questions for every single filing may have the unwanted effect of replacing careful statistical consideration with a “data-driven approach” to regulation. The risk of such an approach is that models may pass standard metrics of performance while failing badly at less-standard-but-more-highly-relevant measures of e.g. predictiveness. Such models would not be easily identified by a data-driven approach, and for this reason a data-driven approach to regulation is no substitute for careful analysis. We therefore recommend that significant consideration be given to whether each of the questions is “essential” if it is marked as such. Perhaps additional categories, such as “essential in certain circumstances” could provide a more nuanced view, allowing sufficient leeway for standard modeling approaches that are within commonly accepted bounds while providing adequate guidance regarding those times when it is essential to ask additional questions. In addition, given the complexity of many machine learning techniques relative to GLMs, many of the recommended questions may be impractical or impossible to answer. Further consideration must be given to the extent to which models, themselves, can be effectively regulated in such cases, or whether regulatory focus should (within reason) be on the resultant rates rather than the models that produced them.
Summary of Recommended Changes 1. Limit the scope of the recommendations provided in the white paper, 2. Provide up-front caveats regarding the use of the term “essential” to describe various
data request items, 3. Provide some additional discussion regarding the ways that rate filing review may differ
depending on the model and line of business, 4. Add a column to the best practices table to indicate which items are likely to require
confidentiality, 5. Significant consideration should be given to whether each of the questions is “essential”
if it is marked as such. Additional categories, such as “essential in certain circumstances” could provide a more nuanced view, allowing sufficient leeway for standard modeling approaches that are within commonly accepted bounds while providing adequate guidance regarding those times when it is essential to ask additional questions, and
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 69
6. Further consideration must be given to the extent to which models, themselves, can be effectively regulated in such cases, or whether regulatory focus should (within reason) be on the resultant rates rather than the models that produced them.
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 70
1
TO: NAIC Casualty Actuarial & Statistical (C) Task Force FROM: The Cincinnati Insurance Companies SUBJECT: Comments on Regulatory Review of Predictive Models White Paper DATE: January 22, 2019
Thank you for this opportunity to comment on the Casualty Actuarial & Statistical (C) Task Force (CASTF) white paper on best practices for the regulatory review of predictive models, which the CASTF voted to expose for public comment during its November 15, 2018 meeting held in conjunction with the NAIC Fall Meeting in San Francisco. INTRODUCTION. The use of predictive analytics has the potential to create significant benefits for both consumers and insurers by dramatically improving pricing precision, thereby promoting a more refined and balanced insurance transaction and transforming the insurer-consumer experience into a more meaningful relationship. Predictive analytics can reveal insights into consumer behavior, lower the cost of insurance for many, and provide tools for the consumer to better control and mitigate loss. Given these benefits, it is important that regulators have the necessary tools to effectively review an insurer’s use of predictive models in insurance applications.
With the forgoing precepts in mind, the CASTF was charged with identifying best practices to serve as a guide to state insurance departments in their review of predictive models underlying rating plans. Specifically, the CASTF was given two charges by the NAIC Property and Casualty Insurance (C) Committee at the request of the NAIC Big Data (EX) Working Group:
1) Draft and propose changes to the Product Filing Review Handbook to include best practices for review of predictive models and analytics filed by insurers to justify rates.
2) Draft and propose state guidance (e.g., information, data) for rate filings that are based on complex predictive models.
The initial step in this process has been completed: the drafting of the CASTF white paper to identify best practices for reviewing predictive models and analytics filed by insurers with regulators to justify rates and provide state guidance for review of rate filings based on predictive models.1 We now offer our comments on the CASTF draft white paper.
1 Upon adoption of the CASTF white paper by the NAIC Executive (EX) Committee and Plenary, the CASTF will evaluate how to incorporate the best practices identified in the white paper into the Product Filing Review Handbook and will recommend such changes to the NAIC Speed to Market (EX) Working Group.
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 71
2
Please note at the outset that we will be suggesting that the CASTF abandon its rules-based approach in favor of a principles-based approach for identifying best practices for regulatory review of predictive models. THE WHITE PAPER IS RULES-BASED AND OVERLY PRESCRIPTIVE. The heart of the 27-page white paper is a fifteen-page section of rules for the regulatory review of predictive models. As explained in the white paper, this section of the document identifies the information a regulator may need to review a predictive model used by an insurer to support a filed P/C insurance rating plan. As the drafters of the white paper acknowledge, the list of 94 prescriptive rules is “lengthy but not exhaustive” and may “seem long.”2 These conclusions inform our own assessment of the white paper: it is far too prescriptive and complicated and would further seem inconsistent with the goal of providing clear, meaningful guidelines for both modeler and regulator. The document is also at odds with what the regulated community would expect from a set of “best practices” – a set of consumer-oriented core guiding principles designed to ensure that an insurer’s predictive models are fairly constructed, not unfairly discriminatory and applied in ways that will sharpen an insurer’s ability to adequate segment and price risks, thus creating significant benefits for both consumers and insurers. Here are just a few of the concerns our actuaries have identified from their review of the CASTF’s rules-based white paper:
1. Many of the 94 rules included in the white paper have overlapping goals. These rules should be collapsed into a set of core principles instead of a series of detailed and prescriptive rules. For example, there are a variety of rules that discuss methods for determining whether a given variable should be included in a model. A better approach would be the pronouncement of a general principle for variable inclusion.
2. Some of the rules deemed “essential” are arguably too specific to be applicable in many cases or go
beyond the scope of necessary details to understand the model (e.g. PCA breakdown discussion, GLM weights, verbose description of all interaction testing).
3. The document seems focused on generalized linear models (GLMs) when methods of performing analysis are constantly evolving. For example, would a regulator actually want all of the “if” statements that would be generated by a random forest model? In addition, many newer models do not produce diagnostics such as p-values. Elastic net models, which are similar to traditional GLMs, have p-values that can be calculated, but these p-values do not represent statistical confidence as they do for a GLM.
4. Producing documentation for all the rules deemed "essential" would result in a very large volume of documentation. Would most regulators be able to properly review the generated model documentation? Proper review of this material would require regulators to allocate full-time staff to this task.
2 The 94 rules are described in a multi-paged table and are organized under 8 subject headings. Each of the rules are numbered using a three-character alphanumeric sequence: upper case letter, number, lower case letter (example: A.1.a.).
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 72
3
5. The 94 rules in the white paper would create an undue burden on modeling teams. We estimate this would increase our model development workload by roughly 33%. That is, an additional modeler would be required for every three current modelers.
6. The white paper’s rules-based approach, which imposes 94 prescriptive modeling rules on insurers, would reduce competition in the marketplace by increasing the amount of effort required to build and file models. It would also increase reliance on consultants and discourage insurers from developing internal models. It would also result in increased expense ratios and reduced innovation.
7. Many of the rules seem to suggest that an insurer should document and explain paths not taken. We think that the focus of regulatory review should be on the proposed model, as opposed to what is not being proposed.
For a more detailed list of our concerns with the 94 prescriptive rules, please see Exhibit A, submitted with this comment letter. We also share the concerns raised by our national P&C trade association, the American Property Casualty Insurance Association (APCI), in its comment letter detailing the many flaws in the 94 prescriptive rules. THE CASTF SHOULD ADOPT A PRINCIPLES-BASED APPROACH FOR REVIEW OF PREDICTIVE MODELS. We believe that a principles-based approach, premised upon a set of core guiding principles, would provide regulators with standards against which they could effectively review an insurer’s use of predictive models in insurance applications and in a timely fashion. We also believe that the regulatory review of predictive models should be more focused on preventing harm to consumers than on explaining how a predictive model functions. The principles-based approach we envision would achieve this balance by taking into consideration core guiding principles which would ensure that modelers follow a set of markers that would guard against unfair discrimination. Here are some examples of the types of core guiding principles which we believe are more focused on consumers and are worthy of the CASTF’s consideration:
1. Ensure that a predictive model does not promote, encourage or permit unfair/improper discrimination.
2. Ensure that a predictive model does not promote, encourage or permit improper strategy (price optimization, for example).
3. Ensure that the covariates and model predictions included in a model bear a reasonable resemblance to the subject matter being modeled.
4. Require adoption of internal company controls designed to periodically review all predictive models for violations of the core principles described above.
We therefore propose that the CASTF issue a call to regulators, the industry and other interested parties to submit proposed core principles for regulatory review of predictive models to the CASTF in advance of the NAIC Spring National Meeting in Orlando (April 6-9, 2019), and that the CASTF hold a discussion or hearing at the Orlando meeting on the question of whether it would be more effective and appropriate to employ a principles-based approach for regulatory review of predictive model than the current rules-based approach now under consideration.
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 73
4
THE NAIC HAS ALREADY EMBRACED PRINCIPLES-BASED REGULATION. The NAIC is no stranger to principles-based insurance regulation, which it embraced in 2017 to modernize the regulation of life insurance reserve calculations. As life insurance products increased in their complexity and companies developed new and innovative life insurance product designs that changed their risk profile, it was necessary to develop new reserve valuation methodologies to address these changes. These needs led the NAIC to replace its rules-based reserve calculation model with a principles-based valuation model. As the NAIC’s Center for Insurance Policy and Research explains in its “Key Issue” paper on principle-based reserving:
“Prior to PBR, static formulas and assumptions were used to determine these reserves as prescribed by state laws and regulations. However, sometimes this rule-based approach leaves an insurer with excessive reserves for certain insurance products and inadequate reserves for others. The solution is to "right-size" reserve calculations by replacing a rule-based approach with a principle-based approach.”3
Likewise, we believe that a principles-based approach would work better for regulatory review of predictive models. Using a set of core guiding principles instead of a prescriptive set of rules would “right size” predictive model regulation and avoid excessive regulation of new and innovative products. CLOSING COMMENTS. Thank you for considering our comments. We encourage the CASTF to consider the benefits of taking a principles-based approach to predictive model regulation and to take the measures we suggested above to take up the issue at the NAIC Spring Meeting in April 2019. Respectfully submitted,
Teresa Cracas Senior Vice President Chief Risk Officer
Luyang Fu Vice President Pricing & Predictive Analytics
Xiangfei Zeng Vice President Personal Pricing & Modeling
Thomas Hogan Vice President Corporate Counsel
Scott Gilliam Vice President Government Relations
Please direct all inquiries and follow up to Scott Gilliam, Vice President—Government Relations
Office: 513-870-2811 | Mobile: 513-607-5717 | Email: [email protected]
3 See “KEY ISSUE: The National System of State Regulation and Principle-Based Reserving, available online at https://naic.org/cipr_topics/principle_based_reserving_pbr.htm.
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 74
EXHIBIT A
NAIC Casualty Actuarial & Statistical (C) Task Force—White Paper on Regulatory Review of Predictive Models
Section VII. PREDICTIVE MODELS – INFORMATION FOR REGULATORY REVIEW
***Additional Concerns of The Cincinnati Insurance Companies***
General Comment. Requiring a company to state the data type of covariates should only be required when it affects model output and is necessary for model review. Often, the key information conveyed by data types are obvious from model structure/output. In other cases, the data type isn't strictly required for evaluation of the model. For example, many tree-based models treat numeric columns as an ordered categorical in the sense that the tree still chooses discrete split points for the numeric column. Rule B.2.e. The rule states that the formula for the distribution and link functions should be provided. We do not think providing statistical formulas necessarily aids in the review process. For commonly used distributions/models a reference should be sufficient.
Rule A.3.d. What purpose would be served by listing outliers? A better alternative would be to just describe the process for handling outliers. For example, loss capping is commonly applied. It would be unrealistic to list every record where a loss was capped. Rule A.4.a. Describing the process of how to join data seems unnecessary. Rule B.1.i. This rule provides: "models take input data at face value and assume 100% credibility." This is not true for all models. GLMs do treat categorical covariates as fully credible, but models such as elastic net and random forest do not. In addition, many implemented credibility procedures do not have rigorous foundations for the standard and credibility standards are often judgmental selected. Furthermore, selected complements often do not fall into a classical credibility framework. Rule B.3.c. Intuitive arguments should not be required. The question is whether the variables are predictive and not unfairly discriminatory. Rule B.3.d. Why is a narrative required to explain the reason PCA was used? PCA is a common technique that is well known for producing uncorrelated versions of the covariates. This property is even stated in the document. Furthermore, PCA is only one of many data transformations and we do not think it requires special treatment. Rule B.4.b. The rule provides: "Ensure that the assessment includes model projection results compared to historical actual results to verify that the modeled results bear a reasonable relationship to actual results." Shouldn't a model built on historical data bear a strong resemblance to the data? Or does this mean historical results of a prior model? Ideally, a new model would show somewhat different behavior as the new model would hopefully perform better as compared to the historical model. Rules B.4.e, B.4.g, and B.4.h. Many models such as elastic nets, neural networks, and tree-based models do not produce p-values or similar statistics. Rule B.4.m. Providing all formulas/coefficients is unrealistic for some models, e.g., random forest and deep neural networks. Technically all formulas/coefficients are able to be provided but would not be comprehensible or meaningful for model review.
EXHIBIT A
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 75
January 14, 2019
NAIC Casualty Actuarial and Statistical Task Force Attn: Kris DeFrain, FCAS, MAAA, CPCU
Via Email – [email protected]
Re: Comments on the Draft White Paper – Best Practices – Regulatory Review of Predictive Models
Dear Ms. DeFrain:
Fair Isaac Corporation (FICO®) is pleased to provide our comments on the recently released draft white paper, Best Practices – Regulatory Review of Predictive Models.
FICO is an independent analytics provider, not a data company, that is dependent on other firms (e.g., credit reporting agencies, insurance companies, lenders) to provide the appropriate and necessary data for our analysis and predictive model development. With a focus on innovation that effectively rewards all parties – insurers, lenders, and consumers alike – FICO is recognized as the pioneer in developing the algorithms and underlying analytics used to produce credit scores, credit-based insurance scores, and other risk management scores. FICO fully understands the value of regulatory scrutiny and the need for regulatory flexibility to help ensure that consumers continue to benefit from these scores by enjoying quick, fair access to credit and to more affordable insurance. In previous years, access to affordable insurance involved a lengthy decision process based, in some cases, on subjective and inconsistent underwriting factors.
In 1993, FICO introduced the first commercially-available credit-based insurance scores to all US insurers as an additional risk segmentation factor that could be used in their private passenger auto and home insurance underwriting and pricing programs. On behalf of several hundred FICO® Insurance Score clients, over these past 25 years we’ve met with state departments of insurance and have testified before dozens of state legislative committees. Our goal in each of these interactions was to provide regulatory support for our clients’ use of FICO Insurance Scores by answering all questions to the best of our ability and by offering as much insight into FICO’s proprietary modeling analytics and technologies as possible.
For nearly two decades, in support of rate filings throughout the nation by our FICO® Insurance Score clients, FICO has provided model documentation—specific consumer credit characteristics, attributes and weights for the filed model—as well as reason code/factor definitions, and a general discussion of our model development process to all requesting departments of insurance able to provide the necessary protections. In addition, FICO has modified our insurance score models as required by those states with either statutory or regulatory mandates. FICO also offers an insurance score educational website (insurancescores.fico.com) that has been accessed by consumers, regulators, legislators, insurers, agents and other interested parties throughout the nation.
It is FICO’s hope that our work over these past two decades with state departments of insurance on behalf of our clients will be fully recognized and allowed to continue unabated – grandfathered under the
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 76
current regulatory review process, as it were, and not impacted by undue burdens on FICO and our hundreds of clients.
Kris DeFrain, FCAS, MAAA, CPCU January 14, 2019 Page 2
Having shared a bit of FICO’s background and our FICO® Insurance Score client support strategies, the remainder of our comments will focus on our Scores business model and the negative implications the recommendations within the draft white paper could have on our Scores business.
The intellectual property underlying much of our predictive modeling and analytics technology has been developed by FICO data scientists over the past six decades. This development work has taken an enormous amount of time, money, research, know-how, and testing. The algorithms used in the various models cut across multiple products and solutions. For example, much of the technology and intellectual property that is used as part of the models and algorithms for FICO’s credit risk scores is also used as part of our other scoring models, including our credit-based insurance scores – FICO® Insurance Scores.
While some of FICO’s intellectual property relating to its scoring models and algorithms is protected by patents, the specifications that describe how the scoring models work and other core parts of the algorithms are FICO’s carefully guarded trade secrets. The continued success of FICO’s Scores business depends on the maintenance of this confidential and proprietary information and these trade secrets—including all non-public aspects of our current and future insurance scoring models.
FICO’s scoring-related trade secrets have substantial independent economic value to the company precisely because they are not generally known by others, including any potential competitors, that could unfairly obtain economic value from their disclosure or use. The value in FICO’s scoring trade secrets and proprietary methods for scoring would be put at risk if the company were required to disclose that information without full and continuing confidentiality. The unique scoring formula intellectual property assets, which are securely guarded and protected by contract and law, are paramount to an independent analytics provider such as FICO. Forcing disclosure of these intellectual property assets would dissipate the value of these assets.
Given the necessary protection of FICO’s intellectual property and trade secrets, our belief is that the depth and breadth of the regulatory review of predictive models proposed by the draft white paper presents serious market-restriction issues for FICO…..and for the hundreds of FICO® Insurance Score clients doing business in all states that allow for the industry’s significant use of credit-based insurance scores within their well-considered and comprehensive rating programs.
As mentioned previously, we believe the state regulatory practices under which FICO has supported our clients for the past two decades are appropriate and quite sufficiently protect all interests – consumers, regulators, and insurers.
The draft white paper’s only references to protection for the intellectual property and trade secrets of an independent analytics provider like FICO are too vague to offer any real protection. The proposal, as highlighted here, leaves the decision about confidentiality of a company’s intellectual property and trade secrets entirely within the discretion of each state regulator.
1. The fourth Key Regulatory Principle: State insurance regulators will maintain confidentiality, where appropriate, regarding predictive models.
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 77
Kris DeFrain, FCAS, MAAA, CPCU January 14, 2019 Page 3
2. Section V. CONFIDENTIALITY warns rate filers:
Insurers and regulators should be aware that a rate filing might become part of the public record. Each state determines the confidentiality of a rate filing, supplemental material to the filing, when filing information might become public, the procedure to request that filing information be held confidentially, and the procedure by which a public records request is made. It is incumbent on an insurer to be familiar with each state’s laws regarding the confidentiality of information submitted with their rate filing.
Nearly a decade ago, FICO was forced to withdraw its credit-based insurance scoring models from new or amended filings in the State of Florida over a very similar issue – lack of appropriate confidentiality protections. Because the Florida Office of Insurance Regulation rules forced FICO to withdraw our models from use by insurers in Florida, there was a significantly negative impact on FICO’s insurance clients and the hundreds of thousands of consumers who benefitted from the use of FICO Insurance Score models in Florida.
Since FICO cannot be left in a precarious position with respect to the protection of its intellectual property, if the drafted white paper is adopted, as written, by any state without necessary trade secrets and otherintellectual property protections in place, FICO may be forced to remove our FICO Insurance Score models from use by our insurance clients in that state, creating wholly unnecessary market disruption.
We look forward to working with the NAIC Casualty Actuarial and Statistical Task Force toward a regulatory review approach that protects the interests of all stakeholders, including the vast numbers of US consumers who benefit from the insurance industry’s continued use of predictive models to enhance their underwriting and pricing policies based on proven risk characteristics.
Sincerely,
Lamont D. Boyd, CPCU, AIMInsurance Industry Director, Scores and Analytics
[email protected] 602-317-6143 (mobile)
FICO Insurance Scores Consumer website at insurancescores.fico.com offers consumers, agents, regulators, legislators and others a thorough understanding of FICO’s credit-based insurance scores, the insurance industry’s use of our insurance scores, and general credit management tips.
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 78
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 79
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 80
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 81
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 82
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 83
January 15, 2019 Mr. Richard Piazza Chief Actuary, Actuarial ServicesLouisiana Department of Insurance P.O. Box 94214 Baton Rouge, LA 70804 Via Email
Dear Mr. Piazza: I appreciate your time the other day to allow LexisNexis Risk Solutions to offer feedback on the Regulatory Review of Predictive Models White Paper as an Interested Party. While our company historically has been known for our Attract Models (predictive models using credit-based insurance scoring used in Auto, Property and Commercial Lines), we have developed hundreds of custom models used by carriers, and are in the process of developing new and innovative models that will be filed in the near future (and many of which have been socialized with you and your staff). Additional information about our company and abilities can be found at https://risk.lexisnexis.com/products/predictive-modeling The following bullet points should be considered high-level feedback and commentary. We would like to meet with you further for a more in-depth discussion for input on your whitepaper.
We believe that the intent of this task force is to create uniformity across states to streamline the process of model filing, evaluation and implementation across states. We feel this is a step in the right direction as we would expect that filing reviews could be expedited.
The majority of items noted in the whitepaper are found in many state model filing check lists. When we develop models, our process includes documenting this type of information for both our own historical model development notes as well as for regulatory filings.
The proposed list is relevant to primarily one type of model (GLMs), which seems to be predominantly used in insurance. Modern methods, such as model blending or GBMs, are not well covered by these questions.
One area of potential concern is that the model requirements focus on the “science” part of the modeling process. Predictive modeling also involves a bit of “art”, including a modeler’s work to iterate variables, explore interaction effects, utilize multiple techniques, use sequential analysis, binning, etc. It may be worth providing some guidance to still allow for the “art” vs. pure science.
The majority of our modeling work at LexisNexis Risk Solutions is spent handling “dirty” or inconsistent data including extracting structure, imputing missing values, and handling incorrect data (to name a few). This step is very important since it will help us build data knowledge needed to apply the scientific standard of estimation based on a true statistical model. Modeling is difficult even if you fix the algorithm of choice, and just focus on the variable selection problem especially when dealing with large data sets with large number of inputs. However, using our knowledge of the data and experience, we can resolve to some heuristic techniques to find adequate solutions.
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 84
o For example, one of the rules is defined by the 5% threshold on p-value (in big samples, the threshold can be even more restricted). The p-value is the probability of finding the observed, or more extreme, result when the null hypothesis is true. Since the null hypothesis is usually associated with “no effect” of an attribute, a smaller p-value means that there is strong evidence that such attribute has indeed an impact on the target. However, many times (especially when sample sizes are small), the statistical approach may not capture the significance of some attributes and shows p-values greater than 5%. Based on previous experience, statistical modelers at LexisNexis Risk Solutions can identify important attributes that did not pass the 5% threshold and use alternative approaches such as bootstrapping or marginal impact to accept attributes with moderate p-values in the model.
Relative to the use of raw data (noted in Section A.5.a), LexisNexis Risk Solutions provides aggregate descriptive statistics related the loss experience of underlying modeling datasets. We work closely with state regulators to comply with various statues and regulations. However, we can only share data with regulators within the confines of our carrier contractual limitations. As good stewards of consumer data, we are extremely cautious with customer data and are often constrained by data privacy laws in the detail we can share. We work diligently with the insurer and regulator in each state regarding the confidentiality of the information submitted with the model filing.
Our biggest concern is sharing our extremely valuable intellectual property (e.g. our step-by- step statistical methods and key decisions). For example, the exact details on data scrubbing, variable selection methods, etc. would become publically available in some states. In particular, answering all of these questions would involve publishing algorithms and data processing, and the value of work on publicly available data, e.g. NFIRS, could plummet as all derived attributes would be exposed to copying. In banking, Basel requirements have similar goals and output, but since all work is kept internal, there is no issue with sharing intellectual property.
o We realize that allowing confidentiality of modeling filings is primarily set by state statute. Some of these allow auto and home models to be confidential, but stop short on other lines of business (e.g. Commercial). What we would propose is, if a state would require the recommended information, and if the state does NOT allow model intellectual property to be considered confidential, the state would accept the information via a different method (e.g. email to the reviewer) and kept out of the SERFF system.
Thank you again for allowing us to submit feedback. Please let me know if you have any questions. Sincerely, Gary T. Sanginario, CPCU Director, Product ManagementAnalytics Products and State Relations
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 85
To: Casualty Actuarial & Statistical (C) Task Force
From: Brent Kabler, Research Supervisor, Missouri Department of Insurance, Financial Institutions & Professional Registration
Re: Draft white paper Regulatory Review of Predictive Models
01-11-2019
First, I extend my appreciation to the CASTF for the obvious effort put into the draft. One thing that jumped out at me is that the document, taken as a whole, doesn’t seem to have an obvious audience. Namely, is this written for a non-technical audience that will have limited familiarity with statistical concepts? Or is the intended audience primarily working actuaries? If it is the former, I strongly encourage a significant rewrite of the draft to avoid jargon that will be entirely unfamiliar to a non-technical audience, and try as much as possible to convey concepts in a way that is broadly accessible as possible. I fully realize that is easier said than done, but as written a non-technical audience won’t be able to make much sense of a majority of the draft.
That aside, my primary concern with the draft is that it fails to address in any kind of serious way a problem that will increase significantly with the adoption of data mining techniques and the increasing availability of very large data sets that dwarf anything available even just a couple of decades ago. Data mining will (as has been shown across a wide variety of fields) dramatically increase the rate of false positives – the technique will inevitably churn up numerous associations between variables that are simply random non-meaningful correlations resulting purely from chance. I believe this is a near certainty. Secondly, the apparent complete disregard of causality that seems common among practitioners of data mining techniques will significantly magnify the problem. Causality forms the basis of the standard model of all natural and social sciences. I would argue that evaluations of models should consider the nature of observed relationships within the context of prior substantive knowledge. While I fully realize that the SOP accords a fairly diminished role for causality, I don’t think those standards justify dispensing with it entirely.
These problems are in no way esoteric or “purely academic.” Numerous disciplines have grown increasingly concerned with the “replicability crisis,” or the fact that a substantial proportion of published research cannot be replicated and are in fact simply false positives. It is worth nothing that these disciplines engage in far less data mining (if at all) than is common now among insurer rating practices. Indeed, in many disciplines, data mining can be considered outright academic fraud, depending on the nature of the disclosures made and egregiousness of the practice. This problem has grown so significant and is so widely recognized that the American Statistical Association took the unusual step of releasing a pretty strong warning against such practices (more on this below).
These comments in no way seek to upend standard actuarial practice. Rather, they are designed to raise awareness of the problematic nature inherit to data mining and hopefully to stimulate discussion about appropriate remediation measures. Data mining will no doubt remain an enduring feature of rating, given the strong competitive market pressures that incentivize rapid innovation. However, the practice raises some thorny problems for regulators seeking to ensure the validity of rating structures.
The truth is that data mining stands the standard scientific hypothetical-deductive method on its head (though it can’t rightly be called an inductive approach either). The standard scientific model usually starts with a body of substantive (or causal) knowledge. Rigorous hypotheses are derived from gaps in such knowledge. In it most rigorous form, all statistical tests are specified prior to conducting such tests, and the implications of each test are understood in terms of what they will reveal of a causal nature. Changing the tests after the fact
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 86
is discouraged, for strong methodological reasons. The correlation alone is not the final arbiter of the validity of findings, but causal understanding is employed to assess which correlations may be entirely due to chance, what are non-causal relationships, and which are most likely to be enduring causal relationships. This approach of employing prior knowledge to assess the validity of results is formalized most rigorously in methods known as Bayesian statistics.
Data mining generally proceeds in exactly the opposite manner. Rates are generally made in a vacuum of substantive or causal knowledge. Data mining employs various algorithms that may perform many thousands of statistical tests as they comb through enormous quantities of data that encompass countless variables. Results are interpreted from a series of purely post hoc explanations (or really, rationalizations), which often consist of little more than pure guesses as to the nature of the relationships that are churned out.
Such methods systematically undermine the level of confidence (in the statistical sense) that can be placed in results. Consider that statistical tests measure the probability that an some relationship observed in a data set is merely due to chance – a association that is entirely random (due, for example, to sampling error). The most common measure is the p-value, a meaningful probability that a relationship will be observed between variables when there is in fact no relationship beyond random chance. Generally, p-values are set to a maximum of .05, meaning that a relationship will be rejected unless there is less than a 5 percent chance that it would occur due to random chance alone. That is, if one hundred different relationships with a p-value of .05 are discovered, the chances are that five of them are non-meaningful chance relationships. When data mining churns out possibly many thousands of correlations, it will significantly increase the problem of such false positives.1
The literature is filled with countless examples of erroneous results produced by more or less random “interrogation” of data sets (which, again, are far less egregious than outright data mining). The CDC recently completed yet another study that found no relationships between vaccines and autism. As noted above, the study was designed in adherence to rigorous standards – testing protocols specifying all statistical tests were adopted at the very outset, and the study did not seek to alter or modify such tests over the course of the study. The study found, as expected, no statistical relationship between the presence or absence of vaccines at time of diagnosis, no observable effect of the timing of vaccines, etc.
A subsequent external “researcher” subsequently “found” a relationship between vaccines and autism diagnoses in a small cohort – African-American males that had received vaccines later than the medically recommended schedule. Examining this cohort alone, all measures proved statistically “significant.” This finding of course unsettled all of the usual suspects, was deemed proof of a “massive cover-up” at the CDC, and even resulted in a Congressional inquiry.
But no serious researcher accepts the finding as at all meaningful. As noted above, the more tests that are run, delving down to more and more narrowly defined subpopulations, the more likely random associations will be uncovered. The test wasn’t specified at the outset, but was clearly the result of simply running random statistical tests across different cohorts until a relationship is found, unguided by any prior knowledge. Nor is there any theoretical reason to accept this finding. After all, what possible causal mechanism would result in a vaccine-autism connection among just this cohort and no other?
It is clear that many historical rating variables are causally understood. The relationship between young drivers and crash risk is generally acknowledged. In part, the elevated risk is attributable to lack of experience. However, we also know that the young have a significantly higher crash risk than older individuals with the same driving experience (say, 30 year-olds that first obtain a license). As such, we can 1 A process sometimes referred to as “torturing the data until it confesses.”
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 87
reasonable infer that youthfulness per se is a risk factor. There is substantial support for this in the psychological literature, which amply documents higher risk-taking behavior, lack of foresight, and impulsiveness among the young. There is also strong evidence for why marital status may be related to risk.
However, there is little if any understanding of how credit score might be related to risk, or vehicle history scores, or any of the other countless rating variables of more recent provenance that have been uncovered via data mining techniques. If we lack any causal understanding that might afford a more Bayesian evaluation of model validity, we have to fall back on the nature of the statistical relationship itself which, as noted above, is increasingly likely to be attributable to pure chance with the explosion of data mining.
That said, I’m not necessarily of the old school of statistical thought that strongly asserts that data mining should never be performed, or that it is invalid on its face (though many statisticians do believe it should never be done). Data mining can have uses for exploratory analysis, or ways of suggesting promising avenues of research. But I am of the school that believes that data mining should never be the final analysis. Nor should analysis dispense entirely with notions of causality. Lack of rigor has produced such findings as
1. Individuals that have more rigid “back – white” cognitive styles are actually less able to physically distinguish different colors (Nosek, Spies, and Matt Motyl. 2012)
2. Bible “codes” existing as statistical proximity of various word pairs perfectly predict the future (Witztum, Rips and Rosenberg, 1994), a “finding” that literally launched a billion dollar industry even though thoroughly debunked as an egregious form of data mining,
3. Aspirin therapy for cardiovascular disease is more likely to result in death for Geminis and Libras but is beneficial for other astrological signs (from findings of a recent international study of survivors of heart attack, involving more than 134,000 patients in over 20 countries)
4. Listening to the Beatles’ When I’m 64 can literally change peoples’ actual chronological age (Simmons, Nelson and Simonsohn, 2011).
As noted above, the American Statistical Association expressed some degree of alarm at approaches similar to (though again, far less egregious than) data mining (Wasserstein and Lazer, 2016). In a formal statement of the ASA, the association warned against a purely “cookbook” approach to statistics: “a p-value near .05 taken by itself offers only weak evidence of the null hypothesis” (page 129). In addition, the ASA emphasized the centrality of causal prior knowledge in interpreting statistical results: “Researchers should bring many contextual factors into play to derive scientific inferences, including the design of the study, the quality of the measurements, the external evidence for the phenomenon under study [i.e. causal or theoretical knowledge], and the validity of the assumptions that underlie the data analysis” (page 130, emphasis added).
Lastly, the ASA warned strongly against an over reliance on data mining: ‘Cherry-picking promising findings, also known by such terms as data dredging, significance chasing…and ‘p-hacking,’ leads to a sspurious excess of statistically significant results…and should be vigorously avoided” (page 131, emphasis added).
But as we have seen, data mining goes well beyond simple offhand “cherry-picking” of findings. Indeed, it is a systematic and methodical system of cherry-picking. It is unclear how regulators should respond to the explosion of data mining, but it seems clear that the issue should be directly confronted. In general, the ASA recommends a model of full disclosure that requires that regulators acquire far more
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 88
information than is present in a typical rate filing. Regulators must be aware not only of the final model, but also the history of model building including the total number of hypotheses tested and all p-values computed.
I would encourage the CASTF to confront these issues far more directly in the draft, including fully outlining the nature of the problem and any remedial measures designed to guard against false positives. In part, the draft already offers some degree of remediation, including the importance of hold out or model validation data sets. This portion should be further accentuated in relation to data mining, and perhaps made mandatory for acceptance of any models predicated on data mining. Additional measures should also be addressed. For example, reported p-values should be adjusted based on the number of tests performed. One such commonly accepted test is the Bonferroni correction, in which the p-value is adjusted upwards in relation to the number of tests performed. Other adjustment methods are available in standard statistical texts.
In addition, full disclosure of model development history is absolutely essential. Proper interpretation of statistical results absolutely requires full knowledge of every stage of model development, and not just the final result. In part, the draft does some justice to this essential need. I would encourage making it a fundamental principle and include a robust discussion in the body of the draft.
Lastly, I strongly believe that regulators need to do better with respect to issues of causality. Actuarial standards of practice state that “While the actuary should select risk characteristics that are related to expected outcomes, it is not necessary for the actuary to establish a cause and effect relationship between the risk characteristic and expected outcome in order to use a specific risk characteristic” ( Actuarial Standard of Practice No. 12, Section 3.2.2, emphasis added). Unfortunately, this statement has most often been interpreted to mean that actuaries (and regulators) can dispense with causality entirely. As noted above, proper statistical interpretation absolutely requires some knowledge of causality. While there are some statements in the draft that could be interpreted as addressing this issue (however obliquely) I encourage the CASTF to tackle this issue head-on. It is entirely appropriate in standard statistical methods of interpretation to bring all prior knowledge and understanding to bear in assessing the validity of a statistical model. As the ASA emphasized in the quote above, it is an essential part of any such interpretation.
Again, I am not suggesting that actuarial practice be upended. It would be quite costly for insurers to adopt the most rigorous and methodical and above all slow methods of the sciences. But the positives of innovation must be tempered with proper evaluation by regulators of models. A false positive, by definition, violates the universal standard of treating like risks the same. These issues aren’t purely academic in nature, but clearly have practical implications for model review. I would strongly encourage they be confronted fully in much the same way that other disciplines are beginning to.
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 89
Bibliography
Nosek, Brian A., Jeffrey R. Spies, and Matt Motyl. 2012. Scientific Utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science. 7(6): 615-631
Simmons, Joseph P, Leif Nelson, and Uri Simonsohn. 2011. False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psycological Science. 22: 1359 – 1366. Text available at http://people.psych.cornell.edu/~jec7/pcd%202015-16%20pubs/Simmons%20PsySci%202011.pdf
Witztum, Doron, Eliyahu Rips and Yoav Rosenberg. 1994. Equidistant letter sequences in the book of Genesis. Statistical Science. 9: 429-438.
Wasserstein, Ronald L. and Nicole A. Lazar. 2011. The ASA’s statement on p-values: Context, process and purpose. The American Statistician. 70: 129-133.
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 90
January 15, 2019 NAIC Casualty Actuarial and Statistical (C) Task Force c/o Kris DeFrain - [email protected] 1100 Walnut Street, Suite 1500 Kansas City, MO 64106-2197 Re: NAMIC Comments on CASTF’s Predictive Model White Paper Dear Task Force Chair, Vice Chair, Task Force Members, and Other Interested Regulators, Please accept the following comments of the National Association of Mutual Insurance Companies (hereinafter “NAMIC”)1 on behalf of its member companies regarding the exposed Predictive Model White Paper. NAMIC wishes to thank the Task Force for the ability to provide comments on the white paper and this extremely important concept of predictive modeling as it impacts the industry and consumers. Further, NAMIC wants to commend the Task Force for its diligence and thoroughness in attempting to ascertain “best practices” for predictive modeling analysis in the property/casualty insurance market. Despite the well-intentioned drafting of predictive model review parameters, there remains concerns that this document does not provide practical real-world examples to achieve the necessary goals of consumer protection and approval of filings so that companies may continue to compete in the marketplace in a timely and efficient fashion. Quite simply, the white paper calls for an inordinate amount of compliance expenditure and leaves very little discretion in the handling of even rudimentary filings. The compliance costs and human capital associated with responding to the various and detailed criteria will be enormous and potentially expose proprietary trade secrets to excessive review and dissemination. All of this may be occurring without the slightest scintilla of concern or other potential regulatory trigger being posited. Some generalized and specific concerns would include the following.
NAMIC is the oldest property/casualty insurance trade association in the country, with more than 1,400-member companies
representing 41 percent of the total market. NAMIC supports regional and local mutual insurance companies on main streets across America and many of the country’s largest national insurers. NAMIC member companies serve more than 170 million policyholders and write more than $253 billion in annual premiums. Our members account for 54 percent of homeowners, 43 percent of automobile, and 35 percent of the business insurance markets. Through our advocacy programs we promote public policy solutions that benefit NAMIC member companies and the policyholders they serve and foster greater understanding and recognition of the unique alignment of interests between management and policyholders of mutual companies.
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 91
Inordinate Compliance Costs The document consists of approximately 27 pages of “best practices” containing more than 86 “essential” and separate items of analysis and review of filings to be considered by each insurance department. Each of these “essential” items can contain multiple steps to complete making the actual compliance threshold much higher. NAMIC respectfully suggests that most of these “essential” steps should provide for more regulator discretion and be converted to “May Be Requested.” As written, the scope of the white paper would inevitably and dramatically increase compliance costs in responding to inquiries about rudimentary filings that will drain resources that could be more efficiently and effectively utilized in policyholder services and operational concerns of the company.2 Speed to Market Inevitably, NAMIC would submit that this paper could cause a significant reduction in timely filing approval causing not only the aforementioned inordinate compliance costs but the ability for companies to compete in the marketplace. If filings are delayed, that delay ultimately harms consumers as market competition is reduced.3
2 By way of example, names and contact information for those who built the model, explanations of how the insurer will help educate consumers to mitigate risk when some factors such as age or gender cannot be “mitigated,” and when the models were begun and finalized, might be information a regulator would like to see but it should by no means be considered “essential” to review and approve a filing.
3 By way of example, items such as the following seem to be a best thought but not necessarily a mandatory best practice: (1) “Provide a complete list of all characteristics/variables used in the proposed rating plan, including those used as input to the model and all other characteristics/variables used to calculate a premium.” (2) “For each characteristic/variable used as both input to the model and as separate univariate rating characteristic, explain how these are tempered or adjusted.” (3) “Provide state specific, book of business specific univariate historical experience data consisting of, at minimum, earned premiums, incurred losses, loss ratios and loss ratio relativities for each category of model output proposed to be used within the rating plan.” The last example completely goes against the idea of multivariate model-based rating plans. If an insurer still needs to be able to support the model with state-specific univariate results, the concepts conflict causing substantial more compliance adherence. Likewise, with “[p]rovide an explanation of any material differences between model indications and state specific univariate indications.” In this situation, if an insurer built a companywide model for deployment in 50 states, it would have to explain all material differences that the model had with individual state univariate analysis. This is excessive, unnecessary, costly, and moves the process in completely the wrong direction. The same concern applies to the comments section of this line that also states, “Credibility of state data should be considered when state indications differ from modeled results based on a broader data set.”
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 92
Department Expertise and Increased Budgetary Costs NAMIC believes that in addition to the previous considerations, additional thought should be expended on the compliance costs for the regulators themselves. While this work is exhibited to be a “best practice” white paper, it is not remote logic that states will extrapolate and adopt this paper as their approval manual. Otherwise, they could be left open to criticism that they failed to follow an “essential” act contained in the paper. It would then appear from these parameters that a great deal of staff, outside expertise, and increased budgetary costs are going to be incurred. A fiscal impact analysis should be ascertained before this paper is adopted. NAMIC is not suggesting that any filing that has reasonable triggers or cause for regulator concern should not be deeply and thoroughly examined for compliance with existing laws. However, CASTF will be causing this rigorous scrutiny of each and every filing. The exposure and ultimate adoption of this paper cannot be accomplished in a vacuum without considering ancillary impacts. Another consideration is that with such complexity of models, it is disconcerting to see guidance in the white paper that states “[g]iven an insurer’s rating plan relies on a predictive model and knowing all characteristics of a risk, a regulator should be able to audit/calculate the risk’s premium without consultation with the insurer.” We fail to see how a best practice would ever encompass failing to discuss a concern with the regulated entity to at least obtain their response as to their position. Basic fairness and due process notions would dictate an outreach when there are discrepancies or unanswered questions. Too Prescriptive Many of the “essential” requirements seem to be too prescriptive as to intent. While in a perfect best-case vision, some of these scenarios may be determined to be useful on a case-by-case basis, many go too far in the required analysis. Some require regulators to inquire about predictor variables in an old model that may no longer be used in a newer version. This requirement seems to disregard the dynamic nature of modeling and may lead to improper assertions and conclusions of the regulatory body over a matter that is no longer utilized. Additionally, there is a discussion about measuring and describing the impacts on expiring policies and describing the “process used by management to mitigate or get comfortable with those impacts.” There does not appear to be a legal requirement for “mitigation” and that concept is not clearly spelled out within the document. This leaves the matter open to broad and problematic interpretation. The same would go with the concept of “get comfortable” with an impact. These are not clarified terms and do not belong in this document. One can envision a regulator making a unilateral decision that management simply never got comfortable with what they implemented. This would provide simply no factual usage in determining the legal and regulatory parameters to ascertain filing approval. It will clearly slow down the process.
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 93
The general legal parameters for rate filings require them to be adequate, not unfairly discriminatory, or excessive. It is difficult to extrapolate why these concepts previously mentioned and contained in the white paper assist in ascertaining those thresholds. The concept of “large premium disruptions” and providing analysis for an anticipated negative impact appears to be already assumed prior to the analysis despite legal requirements and financial prudential mandates that rates be adequate. This inquiry is considered “essential” before lack of actuarial justification has even been identified. Items such as “[e]xplain how the insurer will help educate consumers to mitigate their risk” is another such paradox. This does not appear to be a legal requirement but rather a public policy pronunciation. It is not a best practice to inject such discussion into a rate filing. Likewise, an inquiry that states “[p]rovide the regulator with a description of how the company will respond to consumers’ inquiries about how their premium was calculated,” is deemed “essential” but is not really connected with determining the adequacy, excessiveness, or other criteria for a rate filing. It appears that a market-conduct-type inquiry is already being interjected into a rate filing analysis that will only take up needless time to respond appropriately and is premature when no trigger or actual perceived harm has been demonstrated. Confidentiality, Proprietary Information, Trade Secrets, Contractual Terms, and Information Sharing Exposing confidential and proprietary trade secrets, confidential information, and other business practices simply for accumulation of data in a rate filing, when otherwise unnecessary, is problematic for all involved. States may information share the data by law with other regulators. The data provided for these “essential” requirements subjects the regulator to increased Freedom of Information Act requests, subpoenas, and other types of litigation when there has been no demonstrated harm to consumers or trigger for the inquiry. Additionally, some proprietary models may have contractual terms that prevent disclosure and therefore an interference with contractual relations may occur. Without a demonstrated necessity, exposing this data to additional dissemination appears to be hindering its protection. The NAIC should update and strengthen its information-sharing platforms and protocols to absorb such complex and proprietary data if this is the path forward. It is unfortunate that the only discussion of confidentiality in the document makes essentially two points. One, that the information filed with departments “might become public” and two, that “it is incumbent upon the insurer to be familiar with each state’s laws regarding the confidentiality of information submitted with their rate filing.” There is no concomitant duty mentioned for regulators to protect confidential and proprietary information and the phraseology essentially attempts to alleviate that burden entirely. As custodians of this sensitive and complex information, more should be stated in the paper and, in fact, done to protect the same especially since it is the regulator who is demanding this broad and sensitive information without any particular threshold of concern.
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 94
Confusing or Non-Illuminated Terminology and Word Selection
Additionally, NAMIC believes a revisit of terminology used throughout the document should occur. As mentioned herein, terms such as “clear understanding,” “premium disruption,” “intuitive argument,” “large premium disruption,” “significant changes in premium,” and “get comfortable” are but some of the verbiage used throughout the paper that do not have quantifiable definitions. Consequently, they will be subject to broad interpretation by the reviewer that can only lead to more potential follow-up and compliance requirements in the absence of a concern or issue with the filing itself. These broad terms, including the use of “essential,” will result in every rate filing being a complex and time-consuming compliance endeavor. Again, it is ultimately the consumer who will be potentially harmed when innovative products cannot timely reach the marketplace.
Therefore, in closing, NAMIC would again thank the Task Force for its tireless work on this topic. However, while the best of intentions was undoubtedly attempted in this draft, significant concerns remain about its efficacy in today’s marketplace. We would ask that you take our considerations under advisement and consider another draft of this white paper alleviating the prescriptive rigid dynamic that has been created. NAMIC would offer its assistance to the Task Force and looks forward to working with the Task Force on this critical work where needed.
Sincerely,
Andrew Pauley, CPCU Government Affairs Counsel National Association of Mutual Insurance Companies (NAMIC)
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 95
To:
DeFrain, Kris, Piazza, Rich Vigliaturo, Phillip Darby, Sandra McGill, Mark Stolyarov, Gennady Davis, Daniel J. Eric Hintikka From: Gordon Hay Date: November 16, 2018
Re: Draft CASTF Model Review White Paper
Tomasz Serbinowski commented:
“Several places in the draft suggest that the regulator should require an intuitive or logical support for the selected explanatory variables. Why this sounds reasonable, it probably is not. Historically, auto insurers were allowed the use of gender as a rating variable. I don't know of any acceptable logical or intuitive explanation of that. I would venture a guess that an insurer could not offer any explanation other than the data showing that losses vary by gender.”
I have given this some thought, and I recommend:
We should guide reviewers to challenge variables for which the rate filer provides no explanation that rings true intuitively and logically. Some candidate variables are simply not sensible compared to more intuitive and logical alternatives.
However, “fairness” has always had more than one definition, leading to a balance or trade-off between fairness by solely statistical/actuarial criteria versus fairness that reflects socio-economic or cultural values. Sometimes an issue gets legislative or regulatory outcomes that differ depending on local conditions that may evolve over time. I think the white paper needs to explicitly acknowledge that “intuitive and logical” can be subjective and subject to change. States may need to give reviewers written guidance on variables that are prohibited and/or outline a procedure for regulating the introduction or sun-setting of variables that may be sound actuarially and statistically but not so sound politically.
My thoughts getting to that recommendation?
“Intuitive” and “Logical” may be desirable to all, but definitions will get into philosophy. In any given context, judgments based on intuition or logic inevitably depend on subjective perceptions and/or
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 96
beliefs. There has always been a difference between “unfairly discriminatory” in a statistical sense versus concerns about fairness that get political or legal treatment using logic that could follow statistical evidence, compromise with it, spin it, or simply discount it.
In practice, rate filers should have avoided using variables for rating, underwriting or other purposes that are legally or politically unacceptable, whether statistically predictive or not. So “race” to my knowledge is explicitly omitted everywhere, and gender should be omitted where it is prohibited. There may always be variables that are found to measure something unacceptable, making statistical validation beside the point. E.g. insurance applications for personal credit data have been under scrutiny for decades, and are prohibited for personal lines rating by some states. As modelers reach more deeply into “big data,” new predictive variables keep emerging. Some geodemographic variables with “poor optics” might be avoided for appearances sake, avoiding any arguments over their potential correlation with some prohibited variable.
Rate filing reviewers have always been wary of new classification variables, but seeing new variables several times a decade. Currently there’s a proliferation of new data sources and candidate variables, and inevitably some of them will be “unfairly discriminatory” due to some combination of statistical, legal or political criteria. States will differ in their positions regarding regulatory hurdles for innovation, including acceptability criteria for new variables. If left to the individual reviewer’s discretion, it isn’t realistic to expect consistent review criteria from one filing to the next, or sustained maturity of review from one reviewer to the next. If a State has taken positions on what types of variables are legally and politically acceptable in multi-variate rating systems, it might be a “best practice” to provide model reviewers (and rate filers) with a written guideline.
We should still guide reviewers to challenge variables for which the rate filer provides no explanation that rings true intuitively and logically. Some variables are genuine nonsense. Let’s say we’re trying to predict commercial GL loss ratios and make a Location Index for each zip code that’s fitted (with other non-geographic variables) using three census data elements: Number of death care services establishments, Number of office and stationary stores and Percentage of population between ages 80-84 from US Census data. Does that pass the “intuitive and logical” test, or do we ask for an explanation with reasons why these variables in particular? Is selection using some statistical technique going to suffice, or do we guide the reviewer to request an “intuitive and logical” explanation?
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 97
1
Comments on Draft NAIC CASTF White Paper on the Regulatory Review of Predictive Models Gennady Stolyarov II, FSA, ACAS, MAAA, CPCU, ARe, ARC, API, AIS, AIE, AIAF
Lead Actuary, Property and Casualty Insurance Property and Casualty Section, Nevada Division of Insurance
Kris DeFrain, FCAS, MAAA, CPCU Director, Research and Actuarial Services National Association of Insurance Commissioners (NAIC) Sent via e-mail at [email protected] January 13, 2019 Dear Ms. DeFrain: Thank you for the opportunity to comment on the draft of the NAIC Casualty Actuarial and Statistical (C) Task Force white paper on the Regulatory Review of Predictive Models. My comments below are subdivided into a section of substantive remarks, followed by a section recommending some minor editorial changes. Substantive Remarks While the paper remains in a draft form with various sections that will still need to be added, expanded, or refined, I consider the conceptual framework and specific elements of guidance presented in the current exposure draft to reflect an essentially correct approach. Having participated in the group of regulatory actuaries who contributed to this exposure draft, I am aware of the extent of thoughtful and thorough discussion and consideration of each element that went into this process. While different States have different legal and regulatory frameworks, and the intent of this paper is to be consistent with all of them, the discussions among the paper’s drafters elicited recognition that there are many common areas of focus, desired understanding, and – in certain cases – concern that regulators who review predictive models experience in their work. This paper strongly emphasizes that the best practices therein are guidance and that States’ regulatory autonomy takes precedence; this is exactly as it should be. At the same time, it remains each reviewer’s professional prerogative to seek a deeper understanding of the models he or she is reviewing, to ask questions, and to discern – even if only for his or her benefit (though often with other, more impactful implications as well) – whether a predictive model is appropriately supported, and whether the constituent elements of that predictive model make sense. Different States will apply this guidance differently. Prior-approval States, such as Nevada, already require answers in connection with many of the elements expressed in this paper. Other States may use the guidance in this paper to choose which model elements to focus on and/or to train new reviewers or to gain a superior understanding of how predictive models are developed, supported, and deployed in their markets. The above context for this paper renders it essential that some of the more conceptual and qualitative guidance in it remain in place, notwithstanding potential (and most likely actual) philosophical objections from certain interested parties. Most regulatory actuaries and other filing reviewers are aware that, for a long time, certain elements within the property/casualty
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 98
2
insurance industry have made what I will term the “correlation-only argument” – which asserts that to demonstrate a lack of unfair discrimination for a variable, it is enough to show a statistical correlation between insurance losses and/or expenses, and that even the very discussion of causal possibilities should not be attempted. Both I personally and the Nevada Division of Insurance as a regulatory agency disagree strongly with this “correlation-only” position. While, indeed, it is difficult to prove causation, and such a proof is not a standard against which rate filings are evaluated in any jurisdiction to my knowledge, there is an immense difference of both degree and kind between proving causation and discussing an intuitive or logical connection between a particular attribute and the risk of insurance loss. It is a non sequitur to assert that the lack of requirement for the former (proof) confers immunity upon insurers in regard to the latter (discussion and expression of plausibility).What a State does with the results of such a discussion is, of course, subject to the framework of statutes, regulations, precedents, and processes that comprise the insurance regulatory framework in that State; these will necessarily vary by jurisdiction. However, the very act of discussion of the intuitive, logical, or plausible relationships of individual risk attributes to the risk of insurance loss – and consideration of all related and relevant implications (such as perception by consumers, legislators, and media; philosophical considerations of fairness; interactions with public policy as determined by the relevant policymaking bodies; and relevance to the evolution of the insurance industry, consumer products, and overall impacts on the incentives and opportunities available to consumers) – is crucial to engage in and continue to do so for as long as new predictive models are being developed, new variables are being introduced, and consumer premiums as well as insurer underwriting decisions are being affected. In other words, the discussion needs to continue indefinitely in a variety of venues and evolve along with the industry and the broader society; we as insurance professionals cannot viably insulate ourselves from participation in the conceptual discourse. Furthermore, as actuaries, if we are indeed to practice the discipline called actuarial science, then it is incumbent upon us to adopt the proper scientific mindset of open inquiry – where no questions are off limits and continued systematic exploration and progress are the hallmarks of the scientific approach. Any insistence that certain questions must not be asked, or certain concepts must not be explored, that certain discussions must simply be cut off, entails a departure from the realm of science into the realm of dogma; if widely acquiesced to, this mentality would ossify the profession and quickly deprive it of broader relevance. Regulatory actuaries and other trained staff, especially when they review predictive models, are in a prime position to be the torchbearers for the scientific approach by maintaining the commitment to open but rigorous, systematic, and principled inquiry. It is important to emphasize that the white paper, as currently drafted, does not prescribe any specific answers regarding which particular treatments are to be considered logical or intuitive. Such answers cannot be arrived at without considering the context of a given jurisdiction’s laws, marketplace, and the specific nature of insurers’ proposals. Therefore, to preempt any arguments by some interested parties that the paper may be attempting to prescribe specific solutions or restrictions – it clearly is not. The purpose of the paper is to provide guidance to enable regulatory reviewers to be aware of possible questions to ask and issues to consider so as to add value to the review process and improve the robustness of the regulators’ consumer-protection role. It is entirely possible that different individuals and different jurisdictions will ultimately arrive at different answers to these questions, just as actuarial judgment may be deployed by
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 99
3
different actuaries in different ways and with different opinions and recommendations. All this is part of the continuing discussion and exploration of these essential questions. A common counterargument may be to present certain circumstantial characteristics that have been in use for a long time in most jurisdictions, such as age or gender, and suggest that these do not have a readily apparent intuitive relationship to the risk of loss for certain lines of insurance – such as automobile insurance. Such a counterargument, however, falls short in that it does not recognize that members of the general public are at least capable of positing comprehensible hypotheses as to why these attributes may (or may not) be predictive of risk. Opinions on such matters will certainly vary, and my own anecdotal experiences with laypersons’ perspectives have put me into contact with a wide-ranging diversity of thought. However, the key regarding older and more established rating attributes is that intelligent individuals can at least have discussions and provide plausible narratives regarding such characteristics. (Here I do not take any position as to which, if any, of those narratives is actually correct in explaining the data that insurers observe and present.) This, however, is not the case when it comes to esoteric rating variables that appear to have come into being solely because a predictive model correlated insurance losses with a particular characteristic that happened to be found in a dataset which an insurer purchased from a third party that was eager to market the data as having some insurance applications. While some innovative, reasonable, and truly predictive new attributes could indeed constitute improvements to a rating plan, this is not a license to use just any attribute merely because a correlation was found in some instances for some data sets. Many regulatory reviewers have encountered attributes, proposed to be used in rating, which could not possibly be related to actual consumer behaviors that affect the risk of insurance loss – or, if they are related, the relationship is directionally contrary to the proposed treatment of the attribute. (An example of the latter would be a behavior indicative of financial responsibility – such as refraining from taking out an installment loan or paying off an installment loan – being used adversely, to surcharge a consumer in a credit-based insurance scoring model. The Nevada Division of Insurance has, over the past decade, made efforts to surgically excise such treatments from credit-based insurance scoring models with no loss of the models’ predictive ability.) As ever-accumulating volumes of consumer data are being generated and sold, it is incumbent upon all decision-makers to thoughtfully consider the purposes to which such data are being put – not only whether such uses respect the essential values of privacy and consent, but also whether tying individual consumer decisions (such as purchasing decisions, social-media habits, or other lifestyle choices) to external and unanticipated financial consequences (such as insurance premiums) could trigger situations where a single innocuous (and entirely lawful) or even clearly laudable choice could result in a disproportionate cascade of unforeseen and unforeseeable (by the consumer) impacts that disrupt an individual’s financial situation and prospects. Again, the above is an expression of concerns and important areas of consideration – not a prescription for a solution, which can only emerge in the course of a thoughtful and multifaceted review process which upholds the autonomy of individual States and is consistent with their legal environments. For the above reasons, it is essential in my view that the white paper preserve the existing references to seeking an understanding of the logical or intuitive connections of specific model attributes (and not just the model as a whole) to the risk of loss (or future expense) that any given model is seeking to predict.
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 100
4
Editorial Suggestions The following are suggestions for minor editorial revisions for which I discerned the need upon reading through the exposed draft. I understand that the paper is still in a preliminary draft form, and that there will be various opportunities in the future to make these editorial changes and any others deemed necessary. Section VI (Page 5) The 5th-to-last bullet point should read as follows (revise punctuation): “● Obtain a clear understanding of how the predictive model was integrated into the insurer’s state rating plan and how it improves the state rating plan. (This latter element is only applicable when a new or revised model is introduced into an existing rating plan.)” The 2nd-to-last bullet point should read as follows (revise “each risk characteristics” to “each risk characteristic”): “● Obtain a clear understanding how often each risk characteristic used as input to the model is updated and whether the model is periodically rerun to reflect changes to non-static characteristics.” The last bullet point should read as follows (add “that” after “rating plan”): “● Given an insurer’s rating plan that relies on a predictive model and knowing all characteristics of a risk, a regulator should be able to audit/calculate the risk’s premium without consultation with the insurer.” Section VII (Page 5) First paragraph: In the sentence “Nor is every item on the list intended to be a required for every filing”, change “required” to “requirement”, so that the sentence reads as follows: “Nor is every item on the list intended to be a requirement for every filing.” B. Building the Model - Item B.1.d. – Change “analyses was performed” to “analyses were performed”. - Item B.2.f. – Change “Were there data situations GLM weights were used?” to “Were there data situations in which GLM weights were used?” Thank you again for the opportunity to provide these remarks for the Casualty Actuarial and Statistical Task Force’s consideration. Sincerely,
Mr. Gennady Stolyarov II, FSA, ACAS, MAAA, CPCU, ARe, ARC, API, AIS, AIE, AIAF Lead Actuary, Property and Casualty Insurance, Nevada Division of Insurance
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 101
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 102
Casualty Actuarial and Statistical (C) Task Force Regulatory Review of Predictive Models
White Paper - Exposure Draft
Table of Contents – Given the overall length of Section VII, a refinement to the table of contents to add subsections to Section VII would be helpful.
Section I:
Paragraph 2: o Reading of sentence, “When that back-and-forth learning is history, filing requirements…” was
unclear. o The word “even” is redundant in the sentence “Hopefully, this paper helps bring more
consistency and even uniformity to the art of reviewing predictive models…”.
Section VI:
I found some confusion and lack of clarity around the inclusion and juxtaposition of two guidance’s: o “Determine that individual input characteristics to a predictive model (and its sub-models) are
not unfairly discriminatory” o “Determine that individual outputs from a predictive model and their associated selected
relativities are not unfairly discriminatory”
The unfairly discriminatory language, from an actuarial context, is often used to convey concern that the prices used do not accurately reflect a reasonable relationship to cost for certain elements of the classification system. There are other considerations actuaries must make to comply with law or regulation in specific jurisdictions, where certain pricing practices or individual policy characteristics are forbidden for use in rating (e.g. use of education level, or use of gender for class rating in auto). I was unclear as to whether these statements were meant as statistical guidance, compliance with statute and regulation, or both?
Fourth to last bullet – change “Determine the extent the model…” to “Determine to what extent the model…” Second to last bullet – change “characteristics” to be “characteristic”.
Section VII:
General Statement – At times the guidance provided in this section appears to be very heavily focused on the statistical process of modeling. For example, review of the sections listed below indicate them being an essential element of the Regulator’s review:
o B.1.i o B.2.d o B.2.e o B.3.b o B.3.d o B.4.c
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 103
o B.4.d o B.4.e o B.4.f o B.4.g o B.4.h o B.4.l
While these items are well articulated and would drive a thorough statistical review of the modeling process, my concern would be over the level of requirement put on them. The review, interpretation, and drawing of conclusions from these items would often require an individual who is well trained in actuarial science, statistics, or data science. Is there potential for jurisdictions who do not retain or hire individuals with proper backgrounds in these areas to be overwhelmed or be presented with information that is not valuable to their market regulation process?
Is it reasonable to have more variation in the level of review? For instance,
o “Essential for Regulatory Review”: Review of input variables for compliance with statutes and regulations. That is to say, is the Company including any variables that might be prohibited in the jurisdiction and the model should be withdrawn. Review the target variable of the model. Reviewed to understand whether the model is predicting frequency, pure premium, loss ratio, or some other cost related metric. This would be important for winnowing out loss based models versus life-time value based models for instance. Review of combined output of the model in comparison to performance of current pricing plan. Does the final GLM (in the context of the white paper) perform better at predicting the target variable than the current pricing practice? Review the implementation of the model within the rate plan. Is the rating plan that is actually proposed in line with what the model says, or were other constraints, modifications, or judgments applied post-modeling.
o “Required for Actuarial Opinion to be rendered”: B.1.i B.2.d B.2.e Etc.
o “May be Requested”. A.2. – Consider including a list item on the impact of bias from overlapping data or variables in both sub and primary models in all phases of the model development process (similar to C.1.d) B.1.e – After this section, we believe a clearer call to document how the raw data was divided into the modeling/testing/validation datasets would be helpful. B.1.f – Additional exposition of this element within the comments section would prove valuable. B.3.a –
o This section is the only area where control or offset variables are discussed. Is there any intended review of control or offset variables in the GLM that is implied in the other sections? Similarly, should there be any inquisition into why it makes sense for a variable to be a control or offset within the model. For instance, we sometimes see geodemographic data used as a control variable for territorial rating effects. This process might make sense depending on how the territorial system was developed and how interaction between these two classification systems was established.
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 104
o Additionally, this section and others make no specific mention of the use of interaction terms as input variables within the model. Is the investigation of interaction terms intended to be implied by the other items of Section VII?
B.3.d - Within the charge, it should also be requested for the company to document how the results of the PCA process were used within the GLM. B.4.a – It would be helpful to add something in the comments like “it is useful to understand the rationale for dividing the datasets and why the selected approach was deemed most appropriate.” B.4.k – This statement was unclear upon my reading. Is it asking for a descriptive statement for how the concern around overfitting was addressed, and a descriptive statement for how correlation test were performed or considered? Or is it requesting specific results of correlation tests be documented in the filing (Correlation Matrices for instance). C.1.a & C.7.g – Their appeared to be lack of clarity in these sections between the “Importance to Regulator’s Review” and the “comments” section. This comes from the identification that it is “essential” while the comments suggest that it becomes “essential” upon certain conditions? Is this a “may be requested” with the comments dictating when it changes status? C.1 , C.2, & A.2 – There appears to be unclear requirements as to how the regulator should go about reviewing ensemble models. The section on sub-models largely appears to be considering the use of other commercial models as input variables into the GLM (Credit scores, cat models, etc.) While sections do address questions like whether the GLM was performed by-peril or Frequency/Severity vs pure premium, it is unclear whether anything in the guidance would suggest the regulator question how a model ensemble was performed to derive rate factors proposed. To be clear, I am speaking of a simple situation (which we see commonly), where a GLM might be built on a frequency and severity basis, but they are combined through some process to derive a single indicated rate factor. Similarly, we have seen for homeowners where a separate GLM is created for each peril (fire, wind, theft, liability, etc.) which are then combined to generate a single indicated rate factor applicable to all perils. These processes for combining various GLMs are becoming increasingly common.
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 105
From: Tomasz Serbinowski <[email protected]> Sent: Thursday, November 1, 2018 4:30 PM To: DeFrain, Kris <[email protected]> Cc: Klausmeier, Tracy <[email protected]> Subject: Preliminary comments on the Predictive Modeling white paper I would like to offer some preliminary comments on the CASTF's white paper "Regulatory Review of Predictive Models". These comments are made on my behalf and are not meant to represent the opinions of the State of Utah. 1. Several places in the draft suggest that the regulator should require an intuitive or logical support for the selected explanatory variables. Why this sounds reasonable, it probably is not. Historically, auto insurers were allowed the use of gender as a rating variable. I don't know of any acceptable logical or intuitive explanation of that. I would venture a guess that an insurer could not offer any explanation other than the data showing that losses vary by gender. 2. Several bullets in Section VI (Best Practices) require that a characteristic used have "an intuitive or demonstrable actual relationship to expected loss or expense". Would the "demonstrable" part be satisfied by showing a correlation between the characteristic and the loss or expense? 3. Would the requirement of "intuitive or demonstrable" relationship of characteristics to the loss or expense extend to sub-models? This could create problems. Number of open accounts may be an input to a credit score model. Credit score may have a demonstrable impact on expected loss. However, the insurer may not the data showing that the number of open accounts have demonstrable impact on the loss or expense. 4. Some bullets in Section VI (Best Practices) require determination that individual characteristics not be unfairly discriminatory. Would that determination entail more than showing "intuitive or demonstrable" impact on the loss or expense? Is the idea to make sure that the impact the characteristic has on the rate commensurates with the loss/expense differential? 5. Eight bullet in Section VI (Best Practices) requires understanding "why the insurer believes this type of model works". Would this be conveyed through goodness of fit statistics, lift charts, etc.? 6. Item A.1.g.in Section VII and 14th bullet in Section VI might need a clarification that the insured's data that might have been included in building the model is not subject to the consumer audit. Only the data used in rating is. Any problems with individual insured's data that was included in building the model would fall under date quality and would have impact on all insureds (because it would impact the final version of the model). 7. Item A.5.a stipulates that raw data could be provided if it is in a format that can be made available to the regulator. What does that mean? What would be the format that could not be provided to the regulator? Is this about the size of data file? 8. Item B.2.e requires an explanation of "the link function distribution". How does that relate to Item B.4.l that requires demonstration that "the GLM assumptions are appropriate"?
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 106
Sincerely, -- Tomasz Serbinowski, Actuary Utah Insurance Department State Office Building, Room 3110 | 350 North State Street | Salt Lake City, UT 84114 P: 801-537-9289 | [email protected]
Attachment Three-B Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 107
Attachment Four Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 1
Draft: 2/11/19
Casualty Actuarial and Statistical (C) Task Force Conference Call January 29, 2019
The Casualty Actuarial and Statistical (C) Task Force met via conference call Jan. 29, 2019. The following Task Force members participated: Steve Kelley, Chair, represented by Phillip Vigliaturo and Connor Meyer (MN); James J. Donelon, Vice Chair, represented by Rich Piazza and Lawrence Stewart (LA); Lori K. Wing-Heier represented by Mike Ricker (AK); Jim L. Ridling represented by Daniel J. Davis (AL); Ricardo Lara represented by Lynne Wehmueller (CA); Michael Conway represented by Rolf Kaumann and Sydney Sloan (CO); Paul Lombardo represented by Qing He (CT); David Altmaier represented by Howard Eagelfeld and Robert Lee (FL); Colin M. Hayashida represented by Randy Jacobson (HI); Doug Ommen represented by Travis Grassel and Andria Seip (IA); Kevin Fry represented by Judy Mottar (IL); Vicki Schmidt represented by Nicole Boyd (KS); Eric A. Cioppa represented by Sandra Darby (ME); Anita G. Fox represented by Kevin Dyke (MI); Chlora Lindley-Myers represented by Julie Lederer and Anthony Senevey (MO); Marlene Caride represented by Mark McGill and Carl Sornson (NJ); John G. Franchini represented by Mark Hendrick and Anna Krylova (NM); Barbara D. Richardson represented by Gennady Stolyarov II (NV); Jillian Froment represented by Thomas Botsko (OH); Glen Mulready represented by Andrew Schallhorn (OK); Andrew Stolfi represented by David Dahl (OR); Jessica Altman represented by Kevin Clark and Michael McKenney (PA); Raymond G. Farmer represented by Will Davis (SC); Kent Sullivan represented by J’ne Byckovski, Brock Childs, Nicole Elliott, Miriam Fisk and Jennifer Wu (TX); and Mike Kreidler represented by Eric Slavich (WA). 1. Discussed a Comment Letter on the Statement of Actuarial Opinion Instructions
Mr. Vigliaturo said the Task Force and the Executive (EX) Committee’s ad hoc group exposed revisions to the Property/Casualty (P/C) Statement of Actuarial Opinion instructions on Dec. 15, 2018. The Task Force met Jan. 8, 2019, and Dec. 18, 2018, via conference call to discuss the proposed changes to the instructions and decided to draft a comment letter on the changes proposed. Mr. Vigliaturo said the comments would be made only on the changes the ad hoc group proposed and not on the changes the Task Force proposed. Volunteers from Alabama, Louisiana, Michigan, Minnesota and Nevada, assisted by NAIC staff, drafted a potential comment letter. He said the draft comment letter has two main sections. A short section describes four consensus items. A much longer section includes documentation of the reasons for and against adding American Academy of Actuaries (Academy) membership as a requirement to become an Appointed Actuary. He emphasized that the purpose of the second section is not to debate whether membership should be a requirement but rather to do the best job possible to present both sides of the argument to the ad hoc group. The Task Force discussed the first section. Ms. Lederer said it would be helpful to provide some explanation about why the Task Force would propose to add “member of the Academy” when discussing the Academy’s Casualty Practice Council. She said membership in the Academy is a current requirement for those actuaries who need to be evaluated by the Academy’s Casualty Practice Council. The Task Force agreed. Mary Miller (Academy) agreed, saying the Academy has no authority to evaluate the qualifications of someone who is not a member. If the restrictions on a designation end up including the need to pass a specific exam, Ms. Lederer said the exams and exam content can change over time. Thus, it would be difficult to determine if the restriction would be met when exams were taken many years ago. After the Task Force discussed the intent of the grandfathering provision, Ms. Lederer agreed to propose revised wording to better reflect the discussed intent that actuaries who qualified under the current definition would remain qualified under the new definition. Ms. Miller suggested changing the wording from “four states” supported membership in the Academy as a requirement to “some states.” She said there was no formal vote. Mr. Piazza said he asked twice during the Jan. 8 conference call for any state to express support for explicitly recognizing Academy membership. He said only four states did so. He said that the wording that “four states” supported membership is factual and consistent with the Jan. 8 conference call minutes. Mr. Dyke said there is no formal vote and then not a formal act. He said it would not be accurate to assume the remainder of states are against requiring Academy membership. Mr. Will Davis agreed. Mr. Daniel Davis said some might have still been deciding. The Task Force decided to change “four” to “some.”
Attachment Four Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 2
The Task Force discussed the remainder of the comment letter. Mr. Dyke drafted the section to include Academy membership, and Mr. Stolyarov drafted the section not to include Academy membership. Both summarized the content in each section. Mr. Smith said it might be informative to identify the organizations that would meet the requirements in item (iv). Mr. Gennady and Mr. Daniel Davis agreed. Ms. Miller said the descriptor of the Academy being a “lobbying/advocacy” group should be revised. Mr. Stolyarov said he would change the description to “policy advisory” group and would evaluate other language to remove any reference to the Academy taking any position. Mr. Vigliaturo asked Ms. Miller to suggest any additional revisions prior to the Feb. 12 conference call. Mr. Vigliaturo asked the drafters to make revisions based on the discussion. He said the Task Force will consider adoption of the revised comment letter on its Feb. 12 conference call. Having no further business, the Casualty Actuarial and Statistical (C) Task Force adjourned. W:\National Meetings\2019\Spring\TF\CasAct\1-29 CASTF min.docx
Attachment Five Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 1
Draft: 1/29/19
Casualty Actuarial and Statistical (C) Task Force Conference Call January 8, 2019
The Casualty Actuarial and Statistical (C) Task Force met via conference call Jan. 8, 2019. The following Task Force members participated: James J. Donelon, Chair, represented by Rich Piazza (LA); Steve Kelly, Vice Chair, represented by Phillip Vigliaturo (MN); Lori K. Wing-Heier represented by Mike Ricker (AK); Jim L. Ridling represented by Daniel J. Davis and Jerry Workman (AL); Michael Conway represented by Deborah Batista, Mitchell Bronson and Sydney Sloan (CO); Paul Lombardo represented by Susan Andrews and Qing He (CT); David Altmaier represented by Howard Eagelfeld and Robert Lee (FL); Gordon I. Ito represented by Randy Jacobson (HI); Doug Ommen represented by Travis Grassel and Andria Seip (IA); Karin Zosel represented by Reid McClintock (IL); Ken Selzer represented by Nicole Boyd (KS); Eric A. Cioppa represented by Sandra Darby (ME); Patrick M. McPharlin represented by Kevin Dyke (MI); Chlora Lindley-Myers represented by Gina Clark and Julie Lederer (MO); Marlene Caride represented by Carl Sornson (NJ); Barbara D. Richardson represented by Gennady Stolyarov (NV); Maria T. Vullo represented by Sakman Luk (NY); Jillian Froment represented by Brad Schroer (OH); John D. Doak represented by Andrew Schallhorn (OK); Andrew Stolfi represented by David Dahl and Ying Liu (OR); Jessica Altman represented by Kevin Clark and Michael McKenney (PA); Kent Sullivan represented by J’ne Byckovski, Brock Childs, Nicole Elliott, Miriam Fisk, Eric Hintikka, Brian Ryder and Jennifer Wu (TX); and Mike Kreidler represented by Eric Slavich (WA). 1. Adopted the Statistical Reports
Via e-votes, the Task Force adopted all four statistical reports: 1) the Report on Profitability by Line by State (Profitability Report); 2) the Competition Database Report (Competition Report); 3) the Dwelling Fire, Homeowners Owner-Occupied, and Homeowners Tenant and Condominium/Cooperative Unit Owner’s Insurance Report (Homeowners Report); and 4) the Auto Insurance Database Report (Auto Report).
2. Discussed Revised Statement of Actuarial Opinion Instructions
Mr. Piazza said the Task Force should continue discussion from its Dec. 18, 2018, conference call regarding the revised Statement of Actuarial Opinion instructions jointly exposed by the Executive (EX) Committee’s ad hoc group and the Task Force. He said comments are due Feb. 15. He said the Task Force would decide whether to communicate to the ad hoc group on its proposed changes. He said a few of the items discussed on the prior call include: 1) whether to change the items in the list in the definition of qualified actuary, including whether to change item (i) to change the “or” to “and;” whether to note that the NAIC Accepted Actuarial Designations and the associate grandfathering clause should say “including restrictions”; and 3) whether to require membership in the American Academy of Actuaries (Academy). Mr. Piazza said the purpose of the conference call is for Task Force members to discuss the proposal. Mr. Piazza said membership in the Academy is not required to be a qualified actuary under current rules, except those actuaries who must be approved by the Academy’s Casualty Practice Council must be members of the Academy. He said that there is still the ability to be evaluated by the Academy’s Casualty Practice Council in the proposed instructions, but that wording is missing the requirement to be an Academy member. He suggested that membership requirement be reinserted into the proposed instructions when referring to the Casualty Practice Council’s process. Mr. Piazza asked if any Task Force member objected. No Task Force member objected. Mr. Stolyarov said Nevada maintains a list of commissioner-approved qualified actuaries for captives. He said there are some optional ways an actuary can qualify. He said in addition to other items in the review, the actuary must be a Fellow of the Casualty Actuarial Society (FCAS), a Fellow of the Society of Actuaries (FSA), a member in good standing of the Academy (MAAA) or a person who has otherwise demonstrated competence in the evaluation of loss reserves to the commissioner. Mr. Stolyarov said if membership in the Academy is specifically in the instructions, it should be an option. He said he supports the current wording in (iv) because it is consistent with the status quo and the law in Nevada. Mr. Dyke said he supports requiring membership in the Academy in the definition of a qualified actuary. He said that could be added to the wording in (iv) or completely replace that wording in (iv). He said not having membership in the Academy deviates from what is typically in model laws and what is in the qualified actuary definition for life and health actuarial opinions. He said the current instructions deviate from almost every other standard for development of model laws. Mr. Dyke said the joint
Attachment Five Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 2
qualified actuary task force tried to harmonize the qualified actuary definitions across annual statement blank types. He said membership in the Academy would harmonize the requirements. He said the federal government uses membership in the Academy in laws (e.g., the Affordable Care Act [ACA]). He said at least six states specifically require Academy membership by statute. He said the NAIC benefits from the Academy’s support, including issues regarding professionalism that occurred a few years ago, qualifications standards, etc. Mr. Dyke said membership in the Academy needs to be required. Mr. Piazza said he understands that state laws might have membership requirements, but the aim is to address what the instructions should say now. He said the aim is not to harmonize with the life and health instructions, although the actuarial groups tried to do that years ago and got nowhere. Mr. Piazza said he is not sure that membership in the Academy for the Appointed Actuary adds value to the instructions for purposes of the definition. Mr. Stolyarov agreed. He said the definition of a qualified actuary in the NAIC accreditation standards does not require membership in the Academy. He said differences or distinctions in the model laws might reflect differences in practice. He said the actuarial opinion instructions should not be more stringent than the model law. Mr. Dyke said the basic education pathway is different from being a licensing concept, just like getting education to be a doctor is separate from becoming licensed. He said membership in the Academy would not change the basic education concepts. Mr. Stolyarov said state law does not address general qualification standards for an actuary. Mr. Eagelfeld said the Academy performs a vital function and does good work, but he views the organization more like the American Medical Association (AMA) than a licensing board to practice medicine. He said if a requirement is to be a member of the Academy, then the Academy membership is enshrined as a necessity for a fully functioning Appointed Actuary. He said some people choose not to join the Academy for various reasons. Mr. Piazza asked if any Task Force member would like to see membership in the Academy as a requirement to be an Appointed Actuary. Alabama, Maine, Michigan, and Oregon supported the requirement. Ms. Lederer recused herself because the Missouri law requires membership. Mr. Piazza said another proposed change is for the item (i) to be an “and” instead of an “or.” Mr. Stolyarov and Ms. Lederer disagreed with making the proposed change. Mr. Piazza asked if any Task Force member objected. No Task Force member objected. Mr. Piazza said another proposed change is to add wording about appropriate restrictions in reference to the grandfathering provision. Mr. Stolyarov said that is consistent with the plan for the definition of NAIC Accepted Actuarial Designation to have noted restrictions. Mr. Piazza said the proposed change would affect both the item (ii) in the qualified actuary definition and the grandfathering wording in the NAIC Accepted Actuarial Designation definition. Mr. Piazza asked if any Task Force member objected. No Task Force member objected. Mr. Piazza said the actual restrictions will be insert upon completion of the NAIC’s Educational Standards and Assessment Project. No additional changes were proposed for discussion. Mr. Piazza asked if the Task Force wants to draft a letter to the Executive (EX) Committee’s ad hoc group. Mr. Gennady said a letter should address consensus changes. He suggested the areas of disagreement should be handled in individual or small group letters. Ms. Mottar said the letter could address consensus items and then explain what else was discussed. She said she supports a letter because the instructions are typically drafted by the Task Force and, therefore, the Task Force should weigh in on another group’s proposal. Ms. Elliott agreed with Ms. Mottar and said there could be explanation of both sides to an issue with a statement that consensus was not reached. The Task Force decided to include consensus items and both sides of the issue in the letter. Mr. Piazza asked if any Task Force member objected to a Task Force letter containing consensus items and both sides of the item where the Task Force could not reach consensus. No Task Force member objected. Volunteers agreed to draft a letter for Task Force consideration prior to the Feb. 15 comment deadline. Mary Miller (Academy) said she finds value in Academy membership as a qualification for signing opinions. She said that qualification standards provide for gaps in learning and that they do not have to be put in instructions. She asked for the Academy’s membership application to be shared with the Task Force. She said the Casualty Actuarial Society (CAS) and Society of Actuaries (SOA) are international organizations. If the Academy is not referenced in item (iv), then there is no way to know whether a member of the CAS in another country has knowledge of U.S. laws and regulations. She said the Academy membership form addresses that. She said she would send some of that information to the ad hoc group. She said there is confusion for a company to know whether the items in (iv) are met but very easy for the company and a board of directors to understand whether an actuary is a member of the Academy. She suggested the actuaries on the Task Force with only ratemaking responsibilities and experience might want to discuss the issues with members who have experience in financial regulation and do actuarial opinion analysis work. Having no further business, the Casualty Actuarial and Statistical (C) Task Force adjourned. W:\National Meetings\2019\Spring\TF\CasAct\1-8 CASTF min.docx
Attachment Six Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 1
Draft: 1/10/19
Casualty Actuarial and Statistical (C) Task Force Conference Call
December 18, 2018 The Casualty Actuarial and Statistical (C) Task Force met via conference call Dec. 18, 2018. The following Task Force members participated: James J. Donelon, Chair, represented by Rich Piazza (LA); Jessica Looman, Vice Chair, represented by Phillip Vigliaturo (MN); Lori K. Wing-Heier represented by Mike Ricker (AK); Jim L. Ridling represented by Daniel J. Davis and Jerry Workman (AL); Michael Conway represented by Deborah Batista, Mitchell Bronson and Sydney Sloan (CO); Paul Lombardo represented by Susan Gozzo Andrews and Qing He (CT); David Altmaier represented by Robert Lee (FL); Gordon I. Ito represented by Randy Jacobson (HI); Doug Ommen represented by Travis Grassel and Andria Seip (IA); Karin Zosel represented by Reid McClintock and Shannon Whalen (IL); Ken Selzer represented by Nicole Boyd (KS); Eric A. Cioppa represented by Sandra Darby (ME); Patrick M. McPharlin represented by Kevin Dyke (MI); Chlora Lindley-Myers represented by Gina Clark and Julie Lederer (MO); Marlene Caride represented by Carl Sornson (NJ); Barbara D. Richardson represented by Gennady Stolyarov (NV); Maria T. Vullo represented by Sakman Luk (NY); Jillian Froment represented by Brad Schroer (OH); John D. Doak represented by Andrew Schallhorn (OK); Andrew Stolfi represented by David Dahl and Ying Liu (OR); Jessica Altman represented by Kevin Clark and Michael McKenney (PA); Kent Sullivan represented by J’ne Byckovski, Brock Childs, Nicole Elliott, Miriam Fisk, Eric Hintikka, Brian Ryder and Jennifer Wu (TX); and Mike Kreidler represented by Eric Slavich (WA). 1. Discussed Revised Statement of Actuarial Opinion Instructions
Mr. Piazza said the Executive (EX) Committee’s ad hoc group and the Task Force jointly exposed revised Statement of Actuarial Opinion instructions on Dec. 15 for a 60-day public comment period ending Feb. 15 (Attachment ___). Mr. Piazza said the ad hoc group working on the definition of a qualified actuary is comprised of insurance commissioners from Alabama, Louisiana and Maine. Kris DeFrain (NAIC) presented the proposed changes and explained which changes were proposed by which group. Mr. Slavich asked about the definition of NAIC-Accepted Actuarial Designation. He said it appears that the organizations must meet all requirements to be accepted, with whatever restrictions are listed. Mr. Piazza said the Executive (EX) Committee ad hoc group is conducting a study and will determine which organizations are approved and if there are any restrictions. Mr. Stolyarov said he supports the proposed wording, including the new definition of the NAIC-Accepted Actuarial Designation and the proposed grandfathering approach. Mr. Dyke said the NAIC-Accepted Actuarial Designation seems to be adding complexity to the definition. He suggested identifying the procedure and explaining how other organizations can be accepted. He suggested the Task Force submit a comment letter as a group. He said there are different perspectives and that it might be helpful to see what a collective group of actuaries would say. He said it might add credibility and insight to the process. Mr. Piazza said it might be easier if individuals submit their comments separately. He said if there is a Task Force response, he would not draft the letter given his insurance commissioner is on the ad hoc group. Mr. Chou agreed with Mr. Dyke. Mr. Stolyarov said the exposure contains proposed wording from the Task Force, so it would seem unusual for the Task Force to draft a comment letter. He suggested individuals submit comment letters. He said differences in views should be addressed. He said he prefers the current wording of the definition of NAIC-Accepted Actuarial Designation and would not want that changed to describe the process to become an NAIC-Accepted Actuarial Designation. He said the process itself can be kept outside of the definitions. Mr. Dyke said the instructions can be changed annually, so it would not seem to be a problem to update the process as needed. Ms. Lederer asked whether the actuarial designations in the grandfathering statement are still under consideration. Mr. Piazza said the ad hoc group is currently reviewing the designations to determine which will be accepted. He said the designations listed in the grandfathering part would be the same as the accepted designations. Mr. Davis questioned the grandfathering of old exams that have been changed in order to meet the requirements. Mr. Stolyarov said the requirements of the standards will be known and that the actuaries will be held to having the combined experience and education that is needed. Mr. Davis suggested part 3 should be an “or” statement instead of an “and” statement when listing the ways someone would obtain knowledge. Mr. Piazza said more than half of the knowledge is probably learned outside of basic education. He said basic education will never be enough, alone, to provide enough education and knowledge for an actuary to provide a Statement of
Attachment Six Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 2
Actuarial Opinion in the annual statement. Mr. Davis agreed and said it would be more clear to use an “and” statement. Mr. Dyke said with basic education and experience working together, the new instructions might be more complex than needed. Mr. Dyke asked for an example of restrictions on the actuarial designations. Mr. Piazza said restrictions might be that an Associate of the Casualty Actuarial Society (CAS) must also have passed the CAS Exam 7, which covers advanced reserving topics. Ms. DeFrain said the Fellow of the Society of Actuaries (SOA) might have a restriction that the general insurance track must be passed. Mr. Dyke said that given the complexity of the definition, especially with the restrictions on the actuarial designations, it might be worthwhile to consider making the whole definition simpler by requiring membership in the American Academy of Actuaries (Academy). He said that the Academy designation is a generally accepted designation in NAIC model laws and other laws, generally accepted by the federal government, and that five or six states require membership in the Academy. He said it is a simple objective measure of membership. Mr. Stoylarov said he disagrees with requiring membership in the Academy. He said a non-Academy member must meet all the requirements in Item 4 within the Qualified Actuary definition, essentially addressing the substance of what an Academy membership would achieve. He said not being an Academy member does not exempt someone from those requirements. The only things missing is paying dues to the Academy and getting services such as webinars, publications and networking activities. He said they are good services, but they are not essential to the work of the Appointed Actuary. Mr. Slavich asked about the restrictions to the designations and whether those same restrictions would apply in the grandfathering provisions. Mr. Piazza said it would be the same in both places. Mr. Slavich suggested that the grandfathering sentence should include “with noted restrictions” or similar wording. Mary Miller (Risk & Regulatory Consulting) said that the qualified actuary discussion is not a Task Force definition and that the Task Force has not had a full discussion of the issues that surround the definition. She said it was previously an NAIC position that the education provided by the SOA general insurance (GI) track was not sufficient to be included, yet now the NAIC would be grandfathering those. Ms. Miller said the qualification standards provide for a mechanism for someone who had deficiencies in basic education to get that basic education through experience and to have that documented. She said the need for grandfathering can get into all sorts of permutations that are not necessary because the qualifications already provide a mechanism to address the deficiencies. On behalf of the Academy, she said there are people who do not understand the history and that the CAS and SOA created the Academy more than 50 years ago to address public policy issues and be a single place for state insurance regulators to go. She said the original draft of property/casualty (P/C) instructions, when they were first being created, required the Academy membership. She said that at the time, there were issues with non-casualty actuaries signing California workers’ compensation opinions, so they decided to require the CAS membership instead of the Academy membership. She said 30 years ago, the CAS was not as international as it is today. As well, when the SOA announced their general insurance track, the SOA said it was for their international students to have a full array of topics in their examinations. She said she does not want to see the definition as a passport to international actuaries who do not understand U.S. laws and regulations and did not pass an examination on U.S. laws and regulations. She said the Academy is the only organization that requires someone requesting membership to demonstrate familiarity with U.S. laws and regulations if they are a non-resident or a resident alien of fewer than three years. She said she is not sure who made the recommendations to the Executive (EX) Committee, but she hopes they looked in the mirror and made sure they met the qualification standards for giving that opinion to the ad hoc group of insurance commissioners. She said they were being relied on in their role as actuaries. She said the initial exposure of the definition included a requirement to be a member of the Academy, and neither the CAS nor SOA objected to such. She said it is a disservice to not include membership of the Academy. As a former state insurance regulator, Ms. Miller said insurance commissioners have always had the power to allow someone who is not a qualified actuary to sign an opinion. She said the revised instructions make that more prominent and could encourage that. She said it does not serve the public good and creates a lower standard. Ralph Blanchard (Travelers) said he cannot find “experience period” anywhere other than in the qualification standards and asked whether the NAIC would be asking those standards to be modified. Mr. Piazza said the experience period has been discussed. He said the NAIC would not be asking the Academy to change the qualification standards. He said the combined basic education, experience and continuing education for an appointed actuary would need to be sufficient for a given line of business or company structure. He said at one point, the drafts went beyond relying on the qualification standard, but that was taken out. Mr. Blanchard asked whether there is modification to the U.S. qualifications experience period requirement. Mr. Piazza said there is no suggestion for the qualification standard to be revised in that regard.
Attachment Six Casualty Actuarial and Statistical (C) Task Force
4/6/19
© 2019 National Association of Insurance Commissioners 3
Mr. Blanchard said that the wording of “being subject to the ABCD” needs to be tweaked given the Actuarial Board for Counseling and Discipline (ABCD) only provides recommendations to the other actuarial organization to decide the action to take, if any. Craig Hanna (Academy) asked if there was formal action by the Executive (EX) Committee. Mr. Piazza said the Executive (EX) Committee’s ad hoc group has exposed the document for a public comment period. He said the ad hoc group will make a recommendation to the Executive (EX) Committee. Ms. DeFrain said there would be a public hearing March 22 via conference call. Mr. Dyke asked if the Executive (EX) Committee would have any exposure of the proposal. Ms. DeFrain said the current project timeline does not include another exposure period. Mr. Dyke asked about impediments to the Task Force issuing a comment letter. Mr. Piazza said Task Force members could draft a letter, although he would not be the main drafter given his role with the ad hoc group. Mr. Dyke asked Task Force members to contact him to draft a group comment letter. Mr. Piazza said there would be time on the Task Force’s Jan. 8, 2019, call to discuss a draft comment letter or to attempt to reach some consensus for a Task Force comment letter. Having no further business, the Casualty Actuarial and Statistical (C) Task Force adjourned. W:\National Meetings\2019\Spring\TF\CasAct\12-18 CASTF min.docx