Ethical Guidelines and Best Practices for Institutional ... · Designing the Study Research Question The question is the purpose of the study. Everything hinges on the right question

EEtthhiiccaall GGuuiiddeelliinneess aanndd BBeesstt PPrraaccttiicceess ffoorr IInnssttiittuuttiioonnaall RReesseeaarrcchh iinn tthhee WWiissccoonnssiinn TTeecchhnniiccaall CCoolllleeggee SSyysstteemm

December 2004 Written by The Research, Planning and Development Ethics Advisory Team

Viktor Brenner Lori Rasmussen Tanya Wasmer Don Weimer

Table of Contents Chapter I: Introduction.................................................................................................................... 1

Mission and Purpose ................................................................................................................... 1

Chapter II: Reliability and Validity of Research Studies................................................................ 2

Designing the Study.................................................................................................................... 2 Reporting Results........................................................................................................................ 4

Chapter III: Liability Management ................................................................................................. 5

Research policy........................................................................................................................... 5 Table 1: Pros and Cons of Different Methods of Implementing an IRB................................ 7

IRB Operating Procedures .......................................................................................................... 8 Special situations ...................................................................................................................... 11 Consent procedures and documentations.................................................................................. 12

Chapter IV: The Legal Angle ....................................................................................................... 15

State, Federal, local guidelines ................................................................................................. 15 “Telemarketing Laws” .............................................................................................................. 20 Managing Support Staff............................................................................................................ 21 Data Storage and Security Guidelines ...................................................................................... 21

Appendix A: Review of Reliability and Validity Issues............................................................... 23

Reliability.................................................................................................................................. 23 Validity ..................................................................................................................................... 24

Appendix B: Individuals New to Institutional Research .............................................................. 26

Statistical Analysis.................................................................................................................... 26 Research.................................................................................................................................... 27 Supervisory Skills ..................................................................................................................... 28 Information Technology Skills ................................................................................................. 28 Communication Skills............................................................................................................... 29 Legal and Ethical Issues............................................................................................................ 29

Appendix C: Sample IRB Form and Procedure from Moraine Park (2002)1 Appendix D: Sample Records Retention Schedule from Lakeshore2

1 Thanks to Josh Bulloch for this document

2 Thanks to Tanya Wasmer for this document

WTCS Research, Planning and Development Research and Ethics Handbook 1

Chapter I: Introduction Mission and Purpose

The Ethics Advisory Team3 was formed to document model policies and identify best practices for ethical institutional research and the management of liability in the Wisconsin Technical College System. The Ethics Advisory Team was a subcommittee of the Research, Planning and Development (RPD) Subcommittee of the WTCS President’s Association. Its members include institutional researchers from all 16 WTCS districts. This primary intent of this manual is to provide WTCS institutional researchers with a resource to turn to when ethical questions arise, and as a model of best practices to improve the quality of institutional research conducted in each district. The Ethics Advisory Team is also mindful that other staff and/or external entities consult with RPD members because we are identified as the “research experts” in the technical college system. Thus, the secondary intent of this manual is to provide general guidance on research with human participants in the technical college system, so that when RPD members are called in this capacity they can (1) provide informed advice (2) to protect human participants in such research, (3) reducing the potential liability risk of the technical colleges as well as (4) reducing the potential for personal liability in our roles as research advisors. The primary ethical authority for our profession is the Code of Ethics for Institutional Research published by the Association for Institutional Research. The ethical and best practice recommendations of this document are adjuncts to the Code as they apply to the Wisconsin Technical College System; if in any way these recommendations are inconsistent with the Code, upholding the Code is the researcher’s primary responsibility. If you as a user of this document are not familiar with the Code in its most current form, you should familiarize yourself with the Code before proceeding. You can view the code online at

http://www.airweb.org/images/2002codeofethics.pdf This document is divided into four sections (this being the first) plus several appendices. In Chapter II, practices for maximizing the reliability and validity of studies are discussed. Readers may also be interested in Appendix A, which provides a review of reliability and validity issues as they apply institutional research. Chapter III addresses liability questions and the ethical treatment of human participants in research. Chapter IV focuses on legal issues that affect our work. Appendix B offers advice to individuals new to institutional research.

3 The Ethics Advisory Team consisted of Viktor Brenner (Chair), WCTC; Lori Rasmussen, NWTC; Tanya Wasmer,

first LTC and later FVTC, and Don Weimer, MATC (Milwaukee). Questions and comments regarding this document can be addressed the committee chair at [email protected]

http://www.airweb.org/images/2002codeofethics.pdf


Chapter II: Reliability and Validity of Research Studies The methodology of the research study is the foundation of ethical research. Unvalidated instruments, incorrect or insufficient statistical analysis, and/or inappropriate conclusions can be easily mistaken for valid research by legislators, administrators, and the general public. Too often research instruments and the results from them are received without question or are criticized in hope to discredit them because they are inconsistent with someone’s personal agenda. When every research conclusion can be countered by an (apparently) equal and opposite conclusion, no conclusion is credible. It is our responsibility as institutional researchers to provide accurate and unbiased research, to ensure that policy decisions are not made on the basis of erroneous information. When information is represented as authoritative when the institutional researcher knew or should have known that it was not, it may also indicate negligence. Designing the Study Research Question The question is the purpose of the study. Everything hinges on the right question. It is imperative that the question is clearly stated. The question creates a domino effect to who should be involved in designing the study, to who are the subjects (the sample), to what type of study is most appropriate: should it be a survey, focus groups, statistical analysis of existing data, to finally the results. A poorly specified question compromises all aspects of the research study. Avoidance of conflicts of interest Researchers should not participate in the review of projects where he or she has a vested interest. By definition, a conflict of interest prevents the researcher from maintaining the necessary objectivity. A researcher should excuse him or herself from influential discussions regarding projects where they are integrally involved or stand to personally benefit. Examples of conflict of interest include:

•

•

•

Evaluating the likelihood for success of a new program that one has invested significant personal time in developing Evaluating the best vendor to receive a large contract when the researcher has a financial stake in one of the bidders Evaluating the impact of program discontinuation when one’s spouse is an instructor in that program

There may be times where a conflict of interest may be unavoidable because there is no one else able to do a certain job. In these cases, it is important to (1) take any steps available to reduce the conflict of interest and reduce partiality, and (2) clearly identify the presence and nature of the conflict of interest in reporting the results of the study. Subjects The subjects and number of them are based on the question. The anonymity and confidentiality of participants should be protected. Do not request or keep identifying information, such as social security number, name, etc., on the same record where the subject’s respective study information will be maintained. Also, be aware that a participant’s identity can unwittingly be identified indirectly through combinations of information (see

, page 16). FERPA


Potential respondents should not be bombarded by multiple requests for information—respondents tend to not respond to any survey if they receive more than one from the same organization. It is simply a matter of planning to avoid doing this within the institutional research department, but it can be a problem when different entities with the college independently survey the same population. If possible, research departments should position themselves as survey gatekeepers for the college, such that departments can coordinate research efforts. In order for this to work, the institutional research office must be willing to cooperate with individual departments;4 simply acting as a gatekeeper will only undermine your departmental position. Study Type The study type is determined by the question, subjects, and the number of subjects. Question Design Take care not to have leading questions that would bias the results to be positive or negative to meet an agenda that is a conflict of interest with the study itself. When writing closed-ended questions, be as specific as possible so that there is little room for interpretation as to what is meant by the question. When writing open-ended questions, question should be more broad; narrowing open-ended questions restricts the range of responses received. For instance, on a recent graduate follow-up different departments were given a chance to edit the wording of open-ended questions. One department got much more insightful information than another because of the narrowness of one’s question:

Narrow feedback received: “While attending WCTC, what was your best resource/support at the college?” More useful feedback received: “What do you feel were the strengths of the program?”

Another basic rule of writing questions is this: one topic per question. When you write a question that asks for more than one different thing, respondents get confused. Consider the following examples from graduate follow-up survey forms:

OK: “Do you have any recommendations to improve our programs and student services (registration, advising, etc). Although it asks about several domains, it only asks for one thing: suggestions for improvement. Mixed: “Please tell us how WCTC has helped you and give us suggestions for improving our student services and programs.” This asks for (1) positive experience stories and (2) suggestions for feedback at the same time. It would have been better (were space available) to split these two thoughts into two separate questions.

4 For instance, at WCTC Surgical Technology graduates tended to not respond to the state graduate follow-up survey

because they also received one from their department—which was required in order for the department to maintain accreditation for the program and which, consequently, students knew to expect beforehand. To solve this, a modified version of the graduate follow-up survey was designed that would combined elements required by the state with those required by the accrediting body, so both purposes are now served with a single survey.


Open-ended Questions If the study solicits comments from participants, be prepared for the possibility that non-participants may be identified as part of those comments. Have a system within your organization for handling sensitive information while being aware that the results are public information. Writing Instructions The importance of writing good instructions is one of the most overlooked aspects of good design. Instructions need to (1) clearly instruct subjects what you want them to do and (2) clearly identify what will be done with the data, while at the same time (3) being short and simple enough that participants will actually read them. Confusing or vague instructions can reduce response rates. The instructions form an implicit contract with respondents, spelling out what they will be used for. Using responses for some purpose that was not explicitly laid out in the instructions—for instance, calling selected responders when they were not told up front that this might happen—is often unethical and/or potential grounds for litigation. Pilot Testing We advocate pilot testing when possible. An instrument might be given to a small sample of participants, who can then give the researcher feedback as to questions that were confusing, content domains that might not have been tapped, or other ways in which the survey can be improved. Reporting Results When reporting results, institutional researchers find themselves pulled in two different directions. On the one hand, the researcher has a responsibility to report on the limitations of the study and the generalizability of its findings, including factors like how the data was obtained, how the instrument was designed, description of the subjects studied and how, what validity or reliability data are available, what analyses were performed, etc. On the other hand, parts of the intended audience may not have much research sophistication, and too much detail could lead them to ignore the study completely. Documents produced for public use are especially problematic; in some cases, descriptions of the limitations of a study, taken out of context by unsophisticated readers, could do more harm than good. As a general rule, public-use documents should focus on the interpreted findings of a study and not on methodological issues. Internal-use documents should include the necessary discussion of limitations, but should not read like a scientific journal. Some approaches that have been used to balance these opposing considerations include

• • •

Discussing limitations or methodological issues in footnotes, endnotes, or Appendixes Writing separate technical and non-technical methods and discussion sections Writing a separate, broad overview of findings drawn from a more technical source document


Chapter III: Liability Management One of the primary reasons for the creation of this manual is to assist WTCS member colleges in managing certain specific liability risks associated with institutional and other research. This is a new area of concern to technical colleges because until recently research in our settings was very rare. Today, technical colleges need to have formal research approval procedures in place to protect our students from harm and our schools from liability. Research policy The first step for any technical college is to adopt a formal policy related to research occurring on campus; some already have this, but many do not. It is recommended that this policy codifies the college’s commitment to maintaining the rights of research participants according to the established standards of federal policy 45 CFR 465. This policy includes procedures for creating and operating an Institutional Review Board (IRB) to review research projects. It is neither feasible nor necessary for most technical colleges to obtain official recognition by the Department of Health and Human Services (see Table 1), but it is possible to formalize the application of the standard operating procedures of an IRB. The more faithful a college policy is to the federal regulations, the greater the protection from potential liability. Institutional Review Boards At research-oriented colleges and universities, the standard of practice is to have an Institutional Review Board (IRB) review and approve all research projects. The phrase “standard of practice” is an important one because of its central role in legal definitions of negligence. To be found liable for negligence, it must be proven that 1) a party has been injured, 2) the liable party held a duty towards the injured party, and 3) the liable party “breached that duty by failing to conform to the required standard of conduct” (West’s Encyclopedia of American Law, 1998).6 Of the three, it is the third that is usually the most difficult to prove. However, one standard for proving negligence is to demonstrate that the “usual and customary conduct or practice of others under similar circumstances” (West’s Encyclopedia of American Law, 1998) was not being followed. Because IRBs define the standard of practice in reviewing research, a college would have little defense against a claim of negligence should something occur in a research project that was not pre-approved by an IRB. The rules governing the operation of an IRB can be adopted relatively straightforwardly. However, there are very specific rules regarding the composition of the IRB that are more difficult to adopt. The DHHS requires that the IRB include members that are not associated with the college in any other way, as well as experts in certain areas such as law and the subject matter of the study.7 Any WTCS college could go through the steps of creating a formal IRB and maintaining a DHSS assurance, however, the expense of this process can probably not be justified in relation to the types of projects likely to need evaluation. Fortunately, formal IRB 5 http://ohrp.osophs.dhhs.gov/humansubjects/guidance/45cfr46.htm

6 West’s Encyclopedia of American Law (2nd ed.). (1998). Minneapolis/St. Paul MN: West Publishing.

7 These requirements are just one place where the medical orientation of the DHHS regulations is clear—they are

intended to avoid conflicts of interest arising from situations where the institution stands to gain financially from sponsored medical research


registration is not mandatory; 45 CFR 46 is limited in its legislative authority to organizations that receive research support (not training support) from any of the branches of the Department of Health and Human Services (DHHS). Because primary research is not part of its mission, technical colleges are not likely to receive such funding, and therefore are not legally bound to the regulations—which is why technical colleges have not had need to create IRBs previously. Although 45 CFR 46 technically only pertains to federally funded research, in most cases the assurance that an IRB files with the DHHS specifies that any project involving human participants that takes place at that institution will be reviewed and approved by the IRB. A research-oriented institution is able to offset the operational costs incurred by this policy through their retention of indirect cost percentages from grants received. Two-year colleges are not likely to have this source of revenue and would have to support the IRB from operational funds. However, a technical college’s goal is not to become competitive for research grants but rather to protect itself from potential liability and exert control over activities that take place on campus. The recommendation therefore is to adopt a policy that adopts the review procedures established in 45 CFR 46 but assigns responsibility for that review to some entity other than a formally registered IRB. The rationale for this proposal is twofold. First, following 45 CFR 46 procedures strives to maintain the expected standards of research practice. It is unknown whether a court would accept the argument than an unregistered IRB still attains the expected standard of practice, but clearly it is better than doing nothing. Second and perhaps more importantly, by implementing standard review procedures the institution can avoid any unfortunate events that would land them in court in the first place. Most of the potential harms that could conceivably occur in a technical college research project can be easily avoided with some informed advice on how a study should be designed. Indeed, the greatest risk for technical colleges lies in the overall lack of research sophistication among potential principal investigators. There are several ways that a technical college could go about implementing 45 CFR 46 procedures. Table 1 describes the advantages and disadvantages of five key approaches to implementing research review procedures as identified by RPD members. For most schools, the “local IRB” approach is likely to be the most practical and cost-effective compromise. Depending on size and proximity to neighboring colleges, some members may wish to share a local IRB. One consideration here is that recent lawsuits have attempted to broaden the responsibility for damages not only to host institutions but to individual IRB members as well. Although to date complaints against individual IRB members have not proceeded to court, many now recommend that IRB members carry individual liability insurance. Should this legal tactic eventually prove successful, a shared institution could conceivably be held co-liable should something untoward happen to an institution with which it shares an IRB. There are also independent, formal IRBs that can be retained for the purpose of reviewing research. This approach is likely to be cost-prohibitive, however, especially when compared to the risks associated with the sorts of non-medical research that one might find at a technical college. Another issue is that third-party IRBs will usually require the cooperating institution sign an agreement that indemnifies the IRB from liability. Whether or not such an agreement is


actually legally enforceable has not yet been tested in court, but there is at least the possibility that a technical college using a third-party IRB could be held financially liable for a decision made by someone else.

Table 1: Pros and Cons of Different Methods of Implementing an IRB Formal

(Registered) IRB

Local IRB Shared IRB Third-party (Registered)

IRB

Compliance officer

Liability protection Maximum; this is the standard

Can have good procedures, but not registering increases liability risk

Same as local IRB

More, but will demand IRB indemnification in case something goes awry

Minimal if any; below the expected standard of practice

Cost Time of staff; travel and expenses/ stipends for community members

Time, not money Time, plus some travel

Potentially Prohibitive

Time, not money

Potential for conflict of interest/pressure

Least/Distributed Distributed, but all members part of a single organization

Distributed; may lead to inequity of effort between colleges

Limited to pressure to retain contract

Maximum

Workload Shared within college

Shared within college

Shared between colleges, hence more work than single-college IRB

Minimal; liaison with IRB

Potentially overwhelming for one person

Difficulty of formation

Very: must apply with DHHS and file semi-annual assurances

Limited to identification of appropriate members

Additional requirement of coordination between colleges (and missions/ agendas)

Formal bid process

Little

Committee membership

Fixed rules make this difficult to fill (see page 5)

Existing staff Existing staff None One person

Committee size (external/internal)

Fixed by regulations (state minimum)

Can be smaller because college sets own standards (internal)

Might be smaller, depending on desire for representation by each college (all internal)

Fixed by regulations (all external)

One (internal)

Continuity in case of member departure

Existing members can mentor new members while maintaining operations; training external members would be a more intensive process

Existing committee members can mentor new ones and maintain operations

Existing committee members can mentor new ones and maintain operations

External concern; contract bidding may lead to changes in vendor/procedures from year to year

Operations would be suspended until new individual learns procedures


Response Time Fixed schedule Fixed schedule; may be easier to schedule more frequently as all staff are on-site

Fixed schedule, but may have to meet less frequently on account of travel

Outside of college control

Variable dependent on individual’s workload

Ease of gaining project approvals

More difficult: non-scientifically trained members may require more explanation

Routine Routine, but may involve additional discussions to understand local practices

Varies Easiest

Institutional Control

Full Full Half None; external group is autonomous

Full

Sufficient for DHHS Research Funding?

Yes No No Yes No

RPD Recommendation

Ideal, but not practical for most

Recommended as the best trade-off of protection and cost for most TC

Recommended alternative for small colleges with proximate neighbors

Cost- and time-prohibitive, although it might be used on indi-vidual projects if funding is involved

Not recommended; will not achieve goal of liability protection

The final approach, the “research compliance officer” approach, is not recommended and should only be implemented if nothing else is feasible. The committee was concerned that this latter approach would not sufficiently approximate the standard IRB procedures to provide any protection from claims of negligence; any single person is too susceptible to external pressure, conflicts of interest, or administrative error to really ensure maximum protection to all research participants. However, the research compliance officer approach would still be preferable to having no policy at all, since he or she could help ensure that human protections are built-in to research proposals.8 An example of an IRB Policy and form from Moraine Park can be founding Appendix C. IRB Operating Procedures IRBs have two major functions: (1) to conduct a cost-benefit analysis to evaluate the potential risks to participants compared with the benefits obtainable from the research, and (2) to ensure that participants are able to freely choose whether or not to participate in the research. There are three types of IRB review, although in practice all involve submitting a formal proposal to the IRB. Which type of review is used for a particular protocol is largely determined by the amount and type of risk associated with it. 8 If a college were to adopt a policy of only permitting exempt research projects (MATC-Milwaukee has had such a policy in the past, for instance), then a research compliance office would be sufficient because the only question is whether or not established exemption criteria are met. However, the committee does not recommend restricting research to exempt protocols only, since few protocols qualify for exemption—no Institutional Research projects would qualify for exemption, for instance, because of the use of tracking numbers. Conversely, a policy that restricts research to exempt or expedited projects only—in other words, minimal risk project only—might be an attractive option for some colleges.


There are many kinds of risks that a research participant may be exposed to. Risks are frequently grouped according to certain categories, namely:

•

•

•

•

•

•

Physical risk: physical discomfort, pain, injury, illness or disease brought about by the methods and procedures of the research. Psychological risk: alteration of behavior or the production of negative affective states such as anxiety, depression, guilt, shock, or loss of self-esteem. Social risk: alterations in relationships with others that are to the disadvantage of the participant, including embarrassment, loss of respect of others, or labeling a participant in a way that will have negative consequences. Economic risk: includes payments for procedures not otherwise required, loss of wages or other income, and any other financial costs, such as damage to a participant’s employability. Legal risk: the participant or others will be liable for a violation of the law, either by revealing past actions or requiring future actions that may be criminally or civilly liable. Loss of confidentiality: participants have the right to be protected against invasions of their privacy and to preservation of their personal dignity.

Regardless of the type of risk, however, is level of risk, and here the key concept is minimal risk. Minimal risk is defined as risk whose magnitude and probability are equivalent to that which would be experienced in everyday life, such as those an individual might experience had they chosen to not take part in the research. Research projects that involve no more than minimal risk may qualify for IRB exemption or expedited review. If more than minimal risk is present, researchers must be able to justify the risk by the obtainable benefits through the full review process. IRB Exemption Review In certain specific cases, formal IRB approval is not required when a project involves minimal or indirect human participation. However, proposals must be submitted to the IRB to ensure that exemption criteria are met. Exempt protocols will often also be exempt from the requirements documenting informed consent; participation is generally sufficient evidence of consent for exempt protocols. To qualify for exemption, a project must (1) involve no more than minimal risk and (2) fit one the following federally-defined exemption categories.

1. Research (a) conducted in established or commonly accepted educational settings, involving normal educational practices, such as research on regular and special education instructional strategies, or (b) on the effectiveness or the comparison among instructional techniques, curricula, or classroom management methods. This exemption applies to mentally handicapped participants only if the research involves no changes in the normal content, location, or procedures of instruction.

2. Research involving the use of educational tests (cognitive, diagnostic, aptitude,

achievement); survey procedures; interview procedures; or observation of public behavior is exempt if and only if

a. The information obtained is recorded in such a manner that participants cannot be identified directly or indirectly (i.e. through identifiers that can link the information to the participants), and


b. Disclosure of the participant’s responses outside of the research could reasonably place them at risk of criminal or civil liability or be damaging to the participant’s financial standing, employability, or reputation.

Caveats c. If children or mentally handicapped individuals are being studied, educational

testing is exempt but survey, interview, or observation procedures are not. d. Research is not exempt if the subjects are public officials or candidates for public

office. e. Research is not exempt if federal regulations (such as FERPA) require absolute

confidentiality.

3. Research involving the collection or study of existing data, documents, records, pathological specimens, or diagnostic specimens is exempt if

a. The sources of the information are publicly available, or b. The investigator in such a manner that subjects cannot be identified, directly or

through identifiers linked to the information. Caveat

c. Data that is collected for a routine purpose with the expectation of also using it for research purposes is not archival data and does not qualify for exemption.

4. Research conducted by the college designed to study, evaluate, or otherwise investigate

a. Public benefit or service programs. b. Procedures for obtaining benefits or services under those programs. c. Possible changes in or alternatives to those programs or procedures. d. Possible changes in methods or levels of payment for benefits or services under

those programs.

5. Taste and food quality evaluation and consumer acceptance studies are exempt if a. Wholesome foods without additives are consumed or b. The food does not contain any ingredients or contaminants that have not been

found to be safe by the FDA, EPA, or Dept. of Agriculture at or above the levels present in the food.

Expedited Review Expedited review is conducted by the chair of the IRB and does not require participation of the full IRB. However, the standard requirements for informed consent (or its waiver) apply regardless of the type of review. The expedited review procedure can be used to evaluate projects that (1) involve no more than minimal risk to participants and (2) fall into one of the following federally-defined categories:

1. Clinical studies of drugs and medical devices if and only if a. The study does not involve an investigational drug requiring a new drug

application. b. The study does not utilize medical devices for which an investigational device

exemption application is required and does not utilized cleared/approved medical devices in any way other their cleared/approved labeling and procedures.


2. Collection of blood samples by finger stick, heel stick, ear stick, or venipuncture in

accordance with limitations defined in the Federal Register (c.f. http://ohrp.osophs.dhhs.gov/humansubjects/guidance/expedited98.htm)

3. Prospective collection of biological specimens (such as hair and nail clipping, teeth, excreta and external secretions, mucosal or skin cells, etc) for research purposes by noninvasive means.

4. Collection of data through noninvasive procedures (not involving general anesthesia or sedation) routinely employed in clinical practice, excluding procedures involving x-rays or microwaves (e.g. weighing; testing sensory acuity; or electrocardiography, electroencephalography, or ultrasound testing).

5. Research involving data, documents, records, or specimens that have been collected, or

will be collected solely for non-research purposes (such as medical treatment or diagnosis) and does not qualify for IRB exempt.

6. Collection of data from voice, video, digital, or image recordings made for research

purposes. Caveat: recordings usually require an additional consent form or additional elements added to the basic consent form

7. Research on individual or group characteristics or behavior (such as studies of

perception, cognition, game theory, or test development) or using survey, interview, oral history, focus group, program evaluation, human factor evaluation, or quality assurance methodologies that, where the investigator does not manipulate participants’ behavior, does not involve stress to participants, and does not qualify for IRB exemption.

Full Review Any proposal involving more than minimal risk of psychological, physical, socio-economic (including damage to reputation or employability), or legal risk or that involves more than minimal risk of loss of confidentiality must be reviewed by the entire IRB. Special situations Special populations Looking at the criteria for HSRC exemption, you may notice that there are caveats when research participants are children or mentally retarded. These are two of four groups identified as special populations, together with prisoners and pregnant women. In all four cases, these populations warrant special treatment because, for various reasons, these populations may be unable to provide informed consent to participate. As a result, not only are there changes to the consent procedures, frequently there must be changes to procedures as well. Since it is not anticipated that technical college studies will involve special populations—especially since minors can be treated as adults if they are college students and involved in research involving no more than minimal risk—those with further interest in special populations are referred to the appropriate sections of 45 CFR 46.


Institutional Research Since the primary purpose of most IR projects is to assess outcomes for a specific institution (i.e. job placement of graduates) rather than producing generalizable knowledge, many IRBs consider IR projects to be outside of their purview. However, this does not mean that Institutional Researchers can safely operate in ignorance of IRBs and their functions. IR projects can be designed in ways that will allow indirect identification of respondents, for instance, and if a participant sues your college you will not be protected by the fact that the federal definition was arguably not met. Furthermore, in the IR context such disclosures probably also constitute a FERPA violation, which has its own set of consequences. Thus, while it might not be necessary to have an IRB approve every graduate follow-up survey, IR projects should be conducted in line with DHHS expectations in order to reduce liability. Students as principle investigators Students may be principle investigators more frequently than one might expect; one technical college has marketing class where students are required to perform a mock market analysis survey. It is recommended that the procedures for approval of research proposals require that when a student is the principle investigator, a member of the faculty or staff must champion the proposal and co-sponsor the proposal. Technical college students are not expected to be familiar with the intricacies of participant protection; by requiring faculty sponsorship, the faculty member can provide the guidance that student investigators need in making their proposals meet approval criteria. Third parties Technical colleges should consider whether or not they want to permit third parties, especially for-profit entities, to utilize the college as a research resource. A technical college in California witnessed an impromptu student march on the president’s office after a professor had required student to take certain clerical skills-type tests in order to assist an external corporation norm a new test. Some of the problems with this scenario include (1) failure to obtain informed consent and to offer students the right to withdraw consent (2) failure to disclose how he or she stood to gain from the students’ participation, and (3) inappropriate utilization of student-purchased class time for non-instructional activities. Had there been an IRB at this institution, this project would never have been approved. However, the introduction of third parties—especially for-profit organizations making use of non-profit government facilities—is a situation with a high potential for abuse. It is recommended that, at a minimum, the procedure for obtaining IRB approval require that any third parties involved with a research proposal be identified along with a detailed description of any ways that the principal investigator stands to profit personally from the investigation. Consent procedures and documentations

Every research participant must give free, informed consent to participate in research. Informed consent means that an individual (or his or her legal guardian) freely chooses to participate in the research.

1. Informed means that the individual is making this decision based on an understanding of the risks and benefits involved in participation. To be informed, participants must be advised in advance of all of the anticipated costs, benefits, and alternatives (if applicable) to participation.


2. Freedom of choice means that the individual consents to participate without coercion, and has the right to withdraw that consent at any time without penalty. Participants may receive compensation or other benefits from participation, but cannot be denied benefits to which they would ordinarily be entitled as a means of obtaining consent.

Researchers have a responsibility to document that all participants have freely chosen to take part in the research. Participants must be provided with a consent form that documents the expectations of participation, the anticipated risks and benefits, and all of the other information that an individual would need to make an informed decision to participate. A consent form should include:

1. An explanation of the purposes of the research, the expected duration of the subject's participation, and a description of the procedures to be followed.

2. A description of any reasonably foreseeable risks or discomforts to the subject. 3. A description of any benefits to the subject or to others which may reasonably be

expected from the research. 4. A statement describing the extent to which confidentiality of records identifying the

subject will be maintained. 5. For research involving more than minimal risk, an explanation as to whether any

compensation and an explanation as to whether any medical treatments are available if injury occurs, what they consist of, or where further information may be obtained.

6. Contact information (usually the principle investigator) explaining whom to contact for answers to questions about the research and research subjects' rights, or with regard to a research-related injury.

7. A statement that participation is voluntary, refusal to participate will involve no penalty or loss of benefits to which the subject is otherwise entitled, and the subject may discontinue participation at any time without penalty or loss of benefits to which the subject is otherwise entitled.

In order to be truly free consent, participants must be given the option of withdrawing their consent at any time without penalty—this is one of the key omissions one can expect when research projects are not reviewed by an IRB. A consent form also cannot indemnify an investigator or the college from any liability related to participation in a study; an investigator is responsible for any harm or injury that comes to participants, whether foreseen or not (see point 6 above). Since injury may occur after the fact and the contact information is found on the consent form, usual procedures are to provide the participant with a copy of the consent form they have signed. Since photocopying is usually impractical in the middle of the study, many investigators have participants sign two copies of the consent form and keep one of them. Studies that utilize minors or other individuals for whom legally binding consent must be given by a legal guardian usually have the actual participant sign a parallel assent form. For the purposes of research that involves no more than minimal risk, a college student may provide consent for him or herself regardless of chronological age. This provision would not apply to dual enrollment students. An investigator may request waiver of documentation of consent for a particular project. A waiver of documentation of consent may be approved if


1. The documentation of consent is the only record linking the subject with the research and the principal risk would be potential harm resulting from a breach of confidentiality, or

2. The research presents no more than minimal risk of harm to subjects and involves no procedures for which written consent is normally required outside of the research context.

In unusual circumstances, an investigator may request a waiver of consent. The IRB may approve a consent procedure which does not include, or which alters, some or all of the elements of informed consent set forth above, or waive the requirement to obtain informed consent provided that the research could not practicably be carried out without the waiver or alteration and:

1. The research or demonstration project is to be conducted by or subject to the approval of state or local government officials and is designed to study or evaluate

a. public benefit or service programs, or b. procedures for obtaining benefits or services under those programs, or c. possible changes in or alternatives to those programs or procedures, or d. possible changes in methods or levels of payment for benefits or services under

those programs. 2. The research involves no more than minimal risk to the subjects, the waiver or alteration

will not adversely affect the rights and welfare of the subjects Whenever appropriate, the subjects will be provided with additional pertinent information after participation. Although IRB exempt protocols are generally exempt from requirements for documentation of consent, the major elements required in a consent form should be incorporated into the materials used to request participation, if any.


Chapter IV: The Legal Angle Codes of ethics in many fields sometimes find themselves at odds with legal codes. Practices that would be ethical may sometimes be illegal, which puts practitioners in a bind. If a solution cannot be found that satisfies both legal and ethical responsibilities, the legal responsibility takes precedence, with the ethical next-best-thing being to reduce the damage caused by the legal responsibility.9 Case in point: all states have laws that require certain professionals (health care providers, social workers teachers, school administrators, etc.) to report suspected cases of child abuse to state authorities. Mandatory reporting frequently contradicts ethical (and usually legal) requirements of confidentiality, but failure to report carries considerable penalties and is not an option. Thus, the best ethical practice becomes to minimize the potential for harm as a result of the reporting. Institutional researchers are subject to certain laws, both federal and state, by virtue of working in institutions of higher education. Technical college employees are subject to additional laws by virtue of working for a state institution, and furthermore the existence of the technical college system itself is mandated in the state statutes and as such introduces additional potential legal obligations. The legal environment in which we work has another aspect as well: precedents set in case law. Frequently, situations arise that were not anticipated by the authors of legislation, and it becomes the role of the court system to interpret the appropriate application of existing laws. Because legal rulings come about from legal challenges, it may be the case that no precedent exists for a particular issue; precedents for similar issues from other states (in the case of state laws) can provide some guidance in these cases, and courts will sometimes cite legal precedents from other states in making their decisions. Other times, different states may arrive at contradictory rulings, or rulings for similar cases may differ because of some specific aspect of one case or the other. Even if the precedents are clear-cut, courts can and do change their minds. Thus, case law can provide use with an advisory opinion in some situations—courts concur with previous rulings far more often than they dissent—but there is no guarantee when it comes to law. State, Federal, local guidelines FERPA The Federal Educational Rights and Privacy Act is a frequently feared but infrequently understood document. Most understand that general gist of the legislation: FERPA prohibits non-consensual disclosure of personally identifiable information contained in education records without the written consent of the student or his or her legal guardian. Fewer know that FERPA is a funding legislation; the penalty for failure to meet FERPA requirements is that your institution becomes ineligible for Federal funding, including most forms of financial aid for students. Thus, the good news is that if you fail to meet FERPA demands, you have not committed any personal crime; the bad news is that your institution might be put out of business. Fortunately, there are “good faith” clauses in most of the 16 exceptions to FERPA protection requirements such that if you try to do what you believe to be best, you will not be threatening 9 Sometimes, ethics codes themselves are changed to reflect legal realities. The ethics code for psychologists used to

prohibit the release of raw test data, because of the potential harm in misinterpretation and potential for compromising test validity. However, it became so common for attorneys to successfully subpoena raw test data in legal cases, the ethics codes changed to make this an acceptable practice.


your schools’ existence. Furthermore, in Gonzaga v. Doe (US 01-679, June 2002) the U.S. Supreme Court decided that FERPA does not create an individual right to privacy, which means that an individual or school that fails to maintain that privacy cannot be held liable by an “injured party.” In other words, if you know the law and do what seems to be best, you shouldn’t have anything to worry about. Another aspect of FERPA that is not very clear is that the “educational records” that must be protected are defined rather narrowly around school performance issues like grades; other types of student records are not protected by FERPA. For instance, “directory information” is not protected; directory information is “that [which] would not generally be considered harmful or an invasion of privacy” (34 C.F.R. § 99.3) and includes, but is not limited to: the student’s name, address, telephone listing, email address, photograph, date and place of birth, major field of study, dates of attendance, grade level, enrollment status (e.g., undergraduate or graduate, full-time or part-time), participation in officially recognized activities or sports, weight and height of members of athletic teams, degrees, honors and awards received, and the most recent educational agency or institution attended—some of which many probably would expect to be protected information. Note, however, that a school is required to notify the students or their legal guardians of the school’s policy on directory information and provide them with an opportunity to opt out of disclosures, so it’s a good idea to check what your school is doing.

Although directory information can be disclosed, a school may not provide directory information that is tied to non-directory information such as race or gender; for instance, a school cannot honor a request for directory information for female students (since that requires disclosing the gender of the student). They can instead provide a list of all students, presumably allowing the requestor to guess at gender. This does raise the important general issue that student information does not need to be identified by name or Social Security number to be a disclosure; indirect disclosure of information is also a violation of FERPA, a point that is reinforced by several Wisconsin court decisions described later on. This is significant in an age where OLAP cubes and drill-down capabilities for looking at data are the state of the art; data cannot be released to the public in a state where it is possible to generate cells of small enough size such that indirect identification of individuals may be possible.

Law enforcement information is also not protected by FERPA. Law enforcement information is any student record that is kept by a law enforcement entity for a law enforcement purpose.

With the rise in identity theft, much of the concern regarding FERPA has to do with social security numbers (see , page 17). Social Security Numbers FERPA Exceptions There are circumstances under which even data that is protected under FERPA can and/or must be released. Many of these exceptions involve legal subpoenas, either from a grand jury or from a law enforcement entity. Unless otherwise noted, the school has the responsibility to attempt to contact the student or parent and notify them of the release request, permitting them an opportunity to try to block that request. If there is good cause, a federal grand jury or law enforcement subpoena can order the institution to not contact the student or disclose the request; in the latter case, the school can request to see the good cause determination. A third legal avenue has opened up due to the “Patriot Act” that passed in response to the World Trade Center attack. The attorney general can issue an ex parte order (a court order in which the adverse party is not informed of the court action) to investigate a student’s background as a potential terrorist.


Not only is the student and/or legal guardian not informed of a release, but the school is also not required to record a disclosure of information if it is made under and ex parte order (unlike other types of release). The Patriot Act also protects from liability any individual that responds in good faith to an ex parte order.

Colleges frequently wish to communicate with each other about students that have transferred colleges. FERPA permits disclosure between (and within) colleges if the purpose of disclosure is to conduct studies for the purpose of improving instruction (34 CFR 99.31(6)i) meaning that for most Institutional Research purposes, data sharing is acceptable. The receiving institution must maintain the privacy of that information just as it must protect information on its own students, and the data should be destroyed when no longer needed. FERPA also permits disclosure in the case of a health or safety emergency, i.e. where disclosure may avert an imminent danger to the student or other members of the school community. Foreign students are required to sign a blanket release with the Immigration and Naturalization Service; as a result, information on these students can be given to INS without requiring an exception. However, the college probably ought to obtain a copy of that release for its own records prior to releasing protected information. Social Security Numbers

Until identity theft became common, it was common to take advantage of the uniqueness of social security numbers to track individual information and prevent duplication of records. Colleges commonly used SSN as student ID numbers, and the WTCS still uses them as its primary identifier. However, the Gramm-Leach Bliley Act (GLB, 1999) introduced requirements that “financial institutions” protect individually identifiable numbers, and educational institutions are identified as financial institutions (in part because they participate in student loan programs). It was generally agreed that institutions that were in compliance with FERPA were also in compliance with the “financial privacy” provisions of GLB. However, there are also “safeguards” provisions, which document protections on data records and systems to prevent unauthorized access; computer systems and other procedures had to be in compliance with GLB as of May, 2003.

The WTCS still uses social security numbers as its primary key, and so the use of the WTCS client reporting data warehouse and other uses of social security numbers could potentially breach the security provisions of GLB. The state is moving towards replacing the social security number in all output files by using the local student number used at each college as the primary identifier, but this has not yet happened. Some use of the social security number will continue, furthermore, because the National Student Clearinghouse still uses them. This example provides an interesting case study into the effect legislation can have on operations in Institutional Research. If using the National Student Clearinghouse Enrollment Search to screen applicants for prior college enrollment, use of social security numbers is permitted because the individuals in question are not yet students and therefore not protected. Once the student is admitted, however, his/her social security number is now protected and cannot be included in an Enrollment Search file, even if essentially the same type of search is being performed (usually, looking for transfers-out; fortunately, SSN is not necessary for previous enrollment searches to be successful). In general, institutional researchers should avoid using social security numbers—especially on things such as mailing labels. Other potential security issues remain as well: the


Banner data system, for instance, keeps both an institutional ID and a social security number for each student, but many types of record searches will produce both as results. Also, students themselves tend to use their SSN for things like registration because they remember them better than their student ID numbers. However, students are sensitized to privacy issues, and may be wary of responding to a survey that is clearly identified with themselves—for instance, if the student’s name or student number is on the form. Where possible, to maximize the confidentiality of responses institutional researchers should use their own identifier keys for response tracking purposes. The Freedom of Information Act The Freedom of Information Act ((FOIA; 5 U.S.C. § 552, As Amended by Public Law No. 104-231, 1996) is a general act that allows citizens to petition government agencies to make public requested documents. The original FOIA was broadened in scope by the Shelby Amendment (Public Law No. 105-277, 1998) to include research data. Fortunately, the application of the FOIA to research data is currently limited by the Office of Management and Budget Circular A-110 (1999), whereby only research that is funded by federal sources and affects government actions that have “force and effect of law” (OMB Circular A-110, 1999) are subject to FOIA requests. This means that Institutional Research data is not subject to FOIA release. However, in the Wisconsin Technical College System, this point is moot because it is superceded by Wisconsin’s Open Records Law. Wisconsin’s Open Records Law Wisconsin Statutes 19.35 states “except as otherwise provided by law, any requester has a right to inspect any record.” This means that unless there is a compelling reason why the greater good is served by protecting information, any record produced by any government agency can be obtained by anyone. Several court cases in Wisconsin have helped clarify how the Open Records law interfaces with the protection requirements of FERPA. In Osborn v. Board of Regents of University of Wisconsin (July 2, 2002) an external agency requested information about UW applicants and students including GPA, tests scores, gender, and ethnicity. However, because there was no request for names or other identification, the Wisconsin Supreme Court ruled that these data by themselves do not constitute personally identifiable information and therefore are not subject to protection under FERPA. Previously, Rathie vs. Northeast Wisconsin Technical Institute (1987) had affirmed that FERPA protection was a sufficient reason for denying a records request under the Open Records Law. However, in Osborn the University was required to supply the records. The University of Wisconsin argued that it would be an undue burden to supply those records, because they would have to redact thousands of records to ensure the protection of some information in those same records, citing Schopper vs. Gehring (1997)10 as a precedent. The Supreme Court denied this argument, stating that “the University is not relieved of its duty to redact…simply because the University believes it is burdensome.” The Open Records Law permits the charging the requestor a fee equal to the cost of producing the requested information. Indeed, the court reaffirmed that the University had dual duties to protect some information and release others. This suggests that it might be prudent to avoid producing records where information that might need to be protected (i.e. names or

10

A request that would have required transcribing 180 hours of audio tape was denied as it was not reasonably limited by subject matter or length of time.


social security numbers) with information that might potentially be requested; for instance, a mailing label on a survey form probably should not contain individually identifiable information.

In the Osborn decision, the Supreme Court did not consider whether the requested data constituted protected pupil records under Wisconsin Statutes 118.125 as this was not part of the University’s defense of its denial of that request. However, State ex.rel. Blum v. Board of Education (1997) affirmed that pupil records are exempt from the open records law. Furthermore, the confidentiality of records must be maintained even if the records can only indirectly be identified, consistent with Wisconsin Statutes 19.62 (5), which defines personally identifiable information as information that can be associated with a particular individual through one or more identifiers or other information or circumstances. Blum, apparently the runner-up for an Academic Excellence Scholarship and/or as Valedictorian, had requested the mid-term grades of both herself and the other student. The Board had attempted to argue that interim grades are the property of the instructor and not the Board, and thus not records subject to request. This was denied, in the that Board’s authority to require this information makes it a record in the Board’s control whether or not it is in the Board’s possession—in theory, his logic might be applied to raw survey data by virtue of their being required by the WTCS. However, while those grounds were rejected, because the requested records were pupil records they could not be released; even if released without identifying information, the grades themselves would have made the scholarship recipient’s identity apparent, and on this basis the denial of the Open Records request was upheld. The Blum case carries a mixed message for WTCS Institutional Researchers. While it clearly affirms that pupil records must be kept confidential, it is not at all clear that data such as survey forms are in any way a pupil record. Indeed, the fact that a record that an entity is reasonably expected to have by virtue of its authority to request it might open the door to such a request. Another factor to consider with regards to research data is that respondents are promised confidentiality. In Mayfair Chrysler-Plymouth v. Baldarotta, the release of information that would reveal the identity of an individual that has been promised confidentiality was a legally sufficient reason for denying an Open Records request. However, the context for that decision was considerably different from the educational context. Mayfair Chrysler-Plymouth had requested from the Department of Revenue copies of checks and other documents that had been supplied by a confidential informant as evidence of irregular accounting procedures. The Wisconsin Supreme Court overturned an appeals court decision by deciding that it was more in the public’s interest to maintain a promise of confidentiality than to maintain the Open Records law. Using this argument, if data is collected with a promise of confidentiality, then it ought to not be subject to disclosure. However, a promise of confidentiality in returning a survey is not the same thing as a promise of confidentiality to an informant, and a future court might make this distinction. The Attorney General has attempted to clarify the meaning of this ruling: confidentiality is a reason for denial if (1) a clear pledge has been made, (2) the pledge was necessary to obtain the information, and (3) the harm from disclosure outweighs the good (60 Atty. Gen. 284). If nothing else, this underlines the importance of specifying in writing whether or not a survey is being conducted in confidence or not. However, just when it looks like there’s an answer to the question of what is and is not open under public records law, the Attorney General goes on to state that records obtained under a pledge of confidentiality must be open to inspection anyway if the official or agency has a specific statutory right to require its submission. It is not clear whether or not this last statement could be applied to raw survey data when that data is required by the WTCS.


WTCS Institutional Researchers should also be aware that the Attorney General has specifically stated that Technical College waiting lists are subject to release under the open records law (61 Atty. Gen. 297). “Telemarketing Laws” Many states have enacted legislation that limits the ability of telemarketers to solicit randomly. The popularity of the state legislation led the Federal Trade Commission (FTC) to create a National No-call List which took effect October 1, 2003 and pre-empts state legislation in the same area.11 However, as of August 2003 the State of Wisconsin was maintaining a separate list from the federal list, meaning that there will be two slightly different sets of rules in effect. In either case most Institutional Research functions will still be permissible according to the new regulations, with one major exception: all technical colleges must honor specific do-not call requests by current or former students. Technical colleges must develop procedures for identifying students that have specified that they are not to be contacted and for disseminating this information campus-wide in a timely fashion; so that they are removed from department or other decentralized mailing lists. Most current or former students can be contacted by the colleges because of several exceptions to the do-not-call legislation: Type of Caller Exception The Wisconsin statues had exempted WTCS IR functions from the do-not-call legislation because a telephone solicitor is defined as a person other than a non-profit organization (Wis. Stat 100.52 (1) (j)). However, the FTC regulations more specifically refer to political organizations and charities being exempt from the regulations; our schools fit the state exception but probably not the federal one. Telephone surveying exception Both the Wisconsin and FTC no-call lists do not pertain to the conduct of telephone surveys. As long as the purpose of the survey is just to gather information, it is not subject to do-not-call legislation unless specifically requested as above. However, if a commercial (i.e. recruitment) purpose is attached to a survey, it is considered to be solicitation as if the survey were not attached. Previous customer exception Businesses have a right to call former customers unless specifically requested to not do so (Wis. Stat. 100.52 (6) (b)). However, the FTC regulations limit this contact to within 18 months of the last commercial contact with the student. The American Teleservices Association has argued that the federal no-call list represents an unconstitutional limit on free speech. Initially a federal court ruled against the list because it unfairly exempted exempts non-profit and political organizations, but in February 2004 a US District court upheld the constitutionality of the list because sales calls violated the privacy of one’s home.12 In May 2004 the ATA filed an appeal with the U.S. Supreme Court; it is possible

11

http://www.the-dma.org/cgi/disppressrelease?article=468 12

http://www.washingtonpost.com/ac2/wp-dyn/A49209-2004Feb17?language=printer


that the do-not call list will be found unconstitutional, which would probably have the effect of wiping out all state laws as well. If you are using email reminders or web-based surveying, you should also be aware of the CAN-SPAM Act of 2003. Much like the telephone surveying exception above, the ways that institutional researchers use email does not meet with the definitions of commercial purposes in the act and so do not apply. Managing Support Staff Support staff are frequently involved in many of the most sensitive aspects of research. Most likely support staff are employed to collect data by telephone and to enter data into a database. These staff may have direct access to protected and confidential information. It is important to be proactive in educating your support staff as to the necessity of maintaining this confidentiality. At WCTC, the HR department has a pamphlet about FERPA that all student workers are given when hired, and they sign an acknowledgement that they understand FERPA rules and that they are obliged to abide by them. Other technical colleges review the AIR Code of Ethics with new hires. This document is not intended to be a management guidebook, but as a general rule it is a good practice (and can avoid future problems) to communicate clear expectations from support staff when they are hired, preferably in writing and in a way that they can attest to their comprehension and compliance with your expectations. In the event that a problem occurs, it is much easier to initiate personnel action if you have proof that the employee knew what was expected of him or her and agreed to those expectations. Data Storage and Security Guidelines Record Retention Since researchers are gathering data, researchers have a responsibility to follow federal, state, and local guidelines for storing the data and following security regulations. According to the Wisconsin statutes 19.32, the organization must:

• Have a policy addressing the issue of destruction of obsolete records

• Must retain records for 7 years unless a shorter time period is approved by the State Public Records Board. As long as a record retention schedule has been established and approved, the college would be in compliance. Upon the retention approval from the Wisconsin Public Records and Forms Board, submit all newly approved retention schedules to the Wisconsin Historical Society for determining the Society’s interest in preservation of the records. In many cases the Society has no interest and waives their interest in the record so that when it comes time to destroy the record according to the retention schedule, the record may be destroyed without contacting the Wisconsin Historical Society.

• Contact the Wisconsin Historical Society before destroying any record. The Society must be given 60 days notice per the statute. The Society can also waive this notification requirement for individual types of records. Also, as long as the record retention schedule has been established and approved, the Society would not need to be contacted unless the destruction of records would not be in compliance with the retention schedule.

The following are the steps required to obtain Public Records and Forms Board Approval for destruction of records in less than 7 years:


1. Identify record series and specific retention periods. Decide scope of the ordinance. 2. Working with Corporation Counsel or City Attorney, if available, draft ordinance. 3. Submit to Public Records and Forms Board for review and approval. Board meets

quarterly ion November, February, May, and August. 4. After Public Records and Forms Board approval, put final ordinance before county

board or city council for final adoption. 5. County or city departments implement the ordinance. Training and limited technical

assistance is available to established records management programs to implement the approved ordinance.

Contact Steve Hirsch, Board Executive Secretary at 608.266.2996 for additional information and copies of available materials. The researcher must follow FERPA guidelines with sharing information. Also, as long as the researcher indicates that the results will be kept confidential and anonymous, the researcher has the responsibility to ensure the confidentiality and anonymity. If the researcher does not adhere to there guideline, the researcher may be putting the college or themselves at risk. The best protection a researcher can have to comply with regulations is to follow a retention schedule than to destroy records without having a retention schedule. In addition, if the records are destroyed following the approved retention schedule, the data would be considered unavailable and could not be required to be shared with regulatory agencies or others requesting the information. Attached is the record retention schedule that LTC has had approved and is adhered to for storing and destroying their records. At LTC, after the hard copy completed surveys have been entered, the electronic format needs to follow the retention schedule. However, the hard copies may be destroyed after one year of the survey being conducted. The following is a website to research further details regarding records management:

http://www.doa.state.wi.us/dsas/recordsmgt/

http://www.doa.state.wi.us/dsas/recordsmgt/


Appendix A: Review of Reliability and Validity Issues13 Institutional researchers need to have a significant academic background in research and statistics. This chapter is not designed to substitute for research education; it is intended as a refresher for individuals that are familiar with the constructs discussed, with examples that illustrate how these issues come into play in Institutional Research settings. Reliability

Reliability is the extent to which a given result can be replicated. A result that is not reliable cannot be valid. Classical testing theory states that

X= t + e

which means that a measured score (X) is the sum of a true quantity (t) and an unknown amount of error (e). Reliability is the extent to which a set of measured scores (x) are measures of true quantities (t), because it is error (e) that causes numbers to change. Reliability cannot be directly observed, of course, so it is inferred by evaluating certain types of correlations known as reliability coefficients. Internal Consistency To the extent that a scale of multiple items is measuring a single thing, then all of the items should correlate with one another. If so, then one ought to be able to randomly split the items into two half-tests, and the two should correlate fairly highly (“split-half reliability). However, which specific split occurs would affect the observed correlation, so instead Cronbach’s alpha statistic is used to produce overall consistency estimates based on all possible splits of a test.14 If the test does not measure a single thing, however—a test that mixes algebra and geometry questions perhaps—then there is not the expectation that internal consistency should be present. Alternate-form Reliability Similar to split-half reliability, in that a pool of items is split into two parallel versions of the same test. To the extent that the split is equitable, takers should score about the same on both versions of the test. The availability of alternate forms is unusual because in many instances you would be better off pooling your items into one, longer test, but tests where retaking is common—Compass/Accuplacer, for instance—alternate forms are necessary.15

13

This chapter was written by Lori Rasmussen (NWTC) and Viktor Brenner (WCTC). Updated May 2004. 14

However, a correlation based on fewer items tends to be lower than one based on more items, so sometimes the Kuder-Richardson formula is applied to estimate what alpha would be if the correlation were based on the full test instead of half of it 15

Actually, both Compass and Accuplacer are more complex than this, in that they are adaptive tests. Items are placed into a pool of items of equivalent difficulty. As the taker proceeds through the test, each successive right or wrong answer determines the difficulty of the next item received, which is then selected at random from the pool. Thus, virtually every person takes a slightly different test.


Test-Retest Reliability To the extent that a construct does not change over time (see above), then giving the same test twice at different times should produce similar scores. This would not be true for characteristics that change over time or for tests that assume that the taker is unfamiliar with the content. Validity Validity is the extent to which that thing that you are measuring is, in fact, what you think you are measuring. Reliability is a prerequisite of validity; you can’t be measuring what you think you are measuring if what you are measuring is error. Reliability is the extent to which X (a score) is in fact T (what you’re trying to measure); validity is the extent to which that T is the T that you intend it to be and not t, T, t, T, or θ. Construct Validity Construct validity is the umbrella concept under which all other types of validity falls; it is equivalent to validity as described above. However construct validity cannot be directly measured; instead, it is the result of a preponderance of evidence supporting validity. Thus, all of the measurable forms of validity constitute evidence for construct validity. Criterion-Oriented (or Criterion-Related) Validity The best way to demonstrate validity is to demonstrate that the test is able to predict some characteristic of the individual. For instance, colleges typically give placement tests to determine whether or not a student is academically prepared to take certain classes. To establish criterion-oriented validity, we would expect that students that score high are usually successful, while students that score lower may struggle. This example is also a case of predictive validity, in that the outcome criterion that we are trying to predict is a future event: eventual course success. In some cases, the outcome criterion we are trying to predict can already be measured, and so we don’t have to wait to evaluate validity; this is known as concurrent validity. Discriminative and Convergent Validity One criterion that you might use to establish validity is to correlate the results with other tests that exist. Convergent validity is the extent to which a test correlates with other tests that it is expected to correlate with because they measure similar constructs—keeping in mind that if the correlation is too high, the new test is simply a replication of the original and only beneficial if it has some advantage (shorter, cheaper, etc). Discriminative validity is the extent to which the test does not correlate other tests that it should not correlate with because they measure different constructs. It is usually true, for instance, that achievement tests (such as the ACT) correlate as highly with each other as they do with ability tests (an IQ test), which is a similar but not the same construct; tests have not been successful at discriminating achievement (what one has learned) from ability (one’s capacity for learning). Content The extent to which the entire domain of the construct is included in the content of the test. A test of “mathematics achievement” should not be made up entirely of algebra problems, but a test of “algebra achievement” or “algebra readiness” probably should be. Usually, establishing content validity requires the expert opinions of professionals in the field, who determine if everything that should be included is adequately represented. In Institutional Research, content


validity is often equivalent to talking to your customers. For example, your registration department may ask you to do a survey for them, and you may start to draft a survey. However, since you do not work in registration, it is unlikely that you will know all of the areas that should be included in the survey, and you may include some areas that aren’t really relevant. It is only by talking to your customer, finding out what they want to know and why, that you can ensure adequate content validity. Face Validity Face validity is the extent to which an instrument appears to be valid to someone taking the test. Face validity makes it easier for test-takers to guess the intent behind items and consciously adjust their responses, which is why in some fields face validity is not always desirable. In Institutional Research, however, face validity is beneficial because respondents will have more confidence in the instrument and be more likely to respond. Incremental Validity An instrument with concurrent or predictive validity is good—but may still be unnecessary if that same information is easily available in some other way. Incremental validity refers to the improvement in prediction over and above other sources. For instance, an ACT score is predictive of freshman year college success, but so is high school class rank; sometimes the ACT adds little to predicting beyond what class rank does. In the Tech system, we often use placement tests to identify students that need remediation; however, given the high base rate of success among students, there may be little incremental improvement in predicting success. Generalizability Generalizability, or external validity, refers to the extent that the results of a given study can be generalized to other individuals or groups. The researcher can assume that the results obtained from the sample are also true of some larger population if and only if the sample was representative of the larger population. Generalizability is threatened by non-representative samples, such as self-selection factors and non-response bias. Generalizability is a matter of degree: results may be generalizable to one group (females for example) but not another (ethnic minorities) based on what groups are adequately represented in the sample.


Appendix B: Individuals New to Institutional Research The first section of the Association for Institutional Research (AIR) ethics code addresses professional competence. If you’re reading this, you’ve been hired to work as an institutional researcher in the WTCS. It has been said that “there is no clear path in to a career in institutional research—and there is no clear path out.16” As a result, it is unlikely that you will have arrived on the job already prepared for all that you will be doing. This appendix will address some of the competencies associated with institutional research, with the intent of helping new practitioners identify areas where additional skills would be valuable so that they can formulate goals for professional development. Possible resources are also identified that can help you strengthen your skills in those areas you deem appropriate. The relative importance of a skill to your personal job description is also an important factor to consider in setting professional development goals. Although every WTCS has institutional research, job titles and functions differ—sometimes markedly—from one college to another. You were hired because you were perceived to best fill the requirements for success at one particular college; it is likely that the areas of your strength are more important to the job description at your college than other areas. If your background in strong in database management but weak in statistics, formal statistical studies may not be as important for your job. Thus, not only is it not expected that anyone be at the levels described below in each area of competence, it is also not necessary depending on your specific job description. Again, the competencies below are not intended to pass judgment on your qualifications—in fact, the Technical College system is full of examples of individuals that started working at a young age in an entry-level position, then climbed the administrative ladder as they developed a more extensive skill repertoire. This Appendix may, however, help you to know where to start in expanding your own repertoire. Statistical Analysis

The more statistics you know, the better work you can do. For instance, if you are investigating the validity of admissions tests, you can use regression or ANOVA to see how well test scores predict course performance. However, if you know logistic regression you can go one step beyond and empirically determine optimal cutoff scores. I have somewhat arbitrarily categorized levels of statistical competence in the following table:

Level Examples Reasons/Applications Fundamental Descriptive statistics,

correlation Fundamental to the job description; pre-requisite for employment

Basic Hypothesis testing, ANOVA, simple regression

Necessary to draw conclusions from results

Quality-oriented Control charts A parallel version of basic statistics, redefined for use in quality processes

Advanced Logistic regression Structural Equation Modeling Factor Analysis

Derivation of optimal cutoff scores “Causal” modeling Scale construction

16

Watt, C. F. (2002). The Career Path of IR: How to Hire, Be Hired, and Motivate Staff. Paper presented at the 2002 AIR Forum, Toronto


Statistical sophistication will allow you to do more precise analyses and generate more specific information. However, attempting to use tools one is not sufficiently familiar with can lead to errors which cause far more harm than any good that could come from using them. As with the Research competencies that follow, the best way to obtain competency in sufficient depth and breadth is through formal education. In 2001, the Association for Institutional Research and the National Center for Educational Statistics provided funding for the development of Institutional Research Certificate programs at the post-Masters level. These programs now operate at Arizona State University, Florida State University (home base for AIR), Indiana University, Pennsylvania State University, and the University of Missouri.17 Especially in smaller districts, Master’s degrees may not be required (although a Bachelor’s degree probably is). However, because more advanced statistical competence allows you to do your job better, the field is moving towards becoming a Master’s level profession—and applicants with Doctoral degrees are increasingly common. If you wish to increase your statistical skills, one way would be to take some graduate-level applied statistics courses (an applied statistics course is likely to reside in program, such as in the social sciences, rather than in a math department, where the course is likely to be theoretical). The Association for Institutional Research also runs an annual Foundations for the Practice of Institutional Research Institute in the summer, where you can learn all the basic statistics you need in a four-day intensive format. Another possibility might be to retain a statistical consultant or tutor, as some graduate students do. Research The research area is actually more than one competency area. Research competencies include research design, survey development, and research project management. Research Design Research design is planning a systematic investigation in a way that maximizes the likelihood of being able to draw correct conclusions from the results. Research design includes issues of the reliability and validity of measures as well as the logic of drawing valid conclusions and threats to conclusion validity. Like with statistics, a graduate-level applied research course is probably the best way to improve one’s skills. Survey Development Many formal research courses focus primarily on experimental designs, leaving little time for topics like survey development. Surveys are the most common method for data collection in IR, though, so specialized education is beneficial. Survey development includes issues like sampling, writing good items, item analysis, and issues in administering surveys. Again, an academic research design course may or may not cover survey development issues, although you should find them in courses on testing and assessment. Survey development is usually an alternate track that you can choose to take as part of the AIR Foundations for the Practice of Institutional Research Institute. If you have sufficient research background to be able to teach

17

For more information, see http://www.airweb.org/page.asp?page=466


yourself, there are also a number of good books on the special considerations involved in survey research. Research Project Management Research project management is process of ensuring that a research project gets completed in a timely fashion. Experience is the best teacher of when to do what in order to meet a deadline; some new institutional researchers may already have that experience from graduate school. In the WTCS, many of your deadlines will be common across colleges, so you can tap other RPD members for assistance. Project management tools are also available (Microsoft Project, for instance) and project management courses abound—they may even be taught on your campus as part of a management program or ATC. Qualitative Methods In addition to surveys, you may use focus groups, directed interviews, or other means of obtaining data. Data from these types of projects are generally frequently qualitative in nature, requiring qualitative analysis to turn the data into information. Even within a survey format, you may use open-ended questions that warrant a qualitative analysis. Supervisory Skills Institutional researchers vary in their supervisory responsibilities; by the time they become institutional research directors it may be their primary responsibility. Institutional research practitioners may still supervise support staff, data collectors, and student workers. Most of the technical colleges have supervisory management programs, but these tend to focus on retail applications. Supervisory skills might be a good use of a mentoring program if your college has one. Information Technology Skills As the state moves towards increased use of data warehouses, these are becoming more important. Not all of these skills are necessary for everyone, which is good because probably no one is adept in them all. However, the table below indicates some common Information Technology applications used in IR and what they are used for. Type Common

Application Uses

Word processing Word Create reports to communicate findings Graphs and charts Excel Pictures tell a thousand words Presentation graphics

Powerpoint Generate more audience-friendly versions of live presentations

Statistical Analysis SPSS, SAS, Minitab Hypothesis testing and advanced analysis Data management Access, Peoplesoft,

Banner Enter, maintain, and extract the data needed for a given project

Advanced Querying SQL, MS Query Obtain specific data sets involving multiple simultaneous constraints

User-Interactive Querying

Excel (pivot tables), Cognos

Report data in a format that users can manipulate so that they can see exactly what they want to see


Type Common Application

Uses

Metadata Cognos, Discoverer Combine data from multiple sources to create a seamless view for active queries

VPN (Virtual Private Network)

Citrix Obtain access to remote data, such as the client reporting data warehouse at the state office

Web Development Dreamweaver, Front Page

Create web-based reports or department pages

Web Surveying Remark Create online versions of your surveys OMR (Optical Mark Reader)

Remark, Scantron Create forms that can be scored automatically

Desktop Publishing MS Publisher, Quark

Lay out your own publications for printing

Photo editing Photoshop, Illustrator

Enhance reports and publications with pictures and graphics

WTCS-specific Client reporting workbooks

Obtain historical client reporting data from your own college or comparing with others

Training in any of these areas may need to be local, tailored to the specific platform and implementation at your college. Basic education in Office applications are available at any technical college and usually can be taken in self-paced and/or online format. Some IT skills may be taught as part of the Foundations for the Practice of Institutional Research Institute; more advanced skills can be obtained through the AIR Information Technology Institute, which also runs every summer. Training in your college’s Enterprise Information System (Peoplesoft, Banner, or Datatel) will have to be arranged by your college; training in less widely-used applications (Remark, SPSS) will probably only be available through the vendor. Communication Skills Institutional researchers need written communication skills to produce written reports that are understandable and user-friendly. They need oral communication skills to present results to stakeholders and to take part in discussions, committees, and work teams. As institutional researchers rise to the level of director, their job will involve less time on projects and more time providing key decision-makers with the information they need, so communication (and interpersonal) skills become more important. Legal and Ethical Issues There are a number of legal and ethical factors that are unique to IR, and some unique to IR within the WTCS: that’s why RPD has written this manual. Here, too, other RPD members (not necessarily members of the Ethics Advisory Team) are your best resource. If you are not familiar with RPD, you should attend one of its meetings; the schedule, as well as a listing of current officers and members, is available at

http://matcmadison.edu/research_planning/state/default.htm

If you’re an institutional researcher in the WTCS, you’re a member—welcome!

http://matcmadison.edu/research_planning/state/default.htm