Best Practices: Improving the efficiency and effectiveness ... · Best Practices: Improving the efficiency and ... to the project to create accessory extraction forms and the

SRDR Quarterly Training Brown Evidence-based Practice Center

Brown University

December 5th, 2014 1:00pm-2:00pm

Best Practices: Improving the efficiency and effectiveness, and consistency of data entry

SRDR™ was initially developed by the Tufts EPC and currently maintained by the Brown EPC under contract with AHRQ (Contract No. HHSA 290-2007-10055-I & HHSA 290-2012-1200012-I ).

12/5/2014 http://srdr.ahrq.gov 2

Presentation outline }  Summarize common difficulties / challenges encountered by

users entering data into SRDR }  Demonstrate best practices to mitigate these challenges


}  Inefficient design of extraction forms }  Extraction forms with lots of empty space }  Unintuitive question titles on extraction forms }  Unrestrained extraction forms giving too much latitude to extractors

}  Entering data prospectively into SRDR }  Pitfalls and limitations for copying and pasting data into extraction forms }  Losing track of data extractor progress, and questions they might have

}  Entering data retrospectively into SRDR }  Many different file formats (MS Word, MS Excel, PDFs) }  Preparing results data for importing (coming feature)

Challenges entering data into SRDR


}  Exporting data from SRDR in a format usable to others (statisticians, analysts) or in a format ready to be used as evidence tables in appendix of an EPC report

}  To be addressed in a future Webinar in January }  new features added to SRDR since the last training Webinar held in

September

Challenges exporting data from SRDR


}  Compilations of strategies used by SRDR Users, EPCs, and the SRDR Team

}  Recommendations that will enable you to overcome the challenges associated with entering data into SRDR

}  Benefits of using best practices include: }  Reduce time and effort required to enter data into SRDR }  Improve accuracy and consistency of data collected }  Increase accessibility of data for other SRDR contributors and public

users }  Take advantage of SRDR Tools and features

Best Practices


Best Practices for Designing Efficient Extraction

Forms


}  Background }  Key Questions (KQ) entered into an SRDR project should parallel the

KQs addressed in your systematic review or EPC Report. }  Each KQ can only be associated with a single (primary) extraction form }  Several KQs may use the same extraction form

}  Challenge }  Since a KQ can only be associated with one (primary) extraction form,

collection of certain data across all eligible studies for the KQ (i.e. study quality information for multiple study designs) can result in extraction forms with long lists of mutually exclusive questions (i.e questions relevant to only one type of study design) that could be unwieldy to navigate

Create accessory forms to complement primary extraction form


}  Best Practice }  Create “accessory” extraction forms to complement the data collected

on a primary extraction form (each accessory extraction form will require adding a new KQ to your project)

}  Note: You will need to describe in the “Project notes” field of the Project Information page of the SRDR project, which KQs were added to the project to create accessory extraction forms and the KQs from the report they address.

}  Benefit }  Creating “accessory” extraction forms will shorten the number of data

fields that an extractor need enter data into for any one extraction form.

Create accessory forms to complement primary extraction form (Cont.)


Example: Denoting addition of accessory extraction forms in “Project Notes” field.


}  Background }  A question title (data header) is required for each question added to the

extraction form }  Displayed as column (spreadsheet) headers in simple and advanced

export of data from SRDR

}  Challenge }  It is difficult to understand arbitrarily named or unintuitive question

titles (e.g., BMI_X instead of Average BMI). This difficulty adds time required by the data extractor to interpret the meaning of the question and presents confusion to other users.

Use self-explanatory question titles on extraction forms


}  Best Practice }  Question titles should be self-explanatory and include: units

and/or specific data ranges / formats (i.e. only whole number values; or DD/MM/YYYY date format)

}  Benefits }  Minimize data extraction errors }  Improve accessibility of data by users

Use self-explanatory question titles (Cont.)


Example: Intuitive and unintuitive question titles

Self-explanatory:

Self-explanatory:

Not self-explanatory:


}  Background }  Users can add a list of “suggested” arm and outcome titles to the Arm

and Outcome tabs of SRDR extraction forms. }  During individual data abstraction, data extractors are given the option

to use these pre-specified arm and outcome titles or create their own.

}  Challenge }  When extractors have the latitude to name the arms and outcomes on

their own, they may come up with different names (e.g., Cardiovascular disease could be named as: CVD, CV disease, etc.. ). Variations in the naming of arms and outcomes is unwanted and require time to standardize. It will also confuse users of the data unfamiliar with the project.

Pre-specify arm and outcome titles for data extractors to use on the extraction form


}  Best Practice }  Step 1: The project lead should pre-specify a list of the arm and outcome

titles and then add the titles from this list for other extractors to use as “suggested” arms and outcomes on the Arms and Outcomes tabs of the SRDR extraction form.

}  Step 2: Train data extractors in the team to use suggested arms and outcomes, unless it is absolutely necessary to create a new name (and then the new name should be added to the suggested list).

}  Benefits }  Reduce data entry time }  Reduce spelling errors, case errors and unwanted redundancy }  Reduce data cleaning needs

Pre-specify arm and outcome titles for data extractors to use on the extraction form


Example of the unwanted variation that can result in Outcome titles


}  Background }  Users may choose from the following selection of question types to add

to an SRDR extraction form: }  Text }  Dropdown menu }  Multiple choice (select one) aka. Radio button question }  Multiple choice (select any) aka. Checkbox question }  Matrix of choices (only one answer allowed) aka. Matrix of Radio buttons }  Matrix of choices (select more than one answer) aka. Matrix of Checkboxes }  Matrix of dropdowns (specify option in each cell)

}  Challenge }  SRDR extraction forms relying primarily on text, dropdown menu, and

multiple choice questions could result in very long forms. Long forms are a challenge to use because of excessive vertical scrolling which increases the time needed by data extractors to enter data.

Shortening extraction forms using dropdown and matrix-type questions


}  Best Practice }  When possible, consolidate multiple questions into a single matrix

question. (e.g. when the same set of descriptive statistics and other information is asked for multiple risk factors on the Baseline tab of SRDR extraction forms, the individual questions can be consolidated into separate cells within a single matrix question).

}  Benefits }  Any number of columns or rows of cells may be added to matrix

questions on SRDR extraction forms enabling users to minimize the length of the form.

}  Values entered into each cell of a matrix question are exported top separate columns of simple export spreadsheets allowing for easy analysis/isolation of data.

Using Dropdown and Matrix-type questions (Cont.)


Example contrasting vertical space used by matrix and non-matrix questions Multiple Text Qs Single Matrix Q Multiple Matrix Qs


}  Background }  Arm and outcome descriptions can be added to arms and outcomes

entered into the Arm and Outcomes tabs of SRDR extraction forms.

}  Challenge }  When exporting data from SRDR, Arm Title and Arm Description (same

for Outcome) are concatenated into a single character string (e.g. the title,” Vitamin D”, with description, “consumed”, will appear on export as Arm: “Vitamin D [consumed]”) to distinguish arms that may have the same title but different descriptions. This concatenated string may require additional processing steps before analysis.

Save time cleaning data by creating arm and outcome description questions on your forms


}  Best Practice }  Instead of entering information into arm description and outcome

description fields on Arm and Outcome tabs, create a question on Arm Details, Outcome Details tabs to capture this information.

}  Note: To use this best practice, Arm Details and Outcome Details tabs on extraction forms should be set to display questions for each arm or outcome, respectively.

}  Benefit }  Less time needed to clean arm and outcome titles exported from

SRDR

Avoid using arm, outcome description fields on Arm, Outcome tab (Cont.)


Example: Adding arm description value to arm title on Arm tab

Arms tab of extraction

form

Simple export of Arms tab

SRDR Simple Export tool used to export data entered

into extraction form


Example: Adding arm description value to arm title on Arm tab (Cont.)

Export of extraction form using Report Builder tool


Example: Adding arm description value to question on Arm Details tab

Arm Details tab of

extraction form

Simple export of Arm

Details tab

SRDR Simple Export tool used to export data entered

into extraction form


Example: Adding arm description value to question on Arm Details tab (Cont.)

Export of extraction form using Report Builder tool


}  Background }  Questions asked on an extraction form (excluding questions concerning

study design or study quality), are typically asked for a specific arm or outcome defined in a study.

}  This is typically accomplished by creating questions for specific arms or outcomes on the form, or instead creating blank data fields into which miscellaneous arms or outcomes may be entered (the same applies when creating extraction forms in SRDR).

}  Challenge }  An extraction form (e.g., paper, software) seeking to predefine all

possible arms/outcomes could be lengthy and unwieldy. }  To accommodate unexpected arms/outcomes during data extraction, the

extraction form will need to be revised (this problem also applies to SRDR when features are not used).

Use SRDR’s feature to dynamically create arm/outcomes questions in tabs


}  Best Practice }  Rather than enumerating each question explicitly on an SRDR extraction

form, use the “dynamic” arm/outcome question feature to ask a question on extraction form tabs ( Arm Details, Baseline, Outcome Details, Results, Adverse Events) for just the arms and outcome that are defined in a particular study.

}  Pro-tip: After constructing an extraction form, always check to see how data fields are presented in simple and advanced data export. Also, pilot the extraction form.

}  Benefits }  Minimizes the number of times questions need to be asked on a tab,

resulting in a smaller extraction form and quicker navigation }  Extraction form will readily accommodate unforeseen arm/outcomes

keeping the number of revisions to the extraction form at a minimum

Use SRDR’s feature to dynamically create arm/outcomes questions in tabs (Cont.)


Example: Extraction form that collects data for a predefined number of arms


Example: Replicating the extraction form in SRDR and exporting data

Simple export of Arm Details tab of SRDR extraction form


Simple export of Arm Details tab of SRDR extraction form

Example: Incorporating feature to dynamically create arm questions in form and exporting data

1. 2.

3.


Best Practices for Entering Data Prospectively into SRDR


}  Definition of prospective data entry }  Copying data from the full text of studies included in a systematic review,

directly into data fields on SRDR extraction forms.

}  Method }  Manual entry: Requires typing or copying and pasting data values,

individually, into data fields on the extraction form.

Background


}  Background }  Symbols, such as those listed below are commonly found in the full text

of studies and can be typed or copied and pasted into extraction forms in SRDR }  Mathematical operations (÷, ×, ±, °, >, < , ≥, ≤, ≠, =, ≅,≈,…) }  Greek Letters (α, β, γ, ε, ζ, η,θ, λ, μ, ξ, ω, …) }  Other Symbols ( &, ®, ™, $, €, –, —, ^, ©, …)

}  Challenge }  Among the symbols that can be entered into extraction forms, only a

slim minority can be exported from SRDR. Among the symbols listed above as examples, only >, <, &, $, and ^ can be exported to a simple export spreadsheet. The remainder are substituted with a “blank” space on the simple export spreadsheet.

}  Note: All symbols imported into SRDR using the import tool may be viewed upon export.

Use work-arounds when entering symbols unsupported by SRDR into extraction form data fields


}  Best Practice }  Data extractors should be trained to use the following commonly

accepted work-arounds for symbols that can otherwise not be exported from SRDR }  ≤, ≥, and ≠ should be written as “<=“, “>=“, “/=“ }  “cm3” can instead be written as “cm^3”

}  Otherwise, extractors should refrain from entering symbols into SRDR or when necessary, spell symbols out. (i.e. “α”entered as “alpha”, or “And” instead of “&”)

}  Benefits }  Use of work-arounds or spelling out symbols means that data from a

project can be exported by other users and the absence of specific symbols will not throw off the meaning of data values.

Use work-arounds when entering symbols unsupported by SRDR into extraction form data fields (Cont.)


Example: How string of symbols is not exported completely from SRDR

Response as it appears on Simple Export spreadsheet

Response as it appears on

SRDR extraction form

SRDR Simple Export tool

used to export data entered into extraction form


}  Background }  Superscript and subscript text are often used in full text studies for

footnotes (xi), mathematical operations (cm3) and citations (Smith, 201423).

}  Challenge }  When superscript or subscript text is copied and pasted into data fields

on an extraction form in SRDR, the super/subscripted text reverts to a regular sized font which can change the entire meaning of a particular response thereby reducing the accuracy of data collected (e.g. “Unadjusted RR = 1.34” appears as “Unadjusted RR = 1.34” on extraction form) or otherwise make the data entirely incomprehensible to other users (e.g. “xi” appears as “xi” on extraction form)

Copying and pasting super/subscripted text into extraction form data fields


}  Best Practice }  Data extractors should either avoid copying and pasting super-

subscripted text into data fields on SRDR extraction forms or find an alternative way to enter these values (i.e. enter cm3 as cm^3) such that their meaning is not lost to other users.

}  Benefits }  More accurate data collection }  Less confusion for other users

Copying and pasting super/subscripted text into extraction form data fields


}  Background }  When a question on an SRDR extraction form is left with a blank

response it is recognized by SRDR as a null-value.

}  Challenge }  When reviewing the export of data from a project in SRDR, it is

therefore impossible to know whether the response to an extraction form question was 1) left blank because the full-text of the study simply did not contain the data requested by the question OR 2) left blank because the data extractor forgot to answer the question.

Use of “No Data” (ND) response to answer questions on form when no data is available


}  Best Practice }  Data extractors should enter “ND” into all extraction form data fields

for which no data is given in the article to respond.

}  Benefit }  Project leads are able to more readily identify data fields that were

missed or forgotten by extractors and have them corrected, thus improving accuracy of data collected.

Use of “ND” for questions on form for which no data is available (Cont.)


}  Background }  At the bottom of each tab on an SRDR extraction form is the button

“Mark Section Complete” which data extractors may use to keep track of the sections they have completed on the extraction form.

}  Clicking the button twice will change the status of the tab from “Complete” to “Incomplete”

}  The completion status of all tabs in the extraction form may be viewed as well as “toggled” by the extractor on the Finalize tab of the extraction form.

}  Challenge }  Keeping tabs on the progress of data extractors during data abstraction

can be a difficult and time-consuming process

Use of “Mark Section Complete” button during individual data extraction


}  Best Practice }  Each time a data extractor completes entering data into data fields on an

extraction form in SRDR, they should mark the tab “Complete” using the Mark Section Complete button and remember to review the completion status of each tab from the Finalize tab of the extraction form before exiting.

}  Benefits }  Enables data extractors to remember whether there is a section they

have yet to complete, thereby improving accuracy of data collected. }  Pro-tip: Allows the Project Lead to use the Progress Report tool

located under project Tools to view a report of each extractor’s overall progress.

Use of “Mark Section Complete” button during individual data extraction (Cont.)


Example Report generated by the SRDR Progress Report Tool


}  Background }  Notes can be left by data extractors on the Finalize tab of the extraction

form for a particular study, in the “Note” data field. }  Notes may be viewed by project leads from the Complete Study List of

the project.

}  Challenge }  It is important that project leads are able to address questions

extractors may have while extracting data from studies.

}  Best Practice }  Data extractors should be trained to leave notes on the extraction

form for project lead, when needed.

}  Benefit }  Faster data extraction

Use of “Note” data field on extraction forms during individual data extraction


Example of Notes left by extractor for a study on study list


Best Practices for Entering Data Retrospectively into SRDR


}  Definition of retrospective data entry }  Transferring data that has already been collected from the full text of multiple

studies and saved in an external file (MS Word, MS Excel, PDF file…) into data fields on SRDR extraction forms OR otherwise uploading the external files to a project for easy downloading by other SRDR contributors and public users

}  Method(s) }  Manual entry: Requires typing or copying and pasting data values,

individually, into data fields on the extraction form }  Importing: Requires, first, formatting and saving the data as a Microsoft

Excel spreadsheet. Data from the spreadsheet is then read and transferred, in batch, by the SRDR Import tool into data fields on the extraction form

}  Uploading: Requires using the SRDR Manage Reports tool to “attach” files (PDF, text , MS Word, MS Excel …) such as EPC report appendices to SRDR projects. These files will then be available to download by other users

Background


1.  Log into SRDR with your SRDR username and password.

2.  Check that you are in the MySRDR tab.

3.  Click the edit link under the project to which you would like to upload the PDF files.

4.  In the Navigation menu, on the left side of your project, click the Data Export Tool button located under Project Tools.

5.  Next, on the Export Data Tools page, click the Manage Reports link, located in the box labelled Project Reports to access the Manage Reports Tool.

6.  On the following page, click the Add Report button.

7.  Under Upload Report, in the Description field, enter a brief description of the PDF file you want to upload to your project.

8.  Next to the Description field, click the Choose File button to select the PDF file from your computer.

9.  Select the File.

10.  Below the Description field, click the Upload button, to upload the file to your project.

11.  Repeat steps 1 through 10, to add additional PDF files to your project,

12.  At the top of the page, below the MySRDR tab, click the Edit Project tab to return to your project information page.

Steps for Uploading a file to SRDR


}  Manual Entry }  Pro: Data can be entered into every tab of SRDR extraction form }  Con: Data must be entered into tabs on extraction form individually

(time intensive process)

}  Importing }  Pro: Batch Process, can enter data into tab for any number of studies at

once }  Con: Data on Excel spreadsheets has to be formatted to meet

requirements for importing (see prev webinar for requirements); Data cannot yet be imported into tabs supporting “dynamic” arm/outcome question feature, including the Results tab (feature coming soon)

}  Uploading }  Pro: Files are uploaded quickly; Files are viewable by other SRDR users }  Con: Data values are not incorporated into the SRDR database

Methods for Entering data saved on a Microsoft Excel file into SRDR


}  Manual Entry }  Same Pros/Cons listed for entering data saved on MS Excel files

}  Importing }  Pro: Batch Process, can enter data into tab for any number of studies at

once. }  Con: Additional step required to transfer data to MS Excel files; After

which, same cons listed for importing data from MS Excel files apply. }  Pro-tip: Watch for the inclusion of returns when copying and pasting data from

MS Word into excel, or else each new line in the copied text will be pasted into an individual cell.

}  Uploading }  Same Pros/Cons listed for entering data saved on MS Excel files

Entering data saved on a Microsoft Word file into SRDR



}  Importing }  Pro: Batch Process, can enter data into tab for any number of studies at once. }  Con: Additional step required to transfer data from to MS Excel files (best

accomplished using PDF to Excel conversion software, i.e. Able2Extract Pro distributed by Investintech Inc.); After which, same cons listed for importing data from MS Excel files apply. }  Pro-tips:

¨  Can print MS Word files to PDFs and then use PDF conversion software to convert PDFs to MS Excel files.

¨  Using Conversion software some symbols on text of PDF file may not convert correctly to the MS Excel file (i.e. ≥, ≤ symbols in particular)


Entering data saved on a PDF file into SRDR




}  Importing – Not Yet Available }  Requirements:

}  Data must be saved in a Microsoft Excel spreadsheet }  Data arranged in a vertical fashion with:

¨  Multiple rows per study record ¨  Column headers for Arm, Outcome, Time point, Time point units, Subgroup, and the

desired outcome measures

Methods for entering results data into SRDR


Example: Importing into the Results tab of an SRDR Extraction form

PMID Title Author Year Arm Outcome Time Point Time Point Unit Mean SD

23812661

Compara?ve efficacy of LEAP, TEACCH and non-‐model-‐specific special

educa?on programs for preschoolers with au?sm spectrum disorders

Boyd 2013

LEAP IQ Baseline N/A 5 3

23812661



Boyd 2013

LEAP IQ 1 years 10 4

23812661



Boyd 2013

LEAP AUen?on Baseline N/A 6 2

23812661



Boyd 2013

LEAP AUen?on 1 years 9 3

Template of Microsoft Excel spreadsheet to be imported:


Example: Importing into the Results tab of an SRDR Extraction form (Cont.)

Result of Import as seen in the Results tab of the SRDR extraction from:


Reminder These slides may be downloaded from the Quarterly Training tab

of the SRDR Help & Training page (http://srdr.ahrq.gov/help)


Questions?

Documents

Best Practices: Improving the efficiency and effectiveness ... · Best Practices: Improving the efficiency and ... to the project to create accessory extraction forms and the