Data Management What You Should Know

DATA MANAGEMENT Things that can go wrong with data What you should know and why you should care    Crucial data elements may be missing Data may be incorrect due to errors in:  Data collection  Data entry Data may be not have common identifier     Cannot be merged May be merged incorrectly Data may not be saved or backed up Data files may be lost or corrupted 2 Real World Examples  A few illustrations of common data problems from the popular news sources “Forbes' annual list is out, and Haverford plummeted from No. 7 to No. 27 - for no obvious reason. A College spokesman explained that the error was based on single figure: A zero was incorrectly entered in database instead of 108 for the graduation rate of white women who enrolled in 2004. …But no revision is planned, since the magazine and the online list has already been published.” Data Entry Error 3 Oops: Excel Error Calls Into Question… Vertex stock slides over cystic fibrosis data mistake “Shares in Vertex Pharmaceuticals have taken a hit after the company had to take the rather embarrassing step of correcting previouslyannounced interim mid-stage results of a combination cystic fibrosis treatment. …the result of a misinterpretation [of the denominator of the treatment group] between the firm and its outside statistical vendor Data…” Mismanaged   A book by Harvard Researchers entitled ‘This time it’s different” contained “…serious errors that inaccurately represent the relationship between public debt and GDP growth among 20 advanced economies in the post-war period.” The Authors admitted they forgot to include five rows in an Excel file resulting in exclusion of data from Australia, Austria, Belgium, Canada, and Denmark —a “coding error” which they said was “a significant lapse on our part.” excluded key data 1 Data Entry/Management in Excel How Bright Promise in Cancer Testing Fell Apart       Duke Cancer Center’s gene-based tests proved worthless, research behind them was discredited Statisticians from MD Anderson discovered errors such as columns moved over in a spread-sheet; Duke team “shrugged them off” as “clerical errors.” Four papers were retracted Duke shut down three cancer trials Center leaders resigned or were removed People died and their relatives sued Duke 8 Goal: Convert Data into Electronic Format as Quickly as Possible Data Management 101:     No single “right” way to collect or manage data Consider:  Environment/location  Resources  Regulations Be sure to plan prior to study start Do what works for the study at hand Where to start? DMP Purpose: To help you manage and share your data; meet funder requirements. General elements include: • Project or study description • Documentation, organization, storage • Access and sharing • Archiving 2 DMP: Basic Elements          Study design, data types and sources Storage format and location Naming conventions, documentation Software used for manipulation Project Staff – who has permission to what Identifiers (if applicable) Back ups, security Archiving Sharing Beginning: Identify Key Data Elements     Review hypotheses What are primary, secondary outcomes? What covariates and confounders must be collected? What are the data sources?     Questionnaires Labs, imaging Medical record review other external sources (e.g., lab results, medical records, death certificates) Visit Protocol: Data by Time-point  Determine visit Schedule and data collected at each visit  Questionnaires  Labs  Other?  Consider data not be connected to visits  Adverse events, serious adverse events  Hospitalization  Death  Medical records Timeline GRANT PROPOSAL Design Project Draft DMP Identify Data Sources GET FUNDED! Budget – Including DMP Expand on DMP Re-budget Analyze Data Collect Data Share Data Publish Archive Other Data Elements    Regulatory data:  IRB requirements  Safety (DSMB)  FDA (e.g., 21 CFR, part 11)  Other? Tracking/Study management data:  Tracking participants  Data elements by time-points Harmonization  NIH  Other Sample data/visit grid 3    Timelines and Tasks Create a Visual Timeline Develop Protocol and Analytic Plan Create and pilot of forms/assessments Design/construct data systems         Data Collection/entry Participant/Data Tracking   It doesn’t have to be fancy More detail is better but something simple is better than nothing Plan to review and revise it often Subject recruitment Data collection (baseline and follow-up) Data cleaning, auditing, and QA Preliminary analysis Manuscript preparation & submission Simple overview Timeline Sample Task-based Gantt Design/construct data collection systems Multi-task Indicating Responsible Parties Tools Of The Trade         Analytic plan Detailed protocol Well designed data collection forms Tracking system Data capture/entry system Plan for data query (checking/cleaning) Manuals Data dictionaries 4 Sample Data Dictionary: SAS formatted dataset Data Collection Forms   Creating a Data Collection Form: Get input from all points of view Data will usually end up on some type of “form” whether it is interview, chart review, or imaging results Make sure you plan carefully and leave time as this can be a lengthy process Why is this topic important?  Sloppy forms indicate sloppy research Data Manager Study Coordinator  CRF may not answer study questions  Danger of collecting:  too much data Case Co-Investigators Report Study Sponsor? Form  too little data  the wrong data  Annoyed:  Participants Biostatistician Clinicians  Study Coordinator  Data Analyst… 5 Successful Form: Consider ALL Functions When designing forms…  Data Collection - who is completing form? Think Google  Study Staff (Coordinator, Clinician)  Participant  Clinician  Data entry - who is entering data?  Study staff  Outsourced Not Yahoo  Data management/cleaning  Auditing  Data analysis What makes a good form? Pilot Your Forms Prior to Data Collection  User-friendly, uncluttered, well organized  Provides clear instructions for completion  Terminology familiar to person filling out   Questions only asked/data collected in one place and only one place Easy to refer back and clean data  Avoid Open-ended & Include Response Measure  Are open-ended questions generating common responses? Train clinic personnel/revise or simplify forms Categorize/code Make corrections prior to start of study Do not start data collection until forms are final Include Clear Instructions What is your date of birth? ____ Date of Birth? __/__/____ How much do you weigh? ___ How much do you weigh? __.__ (pounds) How tall are you? ___ /__ (feet)/(inches) MM DD YYYY Record subject’s temperature ______ Are “skip” patterns followed correctly?   Reword/drop question   Unambiguous questions How tall are you? ________ Test in target population (e.g., age, literacy)  Are items left blank?  Reading level matches study participants/evaluators    Record subject’s oral temperature __ __. __ (f) A. What is your race/ethnicity? (Check one) Caucasian 1 African American/Black 2 Asian, Pacific Islander 3 Native American 4 Other _____________ 5 B. What is your race/ethnicity? (Check all that apply) 1  Caucasian 1  African American/Black 1  Asian, Pacific Islander 1  Native American 1  Other _____________ 6 Account For Missing Data Beware of “check all that apply” COMORBIDITIES b. Heart disease c. Diabetes d. Hypertension e. Pulmonary disease  CBC Unit Value 1. Hemoglobin g/dl __ __.__ Not Done 2. Hematocrit % __ __.__ Not Done M/mm3 __ __.__ Not Done 1  1  1  1 a. COMORBIDITIES b. Heart disease 1 2 c. Diabetes 1 2 d. Hypertension 1 2 e. Pulmonary disease 1 2 3. RBC Specify the Units Categorize Anticipated Responses 1 USA 1 USA  2 Guatemala 2 Guatemala 3 Mexico 3 Mexico  4 Dominican Republic 4 Dominican Republic 5 Other 6 El Salvador 5 Other ID Assignment  Must be UNIQUE for each subject  Should appear on every form (preferably page)   May be a simple number 1001  May be multi-part: 102101   Do NOT do this Links paper form with specific record in database Multiple forms, “merge key” in database   Example of an ID that is not unique 1 = Site 02 = Language 101= ID 7 Don’t Underestimate Need for Version# /Date Consider who is Completing a Form Participant Coordinator Clinician In Summary, when designing questions:  Avoid ambiguous questions and open ended responses Include clear instructions Be sure form complexity matches collector (self, study coordinator, clinician) Collect data elements in correct format (“continuous” or “categorical”) Make categories mutually exclusive  Pilot your forms in the target population     Paper or Electronic? The Good The Bad And The Ugly Paper Forms / Manual Entry Advantages  The old “standard”  Shorter start-up time (Word/PDF)  Relatively easy to train staff  Hardcopy document to refer back to  Can be done anywhere Disadvantages  Costs: data entry, storage and shipping  Longer time from collection to database  Errors in data collection (missing, out of range, skips) 8 Electronic Data Capture A Word About “Canned” Software Advantages Cleaner data at entry (required fields, skips, ranges) Can use data in real time (or close to it)  No extra data entry costs  Data can inform next visit even for short follow up Disadvantages  Programming time and costs  Increased hardware and software costs  Infrastructure concerns (software versions, internet connection, back-up equipment)  Data security        Many “canned” sofware packages available No single best choice Cost can vary widely Database structures vary Do your homework to make sure what you get will work for your project Resources available through the CTSI Consider languages when selecting data entry methods and software Find out what software is available through your institution 52 What to use…?  To determine what software is best suited for your project see:    What is available to you? What is the cost (can you afford it)? What has the features you need (e.g., language) Once data are collected…   Get your data into a useful format No “right” format – use what works for you     SQL database SAS datasets SPSS Excel (be careful!) 9 Personal Identifiers Crosswalk for Personal Identifiers    Do not store any identifier unless you have a good reason for it Do not store identifiers in same files with study data. Identifiers should be kept separate! Create “crosswalk” files of identifiers and store them someplace secure. First Name Last Name Study Screen ID ID Phone # Joseph Blow 1234 555-131-1111 50001 Subject ID MRN SSN 1234 64322 ****** Identifiable Data Crosswalk Study Data ID Visit Var 1 Var 2 Var 3 1234 1 5 2 3 HIPAA Identifiers HIPAA Identifiers (cont) 1. Names 2. Addresses other than state, and first three digits of the zip code 3. All elements of date other than year, and all specific ages over 89 years 4. Telephone numbers 5. fax numbers 6. Email addresses 7. Social Security numbers 8. Medical Record numbers 9. Health plan beneficiary numbers 10. Account numbers 11. Certificate/license numbers 12. Vehicle identifiers and serial numbers 13. Device identifiers and serial numbers 14. Web universal resource locators (URLs; web site addresses) 15. Internet protocol (IP) addresses 16. Biometric identifiers, including finger and voice prints 17. Full face photographic images and any comparable images 18. Any other unique identifying number, characteristic, or code Participant Tracking Tracking the Participants You need a system to track participants  Tracking for Study Management:    Tracking tools for study staff:    Screened, Eligible, Enrolled Monitor and report progress Schedule/reminders follow up visits Collection of all data points at each visit Small study may use Outlook or Excel; large study may need a tracking system 10 Simple Participant Tracking Tools More Complex Tracking Tool Track in an Excel Spreadsheet Put right into a Calendar (e.g., Google or Outlook) Consort Diagram Reports  For the study team to view at regular meetings or online as needed Enrollment Phlebotomy Reports: Visual as well as Tabular 11 Actual vs. Targeted Enrollment Screened, Eligible and Enrolled Reports  For the study staff to help manage tasks and know what needs to be done, when 12 Sample Tracking Report: Follow UP Tracking the Data Elements      Look at the Data Early and Often      You cannot fix a problem if you don’t know it exists Get data into electronic format ASAP so it can be more easily reviewed Monitor every data point for the first few participants Ongoing: audit percentage of forms Pay extra attention to key variables This is why you check… Identify what data have been collected  For each Subject at each Visit:  Questionnaires  Imaging, labs results  Other external data Missing data: can you still get it? Data entry: 1st entry/ 2nd entry/ reconciled Data cleaning/ QA / auditing “Clean” frozen datasets Do simple checks       Frequency (count) and distribution (range) of each and every variable Do crosstabs of variables where appropriate What is missing? What is out of range? What contradicts (e.g., pregnant males) Are there systemic problems? Medications are never easy Validated scale agreed upon at trial start (note 1-3 coding) 13 Perform Systematic Data Audits    Audit Example (real data) Data forms and source documents are compared with database on X % of forms Set an “acceptable” error rate. For example:  0.1% for key variables  0.5% overall If audit yields a larger error rate, you must check and correct the database Pay Extra Attention To Key Data Entered under incorrect ID? Be sure to pay particular attention to key data points where applicable.  Query all entries of critical variables (e.g., primary outcome)  Extra attention to problematic variables (e.g., time-line-follow-back)  Query all Serious Adverse Events ? Derived Variables Creating a Derived Variable Many analyses require creation of a derived variable from multiple data points  Be especially careful in creating derived variables  Include all relevant data elements  Don’t forget to account for missing data  Be sure to look at frequencies and crosstabs of derived variables prior to including in models Q1. Does child smoke Q2. Do household members smoke? Q1. Unprotected sex primary partner? Q2. Unprotected sex with casual partner? Q3. Do caretakers smoke? Q3. Share needles? New Var: Smoke_Exp New Variable: HIV_Exp 14 Sample SAS Code for Derived Variable Sample SAS Code for Derived Variable Any_Exp Yes No Total Site 1 50 40 90 Site 2 20 70 90 Total 70 110 180 Any_ExpM Yes No Missing Total Site 1 50 40 Site 2 20 20 50 90 Total 70 60 50 180 90 What’s up with the missing values?  Go back and look at forms:    What are the implications?    A Caution About New Technology Is there an explanation? Is the missing data differential? Example: There are 2 sites and all the forms with missing values came from a single site Did you find this problem early enough to correct it? This is why you check “early and often” Smartphones Too soon, not ready for prime time Too late, becoming obsolete 15 Benefits of smartphones       Electronic data capture Secure if you use to connect to website with encryption SMS (text messaging) in addition to linking to web-form Easy to carry Everyone wants one “Sexy” so funders like the idea Data Security - General     Keep paper records should be kept in locked cabinets and/or offices Store identifiers like names and addresses separate from clinical data Keep particularly sensitive data apart from other identifiers (e.g., SSN) – in a separate file, by ID Do not collect sensitive data unless you really need it Smartphone challenges        Can be Expensive (hardware, data plans) Cannot encryption text messages One question per screen Small screens make view some question types difficult (e.g., grids) Navigating around questionnaire (going back) is challenging Battery life is short Attractive to thieves and easily stolen Data Security - Hardware     Password protect all computers Set to automatically timeout if inactive Encrypt laptops, flash-drives and other storage devices when possible Do not put identifiable data on portable media (e.g., CDs, flash-drives) unless password protected, preferably encrypted Take Home Message      Your team should include someone who understands data issues Budget for data management Planning ahead results in fewer revisions Check your data early and often If you do things correctly from the beginning:    It is less work It is less expensive You are more likely to discover the truth at the end 16