Reliability in Neurological Outcomes for ML Models

Intra-Rater and Inter-Rater Reliability of Manually Quantifying Neurological Outcome of Stroke Patients to be Used for Machine Learning Models INTRODUCTION The modified Rankin Scale (mRS) is a six point global outcome rating scale used to evaluate stroke patient outcomes by assessing levels of functional independence. The purpose of this report is to: manually quantify functional outcomes from patient discharge summaries analyze intra- and inter-rater reliability of manual mRS assessments establish a gold standard label for natural language processing models RESULTS Of the reviewed discharge summaries, 278 (91.2%) were easily assessable on first pass by Investigator A. *Dichotomous mRS split into functional outcomes of favorable (mRS ≤2) or unfavorable (mRS >2) mRS: modified Rankin Scale; N: number of reports reviewed; %: percentage agreement; k: kappa statistic Table 1: modified Rankin Scale scoring guidelines adapted from Zihni, et al¹ Figure 2: Intra-Rater mRS Assessment Agreement METHODS **Darker boxes indicate greater instances of agreement between raters 305 BMC stroke patient discharge summaries were graded using the mRS by a trained investigator (Investigator A) to determine if mRS can be reliably extracted from these reports 30 of the 305 discharge summaries were randomly selected for a second assessment. Percentage agreement and kappa values were calculated Intra-rater reliability Investigator A's two mRS assessments blindly compared; reliability determined using Cohen's kappa Inter-rater reliability Investigator A's first assessment compared to a second investigator's assessment (Investigator B); reliability determined using Cohen's kappa Investigator A's first assessment compared to mRS originally included in discharge summary; reliability determined using Fleiss' kappa Comparing Investigator A's first assessment, Investigator B, and mRS originally included in discharge summary; reliability determined using Fleiss' kappa Reliability when considering Figure 1: Comparisons for Intra- and Inter- Reliability favorable (mRS 0-2) vs. unfavorable (mRS 3-6) outcomes Figure 3: Inter-Rater mRS Assessment Agreement between Investigator A (1st Assessment) and Investigator B Most reviewed reports follow the trend line of perfect agreement, indicating good intraand inter-rater agreement Figure 4: Inter-Rater mRS Assessment Agreement between Investigator A (1st Assessment) and Discharge Summary DISCUSSION/CONCLUSION This study demonstrated that information extracted from stroke patient discharge summaries are sufficient to reliably extract an mRS. A relatively low percentage agreement between Investigators A and B may be due to the fact that Investigator A reviewed the patients' full discharge summaries while Investigator B only reviewed their discharge exams. This discrepancy is important to note when considering what information to train the machine learning model on to generate a mRS. However, the kappa value (0.71) demonstrates that inter-rater reliability is still substantial. Additionally, allowing a tolerance of 1 point shows great percentage agreement between the two investigators (85.2%). Overall, the results, most notably kappa values signifying moderate to near perfect agreement, indicate high intra-rater and inter-rater reliability, giving confidence that these mRS ratings can be employed as a gold standard label to train a natural language machine model. Ultimately, this would expedite evaluating functional outcomes at discharge, helping to better assess poststroke condition and to determine the course of treatment intervention.