Using machine learning and natural language processing in triage for prediction of clinical disposition in the emergency department

Table 3 Prediction ability of seven machine learning models and reference for internal validation at CMUH

	AUROC	p-value*	BS	F1 score	Accuracy	Sensitivity	PPV	Specificity	NPV
Primary outcome
LGB	0.937	-	0.089	0.494	0.954	0.589	0.425	0.968	0.983
CatBoost	0.935	0.06	0.089	0.479	0.950	0.602	0.398	0.964	0.984
GB	0.932	0.06	0.094	0.479	0.951	0.584	0.406	0.966	0.983
LR	0.928	< 0.05	0.114	0.415	0.924	0.703	0.294	0.933	0.987
RF	0.926	0.50	0.072	0.500	0.955	0.591	0.433	0.969	0.983
ET	0.919	< 0.05	0.072	0.484	0.953	0.575	0.417	0.968	0.983
LR-TTAS	0.858	< 0.05	0.095	0.000	0.962	0.000	0.000	1.000	0.962
EPs	N/A	N/A	N/A	0.361	0.942	0.550	0.268	0.954	0.986
Secondary outcome
RF	0.847	-	0.089	0.580	0.782	0.655	0.519	0.820	0.889
ET	0.843	< 0.05	0.089	0.575	0.782	0.643	0.521	0.824	0.886
LGB	0.841	0.06	0.106	0.600	0.786	0.701	0.525	0.811	0.901
CatBoost	0.839	< 0.05	0.106	0.600	0.791	0.682	0.535	0.824	0.897
GB	0.829	< 0.05	0.111	0.586	0.782	0.675	0.518	0.814	0.894
LR	0.800	< 0.05	0.129	0.553	0.758	0.654	0.480	0.789	0.885
LR-TTAS	0.666	< 0.05	0.117	0.171	0.771	0.103	0.499	0.969	0.784
EPs	N/A	N/A	N/A	0.498	0.674	0.626	0.413	0.691	0.842

AUROC Area under receiver-operator curve, BS Brier score, CMUH China Medical University Hospital, EPs Emergency physicians, ET Extremely Randomized Trees, GB Gradient Boosting, LGB Light Gradient Boosting Machine, LR Logistic Regression, LR-TTAS Logistic Regression-Taiwan Triage Acuity scale, NPV Negative predictive value, PPV Positive predictive value, RF Random Forest
*The p-values in the table were derived using the DeLong test, which involved pairwise comparisons of each model’s AUROC with that of the model ranked directly above it in performance

ISSN: 1471-227X