|Year : 2022 | Volume
| Issue : 3 | Page : 186-194
Impact of Machine Learning Prediction on Intraoperative Transfusion in Cranial Operation: Classification, Regression, and Decision Curve Analysis
Thara Tunthanathip, Sakchai Sae-Heng, Thakul Oearsakul, Anukoon Kaewborisutsakul, Chin Taweesomboonyat
Division of Neurosurgery, Department of Surgery, Faculty of Medicine, Prince of Songkla University, Songkhla, Thailand
|Date of Submission||24-May-2022|
|Date of Decision||03-Jun-2022|
|Date of Acceptance||09-Jun-2022|
|Date of Web Publication||3-Oct-2022|
MD, PhD Thara Tunthanathip
Division of Neurosurgery, Department of Surgery, Faculty of Medicine, Prince of Songkla University, Songkhla 90110, Thailand
Source of Support: None, Conflict of Interest: None
| Abstract|| |
Objective: This study aimed to use machine learning (ML) for the prediction of intraoperative packed red cell (PRC) transfusion and the number of units of transfused PRC, as well as estimate the net benefit of the ML models through decision curve analysis. Methods: The retrospective cohort study was conducted on patients who underwent cranial operations. Clinical data and transfusion data were extracted. Supervised ML algorithms were trained and tested as ML classification for the prediction of intraoperative PRC transfusion and ML regression for predicting the number of transfused PRC units. Results: Out of 2683 patients, 42.9% of neurosurgical patients intraoperatively received PRC. Artificial neural network, gradient boosting classifier, and random forest were the algorithms that had high area under the receiver operating characteristic curve of 0.912, 0.911, and 0.909, respectively, in ML classification, while random forest with regression had the lowest root mean squared error and mean absolute error in ML regression. Conclusions: ML is one of the most effective approaches to developing clinical prediction tools that can enhance the efficiency of blood utilization. Additionally, ML has become a valuable tool in modern health technologies as the computerized clinical decision support systems assist the physician in decision-making in real-world practice.
Keywords: Decision curve analysis, intraoperative transfusions, machine learning, neurosurgical operations, prediction
|How to cite this article:|
Tunthanathip T, Sae-Heng S, Oearsakul T, Kaewborisutsakul A, Taweesomboonyat C. Impact of Machine Learning Prediction on Intraoperative Transfusion in Cranial Operation: Classification, Regression, and Decision Curve Analysis. Int J Nutr Pharmacol Neurol Dis 2022;12:186-94
|How to cite this URL:|
Tunthanathip T, Sae-Heng S, Oearsakul T, Kaewborisutsakul A, Taweesomboonyat C. Impact of Machine Learning Prediction on Intraoperative Transfusion in Cranial Operation: Classification, Regression, and Decision Curve Analysis. Int J Nutr Pharmacol Neurol Dis [serial online] 2022 [cited 2022 Dec 5];12:186-94. Available from: https://www.ijnpnd.com/text.asp?2022/12/3/186/357219
| Introduction|| |
Neurosurgical patients who undergo frequent cranial operations require intraoperative blood transfusions. Excessive blood loss during operation causes hypovolemia, hypotension, and reduced oxygen delivery to tissues that increases certain biomarkers such as thrombin-antithrombin III, fibrinopeptide A,, and activation of fibrinolysis which leads to diffuse bleeding that needs an intraoperative blood transfusion. In addition, tissue thromboplastin (TTP) is an integral membrane protein which causes disseminated intravascular coagulation (DIC) after brain injury., However, overpreparation of packed red cells (PRC) is requested more than exact usage for anticipating and providing a safe margin for unexpectedly massive blood losses. Therefore, requesting of PRC has been reported in prior studies that increased blood bank tasks, the wastage of blood products, and unnecessary costs. Chotisukarat et al. and Saringcarinkul and Chuasuwan reported the crossmatch to transfusion (C/T) ratios in craniotomy with tumor removal, endoscopic approach with tumor removal, and clipping aneurysm as 4 to 5, 11 to 13.7, and 11.2, respectively. In detail, a C/T ratio of >2.5 indicated ineffective blood preparation found in real-world practice.
The limitation of blood resources and donations has also occurred due to the COVID-19 pandemic. According to a study by Wang et al., during pandemic the number of blood donors in China decreased by 67%, while a study in the Eastern Mediterranean region found that in blood banks blood supply was decreased, ranging from 26% to 50%. Therefore, optimization between preoperative PRC requisition and intraoperative usage should always be considered.
Currently, machine learning (ML) is one of the prediction tools that has been applied in several surgical fields, including neurosurgery. Tunthanathip et al. used a random forest algorithm for predicting intracranial injury in pediatric traumatic brain injury (TBI) with an area under the receiver operating characteristic curve (AUC) of 0.80 (95% confidence interval, CI: 0.72–0.87). In addition, ML has been used for the prediction of transfusions in the literature review. Liu et al. used the CatBoost algorithm to predict PRC transfusion in patients who underwent mitral valve surgery and reported an AUC of 0.88 (95% CI: 0.84–0.90). While, Walczak and Velanovich used an artificial neural network algorithm for predicting perioperative transfusion with several datasets; the AUC of artificial neural network was reported to range between 0.81 and 0.85.
Based on the aforementioned, this research aimed to use ML for prediction in several aspects. ML classification was done for intraoperative PRC transfusion as dichotomous proposed, while the prediction of the number of transfused PRC units was conducted by various algorithms of ML regression. Moreover, decision curve analysis (DCA) was performed for estimating the value of the ML model beyond a high-risk threshold.
| Methods|| |
Study design and study population
Consecutive neurosurgical patients who underwent cranial operations between January 2014 and January 2019 at Songklanagarind hospital were included. Baseline clinical characteristics and surgical data were collected from electronic-based medical records. Patients with inaccessible or incomplete transfusion data were excluded. Additionally, the cut-offs of age were used as ages between 15 and 60 years old according to the World Health Organization.
Sample size calculation was performed for validation. Using a diagnostic test formula, the AUC of ML reported in a prior study to predict PRC transfusion was 0.88, and the prevalence of PRC transfusion for craniotomy with tumor removal operation was reported at 19.4% in a study by Chotisukarat et al. Therefore, the minimum total sample size needed was 250 patients for testing the predictive model.
The categorical variables were performed as frequencies and percentages using descriptive statistics, while the continuous variables were analyzed using mean and standard deviation (SD). The difference between groups was tested by chi-square test, and the comparison of the mean between two groups was analyzed using an independent t test. In addition, binary logistic regression was performed according to clinical characteristics, preoperative laboratories, and surgical information for the odds ratio (OR) and 95% CI. Therefore, the P-value < 0.05 was accepted as statistically significant.
Using the splitting method, the full dataset was randomly divided into two datasets: 70% of the total data were used for training and building the predictive model, while 30% of the remaining data were used for testing the model. For feature selection, the significant variables from the chi-square test and t test were chosen for training the ML models in various algorithms.
The supervised ML algorithms were used for classification with a fivefold cross-validation process as follows: naïve Bayes (NB), artificial neural network (ANN), support vector machine (SVM), k-nearest neighbor (KNN), decision tree (DT), random forest (RF), logistic regression (LR), and gradient boosting classifier (GBC). The parameters of each ML model were turned for the best performance using the “GridSearch CV” (scikit-learn developers) package. The performance of each model was estimated as follows: sensitivity, specificity, positive predictive value, negative predictive value, accuracy, and AUC with the receiver operating characteristic (ROC) curve. These algorithms served to predict intraoperative PRC transfusion as binary classifiers, while ML regression was used for the prediction of units of PRC transfusion as numeric results by linear regression, ANN, DT, RF, and GBC. Hence, correlation with scatter plot, R2, root mean squared error (RMSE), and mean absolute error (MAE) were calculated for estimating the performance of ML regression. ML was performed using Python version 3.8.7 (The Python Software Foundation) with “scikit-learn.” In addition, the best training model was developed and deployed as a web-based application using R version 4.0.3 with “shiny” package (R Foundation, Vienna, Austria).
Decision curve analysis
According to Zhang et al. and Vickers and Elkin, various classification ML models for predicting intraoperative transfusion were analyzed for standardized net benefit by DCA. The plots of DCA are created that comprise the standardized net benefit on the y-vertical axis was performed against the high-risk threshold and the cost:benefit (C:B) ratio on the x-horizontal axis.,, DCA was performed using R version 4.0.5 (The R Foundation for Statistical Computing; Vienna, Austria) with the “rmda” packages.
This study was approved by the human research ethics committee (REC 64-477-10-1), which waived the need for informed consent because of the observational nature of the retrospective study. Besides, the patients’ identification numbers were encoded before analysis.
| Results|| |
Clinical and radiological characteristics
Three thousand and twenty-one patients admitted to the neurosurgical ward for cranial operations were included; 338 patients who underwent spinal surgery were excluded from the analysis. Therefore, the remaining 2683 patients established the cohort. Baseline characteristics are shown in [Table 1]. More than half of the patients were male, and the mean age was 46.91 (SD: 20.60). Among them, the common neurosurgical conditions comprised a brain tumor, TBI, and cerebral aneurysm at 51.5%, 17.4%, and 15.4%, respectively. Craniotomy was the most common procedure in the present cohort with 40.9% of cases. Moreover, more than half of the cases (50.3%) involved had emergency operations. Preoperative hematologic laboratory findings are presented in [Table 2]. Mean hematocrit and hemoglobin were 37.72% (SD: 5.89) and 12.58 g/dL (SD: 2.07), respectively. More than two-thirds (42.9%) of the present cohort were intraoperatively transfused by PRC. For univariate analysis, variables presented a significant difference of proportion and mean between transfusion and nontransfusion groups as follows: age, male, renal failure, neurosurgical conditions, American Society of Anesthesiologists (ASA) classification, warfarin usage, neurosurgical operations, emergency condition, surgical site infection operation, hematocrit, hemoglobin, white blood cell (WBC) count, neutrophil-to-lymphocyte (N/L) ratio, platelet count, partial thromboplastin time ratio (PTR), international normalized ratio (INR), and estimated blood loss (EBL) by a principal neurosurgeon. According to binary logistic regression analysis, older patients significantly increased the risk of intraoperative transfusion (OR: 1.005, 95% CI: 1.001–1.01). Moreover, operations among patients with TBI, intracranial aneurysm, and stroke were significantly at high risk of intraoperative transfusion compared to tumor surgery.
|Table 1 Baseline characteristics of patient who underwent cranial operation (N = 2683)|
Click here to view
|Table 2 Preoperative hematologic laboratory and other variables (N = 2683)|
Click here to view
Machine learning prediction
For predictors in the ML models, we selected variables based on univariate analysis in the former step. Therefore, the 17 predictors were chosen to train for the ML model with various ML algorithms. These models were validated with a testing dataset for both ML classification and regression. For supervised ML, the performances of each algorithm are shown in Tables 3 and 4. The first three algorithms of ML classification that had the highest AUCs were ANN, GBC, and RF algorithms, as shown in Figure 1. For ML regression, Figure 2 shows the scatter plots compared with the predicted units of transfused PRC against the actual value of each algorithm. Ideally, the perfect positive relationship between them should demonstrate a 45-degree line in the scatter plot. As a result, the RF algorithm was the ML regression that had the highest Pearson correlation (0.77, P < 0.001), and R2 (0.60) with the lowest RMSE (1.17) and MAE (0.61). Additionally, we revealed the web-based application that may be a user-friendly tool for general practice, as shown in Figure 3.
|Figure 1 Receiver operating characteristic curves showing the performance of each machine learning algorithm.|
Click here to view
|Figure 2 Scatter plots of predicted units of transfused packed red cells plotted against actual units for each machine learning regression algorithm: (A) random forest, (B) linear regression, (C) gradient boosting classifier, (D) decision tree, (E) artificial neural network, and (F) k-nearest neighbors.|
Click here to view
|Figure 3 A web-based application to predict intraoperative transfusion as classification (red button), and the number of units of the packed red cell as regression (yellow button).|
Click here to view
Decision curve analysis of machine learning
The DCA graphs of the ML models for intraoperative PRC transfusion are presented in Figure 4. At the selected high-risk threshold of 0.1 (C:B ratio 1:9), the GBC, ANN, RF, and DT algorithms had a greater standardized net benefit than that of the KNN, SVM, and NB algorithms. In detail, the various ML models were compared with other default strategies, which comprised treat all and treat none strategies. The treat all strategy identified that all patients would be intraoperatively transfused, whereas the treat none strategy meant no one would receive PRC during surgery. Therefore, the ML model of the GBC, ANN, RF, and DT algorithms had clinical usefulness beyond the default strategies across a range of high-risk thresholds.
|Figure 4 Decision curves demonstrating the usefulness of machine learning in predicting intraoperative transfusion. The black horizontal line represents the standardized net benefit of transfusion to none, assuming that all patients would not be transfused. The grey line represents the benefit of providing transfusion to all patients. The colored lines represent the benefit of applying machine learning algorithms across a range of reasonable thresholds.|
Click here to view
| Discussion|| |
We found that the prevalence of intraoperative PRC transfusion was 42.9% of neurosurgical patients who underwent cranial operations. The result was same as in prior studies with transfusion risk of 10% to 45%. Therefore, the predictors of intraoperative PRC transfusion that we found were age, male, renal failure condition, neurosurgical conditions, ASA classification, preoperative warfarin use, neurosurgical operations, emergency condition, surgical site infection operation, hematocrit, hemoglobin, WBC, N/L ratio, platelet count, PTR, INR, and EBL. Patients with a brain tumor and TBI had a high proportion in the transfused group in the present study. These results are in agreement with previous research reports. Kisilevsky et al. comprehensively reviewed PRC transfusion in cranial surgery and found that TBI had a transfusion risk of 36%. This finding can be explained by TBI-induced hypercoagulable and hyperfibrinolytic states. The excessive fibrinolytic processes occur from TTP and lead to DIC following brain injury.,, In brain tumor surgery, the large diameter of a tumor, low preoperative hemoglobin, and hypervascularity tumor such as meningioma, were the risk factors for transfusion., The preoperative hematological markers such as low platelet count, prolonged prothrombin time, and prothrombin time have been reported as risk factors of transfusion based on a literature review., Moreover, the N/L ratio has been reported to be a predictor in several outcomes of neurosurgical disease in prior studies., Therefore, we observed that the mean N/L ratio of the transfused group was significantly higher than other groups. The exact mechanisms involved in the role of preoperative N/L concerning the transfusion risk need to be studied further.,
ML is one of the modern computational data analysis approaches that have been studied for clinical prediction in various outcomes, including transfusion prediction. Mitterecker et al. used the ML with several algorithms for predicting the transfusion of patients from the Western Australia database. From the binary classifiers of transfusion, ANN and GBC had high AUCs of 0.966 and 0.966, respectively. Furthermore, Walczak and Velanovich reported AUCs of ANN to prediction perioperative transfusion ranged between 0.814 and 0.858. These results from prior studies are in agreement with our findings, which showed that ANN, GBC, and RF algorithms were the first three highest AUC of 0.912, 0.911, and 0.909, respectively. However, the setting of the present study was slightly different from prior studies, as our study population specifically involved neurosurgical patients.
For ML regression, the predictability of the present study for the number of transfused PRC units was acceptable, especially the ensemble learning algorithms such as GBC and RF. Our results are concordant with earlier studies. From a study by Mitterecker et al., ANN, GBC, and RF with regression were studied to predict the number of transfused PRC. These results revealed that GBC had the highest R2 of 0.176 and lowest RMSE of 16.094. In the field of neurosurgery, Inoue et al. used the extreme gradient boosting algorithm for predicting neurological outcomes in patients with cervical spinal cord injury. They reported an AUC of 0.867. While, Tunthanathip et al. reported that the RF algorithm had the highest AUC of 0.80 for predicting intracranial hematoma in pediatric TBI.
DCA is also a novel method for estimating the predictive models, and/or diagnostic testing has been increasingly performed in clinical research. Van Calster et al. used DCA for decision-making biopsy in patients who should undergo a procedure to diagnose prostate cancer before treatment. The predictive model for diagnose of high-risk prostate cancer was compared with default strategies (treat all and treat none). The results revealed that the predictive model was superior to all other strategies across the threshold.
As a result of the above, the ML models in the present study had the benefit and clinical usefulness to help physicians make better preoperative blood preparations. Moreover, these prediction models can influence the development of the Maximum Surgical Blood Order Schedule (MSBOS) in general practice. Generally, the MSBOS involves several calculation methods, such as 1.5-times of transfusion probability, or expert consensus for institutional guidelines.,, The ANN and ensemble learning are a challenge to apply in real-world practice for balancing blood preparation and utilization and developing MSBOS.
This study has some limitations that should be considered. Some bias may have appeared from the nature design of the retrospective study. For future studies, external validation with prospective unseen data should present the actual performance of the predictive model. Moreover, multicollinearity may be recognized in several parameters. However, we aimed to use all significant predictors from screening univariate analysis because more dimensions of predictors in the training process supported the learning processes and predictability., To the best of the authors’ knowledge, this is the first paper that proposed ML-based prediction tools to calculate the predicted units of PRC in neurosurgical operation. The study of impact analysis of ML to optimize blood preparation and build MSBOS should be conducted in the future. The health and economic benefits of ML implication should be applied as MSBOS could be performed in the future through a cost-effectiveness analysis or the study of direct cost reduction. In addition, the present study proposed the web application built in the ML model in the cloud server that simplified the predictive model to use for the external validation by other hospitals and support physicians in the real-world practice as the computerized clinical decision support systems (CDSS). From a systematic review of Souza et al., the study reported that CCDSSs effectively enhanced the process of care such as screening and treatment and impacted patient outcomes, costs of care, and patient safety.
| Conclusion|| |
ML is one of the most effective approaches in creating clinical prediction tools that can resolve the over-crossmatch of blood products and enhance the efficiency of blood utilization.
Financial support and sponsorship
This study received a grant provided by the Faculty of Medicine, Prince of Songkla University. (Grant no. 64-079-1).
Conflict of interest
There are no conflicts of interest.
| References|| |
Menges T, Wagner RM, Welters I, Ruwoldt R, Boldt J, Hempelmann G. The role of the protein C-thrombomodulin system and fibrinolysis during cardiovascular surgery: influence of acute preoperative plasmapheresis. J Cardiothorac Vasc Anesth 1996;10:482-89.
Pabinger I, Fries D, Schöchl H, Streif W, Toller W. Tranexamic acid for treatment and prophylaxis of bleeding and hyperfibrinolysis. Wien Klin Wochenschr 2017;129:303-16.
Gerald AG. Update on hemostasis: neurosurgery. Surgery 2007;142:S55-60.
Chotisukarat H. Effectiveness index of preoperative blood preparation for elective neurosurgery at Prasat Neurological Institute. Thai J Anesthesiol 2017;43:232-40.
Saringcarinkul A, Chuasuwan S. Maximum surgical blood order schedule for elective neurosurgery in a University Teaching Hospital in Northern Thailand. Asian J Neurosurg 2018;13:329-35. doi: 10.4103/ajns.AJNS_104_16
] [Full text]
Zewdie K, Genetu A, Mekonnen Y, Worku T, Sahlu A, Gulilalt D. Efficiency of blood utilization in elective surgical patients. BMC Health Serv Res 2019;19:804.
Wang Y, Han W, Pan L et al.
Impact of COVID-19 on blood centers in Zhejiang province China. Vox Sang 2020;115:502-06. doi: 10.1111/vox.12931
Al-Riyami AZ, Abdella YE, Badawi MA et al.
The impact of COVID-19 pandemic on blood supplies and transfusion services in Eastern Mediterranean Region. Transfus Clin Biol 2021;28:16-24. doi: 10.1016/j.tracli.2020.11.002
Tunthanathip T, Duangsuwan J, Wattanakitrungroj N, Tongman S, Phuenpathom N. Comparison of intracranial injury predictability between machine learning algorithms and the nomogram in pediatric traumatic brain injury. Neurosurg Focus 2021;51:E7.
Liu LP, Zhao QY, Wu J et al.
Machine learning for the prediction of red blood cell transfusion in patients during or after liver transplantation surgery. Front Med 2021;8:632210. doi: 10.3389/fmed.2021.632210
Walczak S, Velanovich V. Prediction of perioperative transfusions using an artificial neural network. PLoS One 2020;15:e0229450. doi: 10.1371/journal.pone.0229450
Negida A, Fahim NK, Negida Y. Sample size calculation guide − part 4: how to calculate the sample size for a diagnostic test accuracy study based on sensitivity, specificity, and the area under the roc curve. Adv J Emerg Med 2019;3:e33. doi: 10.22114/ajem.v0i0.158
Zhang Z, Rousson V, Lee WC et al.
Decision curve analysis: a technical note. Ann Transl Med 2018;6:308. doi: 10.21037/atm.2018.07.02
Vickers AJ, Elkin EB. Decision curve analysis: a novel method for evaluating prediction models. Med Decis Making 2006;26:565-74. doi: 10.1177/0272989 ( 06295361
Kerr KF, Brown MD, Zhu K, Janes H. Assessing the clinical impact of risk prediction models with decision curves: guidance for correct interpretation and appropriate use. J Clin Oncol 2016;34:2534-40.
Sande SZ, Li J, D’Agostino R, Yin Wong T, Cheng CY. Statistical inference for decision curve analysis, with applications to cataract diagnosis. Stat Med 2020;39:2980-3002. doi: 10.1002/sim.8588
Allyn J, Allou N, Augustin P et al.
A comparison of a machine learning model with EuroSCORE II in predicting mortality after elective cardiac surgery: a decision curve analysis. PLoS One 2017;12:e0169772. doi: 10.1371/journal.pone.0169772
Kisilevsky A, Gelb AW, Bustillo M, Flexman AM. Anaemia and red blood cell transfusion in intracranial neurosurgery: a comprehensive review. Br J Anaesth 2018;120:988-98. doi: 10.1016/j.bja.2017.11.108
Zhang J, Zhang F, Dong JF. Coagulopathy induced by traumatic brain injury: systemic manifestation of a localized injury. Blood 2018;131:2001-6. doi: 10.1182/blood-2017- 11-784108.
Bhatnagar S, Udaya B, Rao GS. An audit of blood transfusion in elective neurosurgery. Indian J Anaesth 2007;51:200-4. [Full text]
Mebel D, Akagami R, Flexman AM. Use of tranexamic acid is associated with reduced blood product transfusion in complex skull base neurosurgical procedures: a retrospective cohort study. Anesth Analg 2016;122:503-8. DOI doi: 10.1213/ANE. 0000000000001065.
Drews RE. Critical issues in hematology: anemia, thrombocytopenia, coagulopathy, and blood product transfusions in critically ill patients. Clin Chest Med 2003;24:607-22. doi: 10.1016/s0272-5231(03)00100-x
Griesdale DE, Sekhon MS, Menon DK et al.
Hemoglobin area and time index above 90 g/L are associated with improved 6-month functional outcomes in patients with severe traumatic brain injury. Neurocrit Care 2015;23:78-84. doi: 10.1007/s12028-014- 0096-3.
Giede-Jeppe A, Reichl J, Sprügel MI et al.
Neutrophil-to-lymphocyte ratio as an independent predictor for unfavorable functional outcome in aneurysmal subarachnoid hemorrhage. J Neurosurg 2019;132:400-7. doi: 10.3171/2018.9.J NS181975.
Mitterecker A, Hofmann A, Trentino KM et al.
Machine learning-based prediction of transfusion. Transfusion 2020;60:1977-86. doi: 10.1111/trf.15935
Celikbilek A, Ismailogullari S, Zararsiz G. Neutrophil to lymphocyte ratio predicts poor prognosis in ischemic cerebrovascular disease. J Clin Lab Anal 2014;28:27-31. doi: 10.1002/jcla.21639
Inoue T, Ichikawa D, Ueno T et al.
XGBoost, a machine learning method, predicts neurological recovery in patients with cervical spinal cord injury. Neurotrauma Rep 2020;1:8-16. doi: 10.1089/neur.2020.0009
Van Calster B, Wynants L, Verbeek JF et al.
Reporting and interpreting decision curve analysis: a guide for investigators. Eur Urol 2018;74:796-804. doi: 10.1016/j.eururo.2018.08.038
Mcpherson RA, Pincus MR. Guidelines for ordering blood for elective surgery also referred to as maximum surgical blood order schedule (MSBOS). In Mcpherson RA, Pincus MR, eds. Henry’s clinical diagnosis and management by laboratory methods. 24th ed. Missouri: Elsevier 2021. pp. 1664-5.
Goldrick S, Sandner V, Cheeks M et al.
Multivariate data analysis methodology to solve data challenges related to scale-up model validation and missing data on a micro-bioreactor system. Biotechnol J 2020;15:e1800684. doi: 10.1002/biot.201800684
Taweesomboonyat C, Kaewborisutsakul A, Tunthanathip T, Saeheng S, Oearsakul T. Necessity of in-hospital neurological observation for mild traumatic brain injury patients with negative computed tomography brain scans. J Health Sci Med Res 2020;28:267-4
Tunthanathip T. Translational medicine in neurosurgery. Bangkok: Sahamit Pattana Printing; 2022.
Souza NM, Sebaldt RJ, Mackay JA et al.
Computerized clinical decision support systems for primary preventive care: a decision-maker-researcher partnership systematic review of effects on process of care and patient outcomes. Implement Sci 2011;6:87.
[Figure 1], [Figure 2], [Figure 3], [Figure 4]
[Table 1], [Table 2], [Table 3], [Table 4]