- Research article
- Open Access
Assessment of central venous catheterization in a simulated model using a motion-tracking device: an experimental validation study
Annals of Surgical Innovation and Researchvolume 10, Article number: 2 (2016)
Central venous catheterization (CVC) is a basic requirement for many medical specialties. Simulated training in CVC may allow the acquisition of this competency but few reports have established a valid methodology for learning and acquiring procedural skills for CVC. This study aims to validate the use of a tracking motion device, the imperial college surgical assessment device (ICSAD), by comparing it with validated global rating scales (GRS) to measure CVC performance in a simulated torso.
Senior year medical students, first and last year residents (PGY1, LYR), and expert anesthesiologists performed a jugular CVC assessment in a simulated model (Laerdal IV Torso). A validated GRS for objective assessment of technical skills and motion analysis by ICSAD was used. Statistical analysis was performed through Mann–Whitney and Kruskal–Wallis tests for construct validity and Spearman correlation coefficients between the ICSAD and GRS scores for concurrent validity between both.
32 subjects were recruited (10 medical students, 8 PGY1, 8 LYR and 8 experts). Total path length measured with ICSAD and GRS scores were significantly different between all groups, except for LYR compared to experts (p = 0.664 for GRS and p = 0.72 for ICSAD). Regarding jugular CVC procedural time, LYR and experts were faster than PGY1 and MS (p < 0.05). Spearman correlation coefficient was −0.684 (p < 0.001) between ICSAD and GRS scores.
ICSAD is a valid tool for assessment of jugular CVC since it differentiates between expert and novice subjects, and correlates with a validated GRS for jugular CVC in a simulated torso.
Central venous catheterization (CVC) is an essential competency required for many medical specialties [1, 2]. Annually, 5 million CVC are performed only in the United States , with serious and life-threatening complications occurring in up to 5–26 %  of the cases. These adverse events are inversely related to practitioner’s clinical experience .
Nowadays, simulated training for acquiring technical skills is becoming widespread for many medical specialties [2, 6, 7], from simple procedures like a venous puncture to more complex surgical procedures like a laparoscopic jejuno-jejunostomy [8–10], shortening the learning curves of residents while doing so in a safe and controlled environment . Few studies have evaluated the acquisition of CVC proficiency through simulated models and their educational effectiveness [11–15]. These studies have shown an increase in the rate of successful CVC  and a decrease of associated complications after the simulated training programs [17, 18].
The Accreditation Council for Graduate Medical Education (ACGME) recommends the use of simulation and checklists as the “most desirable” evaluation methods for the assessment of competency in procedural skills , and they have been commonly used for the evaluation of CVC [4, 16, 19, 20]. However structured global rating scales as the objective structured assessment of technical skill (OSATS) [21, 22] have demonstrated better assessment and discrimination of different levels of skills than checklists previously used .
The imperial college surgical assessment device (ICSAD) is a device that tracks hand-motion of the operator during a procedure, using sensors placed on the back of the trainee’s hands. Total path length of both hands is registered, providing an effective index of technical skill during a procedure [7, 23]. The ICSAD has demonstrated construct validity in many surgical procedures [23, 24] and it has been used for objective assessment of proficiency in anesthetic procedures such as labor epidural placement  and ultrasound-guided peripheral nerve blockade . To our knowledge, there are no previous reports using motion-tracking devices to assess proficiency in CVC.
The validation of a motion-tracking device may complement the use of global rating scales (OSATS) in assessing better the differences between expert and novices procedural skills.
Therefore, the aim of this study is to establish the construct and concurrent validity of the tracking motion device (ICSAD) in assessing CVC in a simulated model.
The Institutional Review Board approved the study, and written informed consent was obtained from all participants.
Four different groups from Pontificia Universidad Católica de Chile Medical School were studied between November–December 2013. All students and postgraduate residents that were available for the period of assessment were recruited. Based on previous studies by our group a sample study of eight participants per group was calculated. All groups underwent a simulated jugular central venous catheterization assessment in an Adult IV simulated model, using tracking motion sensors attached on their hands (ICSAD) as shown in Fig. 1 [23, 24].
Residents keep an active registration of the number of procedures they complete during the 3 year residency. Prior jugular CVC experience of the different groups is shown in Table 1.
Prior to assessment
Before assessment, all groups were gathered at a 2 h master class where they were explained how to perform a jugular CVC in the simulated model. In addition, a DVD video was provided to each student with a step-by-step instructional guide emphasizing the key issues related to the procedure and most common mistakes.
Jugular CVC assessment
After the introductory class, all groups were assessed performing one jugular CVC in an Adult Laerdal IV bench model (Laerdal IV Torso; Laerdal Medical Corp, Wappingers Falls, NY) [13, 27]. Each task was video-recorded and then blindly assessed by three independent expert anesthesiologists using a validated OSATS global rating scale (Modified from Ma et al. ). Economy of movements was assessed using the ICSAD from the beginning of the technical procedure (total path length was measured in meters). Procedure time (in seconds) was also recorded. Inter-rater reliability between OSATS evaluators was calculated using Kappa coefficient (0–1) [28, 29].
Data was analyzed with the Statistical Package for the Social Sciences version 15.0 (SPSS, Chicago, IL, USA) using non-parametric tests. Mann–Whitney and Kruskal–Wallis tests were used for each variable and the results were exposed in median (range).
Spearman correlation coefficients between the ICSAD scores and the validated OSATS global rating scale scores were calculated to establish the concurrent validity of ICSAD . Following the Cohen guidelines, a positive or negative value between 0.5 and 1.0 indicates a large effect, 0.3–0.5 indicates a medium effect, and 0.3–0.1 indicates a small effect . P value was considered statistically significant when <0.05.
A total of 32 subjects were recruited, divided in 10 medical students (MS), 8 PGY1, 8 LYR and 8 expert anesthesiologist. Inter-rater reliability was established between the three OSATS evaluators, obtaining a Kappa coefficient of 0.76 (CI 0.58–0.92).
Results of total path length measured with ICSAD, OSATS global rating scores and procedural time are shown in Table 2. Regarding ICSAD total path length, all groups had significant differences between them, except for LYR compared to experts (p = 0.664; Fig. 2a).
In OSATS median scores, there were significant differences between all groups, except for LYR compared to experts (p = 0.72; Fig. 2b).
Finally, concerning procedural time, there were no differences between MS compared to PGY1 (p = 0.172) and LYR compared to experts (p = 0.694). However, the last two had significant less procedural time than PGY1 and MS (p = 0.015).
Spearman correlation coefficient between the total path length measured with ICSAD and the validated OSATS global rating scale score demonstrated a strong correlation, with a Spearman correlation coefficient of −0.684 (p < 0.001).
Central venous catheterization is commonly learned during residences trough the “see one, do one, teach one” approach . This apprenticeship model requires that inexperienced residents perform the procedure on patients, in a clinical setting with few or none standardized methodology to teach or evaluate the procedure . Therefore, this learning model does not ensure proficiency in practical skills and jeopardizes patients’ safety .
Surgical specialties have vast experience in objective assessment of technical skills for a procedure . Global rating scales, specific checklists and motion analysis like ICSAD have been used to evaluate many surgical procedures, establishing a good correlation between scores obtained and surgeon’s competency level [7, 23, 24]. In the case of anesthesia, only few reports have used objective assessment of technical skills, demonstrating a good correlation between scores obtained with the assessment tools and the expertise level [19, 20]. Tracking motion devices like ICSAD have demonstrated construct validity in discriminating the different levels of expertise in anesthetic procedures, such as an epidural catheter insertion  or an ultrasound-guided peripheral nerve blockade .
This is the first validation study reporting the use of ICSAD as an assessment tool for jugular CVC. The exercise of this motion device in the evaluation of jugular CVC allows the obtainment of quantitative data that complements global rating scales for differentiating between novice and expert, thus, adding construct validity to the simulated model. Both of these tools were more useful in discriminating the level of expertise when compared to procedural time as an assessment measurement (Table 2).
In the case of concurrent validity of ICSAD for jugular CVC, a good correlation was achieved with the previous validated OSATS global rating scale. The ICSAD is an objective numeric tool, non-dependent of the evaluating teacher which reduces assessment bias.
No significant differences were found between experienced residents (LYR) and expert anesthesiologists in terms of total path length measured by ICSAD, global rating scale or time of procedure. A possible explanation for this result is that most LYR residents have already flattened their learning curves and achieved the minimum proficiency needed to perform this procedure. They are considered experienced non-experts and perform well on routine problems by unreflectively and automatically applying the standard technique . Hayter et al.  had similar findings when they assessed the epidural catheter insertion in residents, and proposed to add variables such as non-standard patient scenarios in order to discriminate in a more subtle way the expertise level .
Our main study limitation is small sample groups, mainly due to local difficulties in gathering residents for an experimental protocol in our institution. However, there were statistically significant differences between the assessed groups when comparing the tracking motion measures; concluding that ICSAD may help differentiate between different skills level in CVC simulated assessment.
Finally, this is the first report establishing construct validity of the Laerdal IV Torso model. This training bench model, with ICSAD and GRS used as assessing tools, allows to discriminate between different levels of expertise. The technical skills gap observed in this simulated model between novices and experts provides learning opportunities for trainees, setting the cutoff scores to be achieved. Further predictive validity studies are needed in order to determine whether the skills acquired through the simulated training may or not transfer to real life scenario with patients.
In conclusion, ICSAD was correctly validated for assessing jugular CVC in a simulated model, as it discriminates between expert and novices and correlates with validated OSATS global rating scale. To have as many as possible instruments for evaluating procedural skills such a jugular CVC may improve the objectification of competency acquisition in real patients.
central venous catheterization
global rating scales
imperial college surgical assessment device
last year residents
objective structured assessment of technical skill
post-graduate year 1
Duffy FD, Holmboe ES. What procedures should internists do? Ann Intern Med. 2007;146:392–3.
Ma IW, Sharma N, Brindle ME, Caird J, McLaughlin K. Measuring competence in central venous catheterization: a systematic-review. Springerplus. 2014;17(3):33.
McGee DC, Gould MK. Preventing complications of central venous catheterization. N Engl J Med. 2003;348:1123–33.
Dong Y, Suri HS, Cook DA, Kashani KB, Mullon JJ, Enders FT, et al. Simulation-based objective assessment discerns clinical proficiency in central line placement: a construct validation. Chest. 2010;137:1050–6.
Sznajder JI, Zveibil FR, Bitterman H, Weiner P, Bursztein S. Central vein catheterization. Failure and complication rates by three percutaneous approaches. Arch Intern Med. 1986;146:259–61.
Aggarwal R, Mytton O, Derbrew M, Hananel D, Heydenburg M, Issenberg B, et al. Training and simulation for patient safety. Qual Saf Health Care. 2010;19:34–43.
Reznick R, MacRae H. Teaching surgical skills—changes in the wind. N Engl J Med. 2006;355:2664–9.
Varas J, Mejia R, Riquelme A, Maluenda F, Buckel E, Salinas J, et al. Significant transfer of surgical skills obtained with an advanced laparoscopic training program to a laparoscopic jejunojejunostomy in a live porcine model: feasibility of learning advanced laparoscopy in a general surgery residency. Surg Endosc. 2012;26:3486–94.
Boza C, Varas J, Buckel E, Achurra P, Devaud N, Lewis T, et al. A cadaveric porcine model for assessment in laparoscopic bariatric surgery: a validation study. Obes Surg. 2013;23:589–93.
Aggarwal R, Boza C, Hance J, Leong J, Lacy A, Darzi A. Skills acquisition for laparoscopic gastric bypass in the training laboratory: an innovative approach. Obes Surg. 2007;17:19–27.
Okuda Y, Bryson EO, DeMaria S Jr, Jacobson L, Quinones J, Shen B, et al. The utility of simulation in medical education: what is the evidence? Mt Sinai J Med. 2009;76:330–43.
Britt RC, Reed SF, Britt LD. Central line simulation: a new training algorithm. Am Surg. 2007;73:680–3.
Ma IW, Zalunardo N, Pachev G, Beran T, Brown M, Hatala R, et al. Comparing the use of global rating scale with checklists for the assessment of central venous catheterization skills using simulation. Adv Health Sci Educ Theory Pract. 2012;17:457–70.
Li T, Okuda Y. Impact of simulation training on central venous catheter insertion: a review. JEDM. 2012;1:1–9.
Marmol MT, Braga FT, Garbin LM, Moreli L, dos Santos CB, de Carvalho EC. Central catheter dressing in a simulator: the effects of tutor’s assistance or self-learning tutorial. Rev Lat Am Enfermagem. 2012;20:1134–41.
Evans LV, Dodge KL, Shah T, Kaplan LJ, Siegel MD, Moore CL, et al. Simulation training in central venous catheter insertion: improved performance in clinical practice. Acad Med. 2010;85:1462–9.
Barsuk JH, Cohen ER, Feinglass J, McGaghie WC, Wayne DB. Use of simulation-based education to reduce catheter-related bloodstream infections. Arch Intern Med. 2009;169:1420–3.
Barsuk JH, McGaghie WC, Cohen ER, O’Leary KJ, Wayne DB. Simulation-based mastery learning reduces complications during central venous catheter insertion in a medical intensive care unit. Crit Care Med. 2009;37:2697–701.
Barsuk JH, Ahya SN, Cohen ER, McGaghie WC, Wayne DB. Mastery learning of temporary hemodialysis catheter insertion by nephrology fellows using simulation technology and deliberate practice. Am J Kidney Dis. 2009;1:70–6.
Friedman Z, Katznelson R, Devito I, Siddiqui M, Chan V. Objective assessment of manual skills and proficiency in performing epidural anesthesia-video-assisted validation. Reg Anesth Pain Med. 2006;31:304–10.
Martin JA, Regehr G, Reznick R, MacRae H, Murnaghan J, Hutchison C, et al. Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg. 1997;84:273–8.
Wilkinson J, Crossley J, Wragg A, Mills P, Cowan G, Wade W. Implementing workplace-based assessment across the medical specialties in the United Kingdom. Med Educ. 2008;42:364–73.
Datta V, Mackay S, Mandalia M, Darzi A. The use of electromagnetic motion tracking analysis to objectively measure open surgical skill in the laboratory-based model. J Am Coll Surg. 2001;193:479–85.
Aggarwal R, Dosis A, Bello F, Darzi A. Motion tracking systems for assessment of surgical skill. Surg Endosc. 2007;21:339.
Hayter MA, Friedman Z, Bould MD, Hanlon JG, Katznelson R, Borges B, et al. Validation of the Imperial College Surgical Assessment Device (ICSAD) for labour epidural placement. Can J Anaesth. 2009;56:419–26.
Chin KJ, Tse C, Chan V, Tan JS, Lupu CM, Hayter M. Hand motion analysis using the imperial college surgical assessment device: validation of a novel and objective performance measure in ultrasound-guided peripheral nerve blockade. Reg Anesth Pain Med. 2011;36:213–9.
Millington SJ, Wong RY, Kassen BO, Roberts JM, Ma IW. Improving internal medicine residents’ performance, knowledge, and confidence in central venous catheterization using simulators. J Hosp Med. 2009;4:410–6.
Van Nortwick SS, Lendvay TS, Jensen AR, Wright AS, Horvath KD, Kim S. Methodologies for establishing validity in surgical simulation studies. Surgery. 2010;147:622–30.
Carlson J, Tomkowiak J, Knott P. Simulation-based examinations in physician assistant education: a comparison of two standard-setting methods. J Physician Assist Educ. 2010;21:7–14.
Cohen J. Statistical power analysis. Curr Dir Psychol Sci. 1992;1:98–101.
Smith CC, Huang GC, Newman LR, Clardy PF, Feller-Kopman D, Cho M, et al. Simulation training and its effect on long-term resident performance in central venous catheterization. Simul Healthc. 2010;5:146–51.
Volpp KG, Grande D. Residents’ suggestions for reducing errors in teaching hospitals. N Engl J Med. 2003;348:851–5.
Moulton CA, Regehr G, Mylopoulos M, MacRae HM. Slowing down when you should: a new model of expert judgment. Acad Med. 2007;82:S109–16.
JV, FL, RC, MC, RM helped design the study, conduct the study, analyze the data, and write the manuscript. PA helped design the study, analyze the data, and write the manuscript. NDLF helped design the study and analyze the data. RA helped analyze the data and technical support. LC, MPB helped conduct the study and analyze the data. MC helped design the study, conduct the study, analyze the data, and write the manuscript. JV, PA, FL, RC, MC has seen the original study data, reviewed the analysis of the data, and approved the final manuscript. NDLF, LC, MPB has seen the original study data, reviewed and approved the final manuscript. RA has seen the original study data, revised critically the article and approved the final manuscript. RM has seen the original study data, reviewed the analysis of the data, approved the final manuscript, and is the author responsible for archiving the study files. All authors read and approved the final manuscript.
This study was funded by the Anesthesiology Division, Faculty of Medicine, Pontificia Universidad Católica de Chile. RA is a consultant for Applied Medical. The rest of the authors report no competing interests.