Open Access

Assessment of central venous catheterization in a simulated model using a motion-tracking device: an experimental validation study

  • Julián Varas1,
  • Pablo Achurra1,
  • Felipe León1,
  • Richard Castillo1,
  • Natalia De La Fuente2,
  • Rajesh Aggarwal3, 4,
  • Leticia Clede2,
  • María P. Bravo2,
  • Marcia Corvetto2 and
  • Rodrigo Montaña2Email author
Annals of Surgical Innovation and Research201610:2

DOI: 10.1186/s13022-016-0025-6

Received: 23 October 2015

Accepted: 1 February 2016

Published: 12 February 2016

Abstract

Background

Central venous catheterization (CVC) is a basic requirement for many medical specialties. Simulated training in CVC may allow the acquisition of this competency but few reports have established a valid methodology for learning and acquiring procedural skills for CVC. This study aims to validate the use of a tracking motion device, the imperial college surgical assessment device (ICSAD), by comparing it with validated global rating scales (GRS) to measure CVC performance in a simulated torso.

Methods

Senior year medical students, first and last year residents (PGY1, LYR), and expert anesthesiologists performed a jugular CVC assessment in a simulated model (Laerdal IV Torso). A validated GRS for objective assessment of technical skills and motion analysis by ICSAD was used. Statistical analysis was performed through Mann–Whitney and Kruskal–Wallis tests for construct validity and Spearman correlation coefficients between the ICSAD and GRS scores for concurrent validity between both.

Results

32 subjects were recruited (10 medical students, 8 PGY1, 8 LYR and 8 experts). Total path length measured with ICSAD and GRS scores were significantly different between all groups, except for LYR compared to experts (p = 0.664 for GRS and p = 0.72 for ICSAD). Regarding jugular CVC procedural time, LYR and experts were faster than PGY1 and MS (p < 0.05). Spearman correlation coefficient was −0.684 (p < 0.001) between ICSAD and GRS scores.

Conclusions

ICSAD is a valid tool for assessment of jugular CVC since it differentiates between expert and novice subjects, and correlates with a validated GRS for jugular CVC in a simulated torso.

Keywords

Medical simulation Central venous catheterization Tracking motion device Objective skills assessment Medical training

Background

Central venous catheterization (CVC) is an essential competency required for many medical specialties [1, 2]. Annually, 5 million CVC are performed only in the United States [3], with serious and life-threatening complications occurring in up to 5–26 % [4] of the cases. These adverse events are inversely related to practitioner’s clinical experience [5].

Nowadays, simulated training for acquiring technical skills is becoming widespread for many medical specialties [2, 6, 7], from simple procedures like a venous puncture to more complex surgical procedures like a laparoscopic jejuno-jejunostomy [810], shortening the learning curves of residents while doing so in a safe and controlled environment [11]. Few studies have evaluated the acquisition of CVC proficiency through simulated models and their educational effectiveness [1115]. These studies have shown an increase in the rate of successful CVC [16] and a decrease of associated complications after the simulated training programs [17, 18].

The Accreditation Council for Graduate Medical Education (ACGME) recommends the use of simulation and checklists as the “most desirable” evaluation methods for the assessment of competency in procedural skills [13], and they have been commonly used for the evaluation of CVC [4, 16, 19, 20]. However structured global rating scales as the objective structured assessment of technical skill (OSATS) [21, 22] have demonstrated better assessment and discrimination of different levels of skills than checklists previously used [13].

The imperial college surgical assessment device (ICSAD) is a device that tracks hand-motion of the operator during a procedure, using sensors placed on the back of the trainee’s hands. Total path length of both hands is registered, providing an effective index of technical skill during a procedure [7, 23]. The ICSAD has demonstrated construct validity in many surgical procedures [23, 24] and it has been used for objective assessment of proficiency in anesthetic procedures such as labor epidural placement [25] and ultrasound-guided peripheral nerve blockade [26]. To our knowledge, there are no previous reports using motion-tracking devices to assess proficiency in CVC.

The validation of a motion-tracking device may complement the use of global rating scales (OSATS) in assessing better the differences between expert and novices procedural skills.

Therefore, the aim of this study is to establish the construct and concurrent validity of the tracking motion device (ICSAD) in assessing CVC in a simulated model.

Methods

The Institutional Review Board approved the study, and written informed consent was obtained from all participants.

Four different groups from Pontificia Universidad Católica de Chile Medical School were studied between November–December 2013. All students and postgraduate residents that were available for the period of assessment were recruited. Based on previous studies by our group a sample study of eight participants per group was calculated. All groups underwent a simulated jugular central venous catheterization assessment in an Adult IV simulated model, using tracking motion sensors attached on their hands (ICSAD) as shown in Fig. 1 [23, 24].
Fig. 1

Simulated jugular central venous catheterization assessment in an Adult IV simulated model, using a tracking motion device on their hands: the ICSAD

Residents keep an active registration of the number of procedures they complete during the 3 year residency. Prior jugular CVC experience of the different groups is shown in Table 1.
Table 1

Number of prior CVC insertion by each group evaluated

 

Students (n = 10)

PGY1 (n = 8)

LYR (n = 8)

Experts (n = 8)

Median (range)

0 (0–1)

4 (0–20)

50 (15–80)

>100 (200–400)

CVC Central venous catheterization, PGY1 first year post graduate residents, LYR last year residents

Prior to assessment

Before assessment, all groups were gathered at a 2 h master class where they were explained how to perform a jugular CVC in the simulated model. In addition, a DVD video was provided to each student with a step-by-step instructional guide emphasizing the key issues related to the procedure and most common mistakes.

Jugular CVC assessment

After the introductory class, all groups were assessed performing one jugular CVC in an Adult Laerdal IV bench model (Laerdal IV Torso; Laerdal Medical Corp, Wappingers Falls, NY) [13, 27]. Each task was video-recorded and then blindly assessed by three independent expert anesthesiologists using a validated OSATS global rating scale (Modified from Ma et al. [13]). Economy of movements was assessed using the ICSAD from the beginning of the technical procedure (total path length was measured in meters). Procedure time (in seconds) was also recorded. Inter-rater reliability between OSATS evaluators was calculated using Kappa coefficient (0–1) [28, 29].

Statistical analysis

Data was analyzed with the Statistical Package for the Social Sciences version 15.0 (SPSS, Chicago, IL, USA) using non-parametric tests. Mann–Whitney and Kruskal–Wallis tests were used for each variable and the results were exposed in median (range).

Spearman correlation coefficients between the ICSAD scores and the validated OSATS global rating scale scores were calculated to establish the concurrent validity of ICSAD [13]. Following the Cohen guidelines, a positive or negative value between 0.5 and 1.0 indicates a large effect, 0.3–0.5 indicates a medium effect, and 0.3–0.1 indicates a small effect [30]. P value was considered statistically significant when <0.05.

Results

A total of 32 subjects were recruited, divided in 10 medical students (MS), 8 PGY1, 8 LYR and 8 expert anesthesiologist. Inter-rater reliability was established between the three OSATS evaluators, obtaining a Kappa coefficient of 0.76 (CI 0.58–0.92).

Results of total path length measured with ICSAD, OSATS global rating scores and procedural time are shown in Table 2. Regarding ICSAD total path length, all groups had significant differences between them, except for LYR compared to experts (p = 0.664; Fig. 2a).
Table 2

JCVC assessment (bench model)

 

Students (n = 10)A

PGY1 (n = 8)B

LYR (n = 8)C

Experts (n = 8)D

p valueAB

p valueBC

p valueBD

p valueCD

GRS (8–32)

11.5 (8–28)

18 (13–27)

27 (17–32)

29 (24–32)

0.029

0.014

0.004

0.664

TPL (m)

48.5 (44–89)

43 (33–56)

35 (28–42)

34 (28–44)

0.028

0.015

0.04

0.72

Procedural time (s)

344 (218–609)

243 (121–571)

133 (111–339)

122 (116–201)

0.172

0.053

0.015

0.694

Comparison between final year medical students, PGY1 and LYR anesthesiology residents, and expert Anesthesiologists

GRS Global rating scores, TPL total path length, CVC central venous catheterization, PGY1 first year postgraduate residents, LYR last year residents

ABp values obtained when comparing columns A and B with Mann–Whitney test

BCp values obtained when comparing columns B and C with Mann–Whitney test

BDp values obtained when comparing columns B and D with Mann–Whitney test

CDp values obtained when comparing columns C and D with Mann–Whitney test

Fig. 2

Comparison between final year medical students, PGY1, LYR and expert in total path length (TPL) (a) and global rating scale (GRS) scores (b)

In OSATS median scores, there were significant differences between all groups, except for LYR compared to experts (p = 0.72; Fig. 2b).

Finally, concerning procedural time, there were no differences between MS compared to PGY1 (p = 0.172) and LYR compared to experts (p = 0.694). However, the last two had significant less procedural time than PGY1 and MS (p = 0.015).

Spearman correlation coefficient between the total path length measured with ICSAD and the validated OSATS global rating scale score demonstrated a strong correlation, with a Spearman correlation coefficient of −0.684 (p < 0.001).

Discussion

Central venous catheterization is commonly learned during residences trough the “see one, do one, teach one” approach [31]. This apprenticeship model requires that inexperienced residents perform the procedure on patients, in a clinical setting with few or none standardized methodology to teach or evaluate the procedure [31]. Therefore, this learning model does not ensure proficiency in practical skills and jeopardizes patients’ safety [32].

Surgical specialties have vast experience in objective assessment of technical skills for a procedure [7]. Global rating scales, specific checklists and motion analysis like ICSAD have been used to evaluate many surgical procedures, establishing a good correlation between scores obtained and surgeon’s competency level [7, 23, 24]. In the case of anesthesia, only few reports have used objective assessment of technical skills, demonstrating a good correlation between scores obtained with the assessment tools and the expertise level [19, 20]. Tracking motion devices like ICSAD have demonstrated construct validity in discriminating the different levels of expertise in anesthetic procedures, such as an epidural catheter insertion [25] or an ultrasound-guided peripheral nerve blockade [26].

This is the first validation study reporting the use of ICSAD as an assessment tool for jugular CVC. The exercise of this motion device in the evaluation of jugular CVC allows the obtainment of quantitative data that complements global rating scales for differentiating between novice and expert, thus, adding construct validity to the simulated model. Both of these tools were more useful in discriminating the level of expertise when compared to procedural time as an assessment measurement (Table 2).

In the case of concurrent validity of ICSAD for jugular CVC, a good correlation was achieved with the previous validated OSATS global rating scale. The ICSAD is an objective numeric tool, non-dependent of the evaluating teacher which reduces assessment bias.

No significant differences were found between experienced residents (LYR) and expert anesthesiologists in terms of total path length measured by ICSAD, global rating scale or time of procedure. A possible explanation for this result is that most LYR residents have already flattened their learning curves and achieved the minimum proficiency needed to perform this procedure. They are considered experienced non-experts and perform well on routine problems by unreflectively and automatically applying the standard technique [33]. Hayter et al. [25] had similar findings when they assessed the epidural catheter insertion in residents, and proposed to add variables such as non-standard patient scenarios in order to discriminate in a more subtle way the expertise level [25].

Our main study limitation is small sample groups, mainly due to local difficulties in gathering residents for an experimental protocol in our institution. However, there were statistically significant differences between the assessed groups when comparing the tracking motion measures; concluding that ICSAD may help differentiate between different skills level in CVC simulated assessment.

Finally, this is the first report establishing construct validity of the Laerdal IV Torso model. This training bench model, with ICSAD and GRS used as assessing tools, allows to discriminate between different levels of expertise. The technical skills gap observed in this simulated model between novices and experts provides learning opportunities for trainees, setting the cutoff scores to be achieved. Further predictive validity studies are needed in order to determine whether the skills acquired through the simulated training may or not transfer to real life scenario with patients.

In conclusion, ICSAD was correctly validated for assessing jugular CVC in a simulated model, as it discriminates between expert and novices and correlates with validated OSATS global rating scale. To have as many as possible instruments for evaluating procedural skills such a jugular CVC may improve the objectification of competency acquisition in real patients.

Abbreviations

CVC: 

central venous catheterization

GRS: 

global rating scales

ICSAD: 

imperial college surgical assessment device

LYR: 

last year residents

MS: 

medical students

OSATS: 

objective structured assessment of technical skill

PGY1: 

post-graduate year 1

Declarations

Authors’ contributions

JV, FL, RC, MC, RM helped design the study, conduct the study, analyze the data, and write the manuscript. PA helped design the study, analyze the data, and write the manuscript. NDLF helped design the study and analyze the data. RA helped analyze the data and technical support. LC, MPB helped conduct the study and analyze the data. MC helped design the study, conduct the study, analyze the data, and write the manuscript. JV, PA, FL, RC, MC has seen the original study data, reviewed the analysis of the data, and approved the final manuscript. NDLF, LC, MPB has seen the original study data, reviewed and approved the final manuscript. RA has seen the original study data, revised critically the article and approved the final manuscript. RM has seen the original study data, reviewed the analysis of the data, approved the final manuscript, and is the author responsible for archiving the study files. All authors read and approved the final manuscript.

Competing interests

This study was funded by the Anesthesiology Division, Faculty of Medicine, Pontificia Universidad Católica de Chile. RA is a consultant for Applied Medical. The rest of the authors report no competing interests.

Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Authors’ Affiliations

(1)
Experimental Surgery and Simulation Center, Department of Digestive Surgery, Clinic Hospital, School of Medicine, Pontificia Universidad Católica de Chile
(2)
Anesthesiology Division, Clinic Hospital, School of Medicine, Pontificia Universidad Católica de Chile
(3)
Department of Surgery, Faculty of Medicine, McGill University
(4)
Arnold and Blema Steinberg Medical Simulation Centre, Faculty of Medicine, McGill University

References

  1. Duffy FD, Holmboe ES. What procedures should internists do? Ann Intern Med. 2007;146:392–3.View ArticlePubMedGoogle Scholar
  2. Ma IW, Sharma N, Brindle ME, Caird J, McLaughlin K. Measuring competence in central venous catheterization: a systematic-review. Springerplus. 2014;17(3):33.View ArticleGoogle Scholar
  3. McGee DC, Gould MK. Preventing complications of central venous catheterization. N Engl J Med. 2003;348:1123–33.View ArticlePubMedGoogle Scholar
  4. Dong Y, Suri HS, Cook DA, Kashani KB, Mullon JJ, Enders FT, et al. Simulation-based objective assessment discerns clinical proficiency in central line placement: a construct validation. Chest. 2010;137:1050–6.View ArticlePubMedGoogle Scholar
  5. Sznajder JI, Zveibil FR, Bitterman H, Weiner P, Bursztein S. Central vein catheterization. Failure and complication rates by three percutaneous approaches. Arch Intern Med. 1986;146:259–61.View ArticlePubMedGoogle Scholar
  6. Aggarwal R, Mytton O, Derbrew M, Hananel D, Heydenburg M, Issenberg B, et al. Training and simulation for patient safety. Qual Saf Health Care. 2010;19:34–43.View ArticleGoogle Scholar
  7. Reznick R, MacRae H. Teaching surgical skills—changes in the wind. N Engl J Med. 2006;355:2664–9.View ArticlePubMedGoogle Scholar
  8. Varas J, Mejia R, Riquelme A, Maluenda F, Buckel E, Salinas J, et al. Significant transfer of surgical skills obtained with an advanced laparoscopic training program to a laparoscopic jejunojejunostomy in a live porcine model: feasibility of learning advanced laparoscopy in a general surgery residency. Surg Endosc. 2012;26:3486–94.View ArticlePubMedGoogle Scholar
  9. Boza C, Varas J, Buckel E, Achurra P, Devaud N, Lewis T, et al. A cadaveric porcine model for assessment in laparoscopic bariatric surgery: a validation study. Obes Surg. 2013;23:589–93.View ArticlePubMedGoogle Scholar
  10. Aggarwal R, Boza C, Hance J, Leong J, Lacy A, Darzi A. Skills acquisition for laparoscopic gastric bypass in the training laboratory: an innovative approach. Obes Surg. 2007;17:19–27.View ArticlePubMedGoogle Scholar
  11. Okuda Y, Bryson EO, DeMaria S Jr, Jacobson L, Quinones J, Shen B, et al. The utility of simulation in medical education: what is the evidence? Mt Sinai J Med. 2009;76:330–43.View ArticlePubMedGoogle Scholar
  12. Britt RC, Reed SF, Britt LD. Central line simulation: a new training algorithm. Am Surg. 2007;73:680–3.PubMedGoogle Scholar
  13. Ma IW, Zalunardo N, Pachev G, Beran T, Brown M, Hatala R, et al. Comparing the use of global rating scale with checklists for the assessment of central venous catheterization skills using simulation. Adv Health Sci Educ Theory Pract. 2012;17:457–70.View ArticlePubMedGoogle Scholar
  14. Li T, Okuda Y. Impact of simulation training on central venous catheter insertion: a review. JEDM. 2012;1:1–9.Google Scholar
  15. Marmol MT, Braga FT, Garbin LM, Moreli L, dos Santos CB, de Carvalho EC. Central catheter dressing in a simulator: the effects of tutor’s assistance or self-learning tutorial. Rev Lat Am Enfermagem. 2012;20:1134–41.View ArticlePubMedGoogle Scholar
  16. Evans LV, Dodge KL, Shah T, Kaplan LJ, Siegel MD, Moore CL, et al. Simulation training in central venous catheter insertion: improved performance in clinical practice. Acad Med. 2010;85:1462–9.View ArticlePubMedGoogle Scholar
  17. Barsuk JH, Cohen ER, Feinglass J, McGaghie WC, Wayne DB. Use of simulation-based education to reduce catheter-related bloodstream infections. Arch Intern Med. 2009;169:1420–3.View ArticlePubMedGoogle Scholar
  18. Barsuk JH, McGaghie WC, Cohen ER, O’Leary KJ, Wayne DB. Simulation-based mastery learning reduces complications during central venous catheter insertion in a medical intensive care unit. Crit Care Med. 2009;37:2697–701.View ArticlePubMedGoogle Scholar
  19. Barsuk JH, Ahya SN, Cohen ER, McGaghie WC, Wayne DB. Mastery learning of temporary hemodialysis catheter insertion by nephrology fellows using simulation technology and deliberate practice. Am J Kidney Dis. 2009;1:70–6.View ArticleGoogle Scholar
  20. Friedman Z, Katznelson R, Devito I, Siddiqui M, Chan V. Objective assessment of manual skills and proficiency in performing epidural anesthesia-video-assisted validation. Reg Anesth Pain Med. 2006;31:304–10.View ArticlePubMedGoogle Scholar
  21. Martin JA, Regehr G, Reznick R, MacRae H, Murnaghan J, Hutchison C, et al. Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg. 1997;84:273–8.View ArticlePubMedGoogle Scholar
  22. Wilkinson J, Crossley J, Wragg A, Mills P, Cowan G, Wade W. Implementing workplace-based assessment across the medical specialties in the United Kingdom. Med Educ. 2008;42:364–73.View ArticlePubMedGoogle Scholar
  23. Datta V, Mackay S, Mandalia M, Darzi A. The use of electromagnetic motion tracking analysis to objectively measure open surgical skill in the laboratory-based model. J Am Coll Surg. 2001;193:479–85.View ArticlePubMedGoogle Scholar
  24. Aggarwal R, Dosis A, Bello F, Darzi A. Motion tracking systems for assessment of surgical skill. Surg Endosc. 2007;21:339.View ArticlePubMedGoogle Scholar
  25. Hayter MA, Friedman Z, Bould MD, Hanlon JG, Katznelson R, Borges B, et al. Validation of the Imperial College Surgical Assessment Device (ICSAD) for labour epidural placement. Can J Anaesth. 2009;56:419–26.View ArticlePubMedGoogle Scholar
  26. Chin KJ, Tse C, Chan V, Tan JS, Lupu CM, Hayter M. Hand motion analysis using the imperial college surgical assessment device: validation of a novel and objective performance measure in ultrasound-guided peripheral nerve blockade. Reg Anesth Pain Med. 2011;36:213–9.View ArticlePubMedGoogle Scholar
  27. Millington SJ, Wong RY, Kassen BO, Roberts JM, Ma IW. Improving internal medicine residents’ performance, knowledge, and confidence in central venous catheterization using simulators. J Hosp Med. 2009;4:410–6.View ArticlePubMedGoogle Scholar
  28. Van Nortwick SS, Lendvay TS, Jensen AR, Wright AS, Horvath KD, Kim S. Methodologies for establishing validity in surgical simulation studies. Surgery. 2010;147:622–30.View ArticlePubMedGoogle Scholar
  29. Carlson J, Tomkowiak J, Knott P. Simulation-based examinations in physician assistant education: a comparison of two standard-setting methods. J Physician Assist Educ. 2010;21:7–14.View ArticlePubMedGoogle Scholar
  30. Cohen J. Statistical power analysis. Curr Dir Psychol Sci. 1992;1:98–101.View ArticleGoogle Scholar
  31. Smith CC, Huang GC, Newman LR, Clardy PF, Feller-Kopman D, Cho M, et al. Simulation training and its effect on long-term resident performance in central venous catheterization. Simul Healthc. 2010;5:146–51.View ArticlePubMedGoogle Scholar
  32. Volpp KG, Grande D. Residents’ suggestions for reducing errors in teaching hospitals. N Engl J Med. 2003;348:851–5.View ArticlePubMedGoogle Scholar
  33. Moulton CA, Regehr G, Mylopoulos M, MacRae HM. Slowing down when you should: a new model of expert judgment. Acad Med. 2007;82:S109–16.View ArticlePubMedGoogle Scholar

Copyright

© Varas et al. 2016

Advertisement