The vampire challenge: Results of an international multi-institutional validation study to evaluate CT ventilation imaging algorithms.

Abstract

Purpose: CT ventilation imaging (CTVI) is being used to achieve functionally- adaptive lung cancer radiation therapy in three clinical trials (NCT02528942, NCT02308709, NCT02843568). We have built VAMPIRE (Ventilation Archive for Medical Pulmonary Image Registration) to address the need for a common validation dataset to evaluate the accuracy of different CTVI algorithms. Here we present the results of the first VAMPIRE Challenge, launched in 2016. Methods: The VAMPIRE dataset includes 50 pairs of 4DCT and ground truth ventilation scans - 25 humans imaged with Galligas 4DPET/CT, 21 humans imaged with DTPA-SPECT and 4 sheep imaged with Xenon-CT. For the VAMPIRE Challenge, 16 subjects were set as the "training" data (with ground truth provided) and 34 subjects were set as the "test" data (with ground truth blinded). 7 groups downloaded the Challenge dataset and uploaded CTVIs based on deformable image registration (DIR) between the 4DCT inhale/exhale phases. Participants used at least one DIR method broadly classified into B-splines, Diffeomorphisms or Biomechanical modeling (BM), and at least one ventilation metric based on DIR evaluation of volume change ("DIR-DVol"), Hounsfield Unit change ("DIR-DHU"), or a "Hybrid" approach. All CTVIs were uploaded with minimal post-processing, and evaluated against the ground truth using the voxel-wise Spearman correlation coefficient rS. Results: Participants submitted 23 unique combinations of DIR/metric. The overall best performing combination was BM-DIR/ Hybrid, with rS values having a median (range) of 0.50 (0.24-0.73) across the 34 test subjects. By comparison the other DIR/metric combinations had median rS values between 0.04 (B-spline with DIR-DVol) and 0.38 (B-spline with DIR-DHU). Conclusion: VAMPIRE is a resource for analyzing and comparing new CTVI algorithms. The VAMPIRE Challenge reveals varying levels of CTVI accuracy depending on the choice of DIR and ventilation metric, highlighting the importance of validation studies as this technology is translated from academic centers to the clinic.59th Annual Meeting Program of the American Association of Physicists in Medicine, AAPM 2017. United States