A usual interstitial pneumonia (UIP) imaging pattern can be seen in both idiopathic pulmonary fibrosis (IPF) and connective tissue disease-related interstitial lung disease (CTD-ILD). The purpose of this multicenter study was to assess whether quantitative imaging data differ between IPF and CTD-ILD in the setting of UIP. Patients evaluated at two medical centers with CTD-ILD or IPF and a UIP pattern on CT or pathology served as derivation and validation cohorts. Chest CT data were quantitatively analyzed including total volumes of honeycombing, reticulation, ground-glass opacity, normal lung, and vessel related structures (VRS). VRS was compared with forced vital capacity percent predicted (FVC%) and percent predicted diffusing capacity of the lungs for carbon monoxide (DLCO%). There were 296 subjects in total, with 40 CTD-ILD and 85 IPF subjects in the derivation cohort, and 62 CTD-ILD and 109 IPF subjects in the validation cohort. VRS was greater in IPF across the cohorts on univariate (p < 0.001) and multivariable (p < 0.001–0.047) analyses. VRS was inversely correlated with DLCO% in both cohorts on univariate (p < 0.001) and in the derivation cohort on multivariable analysis (p = 0.003) but not FVC%. Total volume of normal lung was associated with DLCO% (p < 0.001) and FVC% (p < 0.001–0.009) on multivariable analysis in both cohorts. VRS appears to have promise in differentiating CTD-ILD from IPF. The underlying pathophysiological relationship between VRS and ILD is complex and is likely not explained solely by lung fibrosis.