Purpose: Photon counting x-ray detectors (PCXD) may improve dose-efficiency but are hampered by limited count rate. They generally have imperfect energy response. Multi-layer ("in-depth") detectors have been proposed to enable higher count rates but the potential benefit of the depth information has not been explored. We conducted a simulation study to compare in-depth detectors against single layer detectors composed of common materials. Both photon counting and energy integrating modes were studied. Methods: Polyenergetic transmissions were simulated through 25cm of water and 1cm of calcium. For PCXD composed of Si, GaAs or CdTe a 120kVp spectrum was used. For energy integrating x-ray detectors (EIXD) made from GaAs, CdTe or CsI, spectral imaging was done using 80 and 140kVp and matched dose. Semi-ideal and phenomenological energy response models were used. To compare these detectors, we computed the Cramér-Rao lower bound (CRLB) of the variance of basis material estimates. Results: For PCXDs with perfect energy response, depth data provides no additional information. For PCXDs with imperfect energy response and for EIXDs the improvement can be significant. E.g., for a CdTe PCXD with realistic energy response, depth information can reduce the variance by 50%. The improvement depends on the x-ray spectrum. For a semi-ideal Si detector and a narrow x-ray spectrum the depth information has minimal advantage. For EIXD, the in-depth detector has consistent variance reduction (15% and 17%19% for water and calcium, respectively). Conclusions: Depth information is beneficial to spectral imaging for both PCXD and EIXD. The improvement depends critically on the detector energy response.