Background: The quality of transitional care is associated with important health outcomes such as rehospitalization and costs. The widely used Care Transitions Measure (CTM‐15) was developed with a classic test theory approach; its short version (CTM‐3) was included in the CAHPS Hospital Survey. We conducted a psychometric evaluation of both measures and explored whether item response theory (IRT) could produce a more precise measure. Methods and Results: As part of the Transitions, Risks, and Actions in Coronary Events Center for Outcomes Research and Education, 1545 participants were interviewed during an acute coronary syndrome hospitalization, providing information on general health status (Short Form‐36), CTM‐15, health utilization, and care process questions at 1 month postdischarge. We used classic and IRT analyses and compared the measurement precision of CTM‐15–, CTM‐3–, and CTM‐IRT–based score using relative validity. Participants were 79% non‐Hispanic white and 67% male, with an average age of 62 years. The CTM‐15 had good internal consistency (Cronbach's α=0.95) but demonstrated acquiescence bias (8.7% participants responded “Strongly agree” and 19% responded “Agree” to all items) and limited score variability. These problems were more pronounced for the CTM‐3. The CTM‐15 differentiated between patient groups defined by self‐reported health status, health care utilization, and care transition process indicators. Differences between groups were small (2 to 3 points). There was no gain in measurement precision from IRT scoring. The CTM‐3 was not significantly lower for patients reporting rehospitalization or emergency department visits. Conclusion: We identified psychometric challenges of the CTM, which may limit its value in research and practice. These results are in line with emerging evidence of gaps in the validity of the measure
