Background: The Dutch Committee for the Evaluation of Oncological Agents (cieBOM) assesses the clinical benefit of systemic anti-cancer treatments (SACTs). For SACTs tested in non-randomized trials (NRTs), cieBOM primarily utilizes response-related thresholds as assessment criteria. As sufficiency of NRT-based evidence for benefit assessments is questionable, this study investigated whether and how NRTs can be used to assess the clinical benefit of new SACTs initially appraised by cieBOM based on randomized controlled trials (RCTs). Methods: Using the RCTs underpinning cieBOM recommendations issued between 2015 and 2017, we searched for matching NRTs and applied the NRT-related assessment criteria by cieBOM to them. We then compared the assessment outcomes to the respective RCT-based cieBOM recommendations. Further, we investigated how the assessments would change when applying different response-related thresholds and adding a progression-free survival (PFS) threshold. Results: For 13 of the 37 eligible recommendations, a matching NRT was found. Two treatments were assessed positively and six negatively; five treatments were non-assessable. Two positive recommendations matched a positive NRT-based assessment; one matching negative assessment was found, and one treatment could not be assessed based on either trial results. Adding a > 6 months PFS threshold decreased the number of non-assessable NRTs (five to two). Conclusions: Limited publications and inconsistent data reporting hampered the viability of NRTs for clinical benefit assessments of SACTs beyond the scope of rare indications. Further, response-related assessment criteria alone might not fully grasp the clinical benefit of novel SACTs. NRT-based assessments should be considered with caution due to uncertainty of the trial results
