SHADAC Resource on Required Evaluation Plans for DSRIP/Delivery System Reforms through 1115 Waivers

Introduction
SHADAC researchers have extensive experience providing technical assistance to states on payment and delivery system reform, often supporting the evaluation and monitoring of these activities. Recently, we have been working with a subset of states pursuing innovative policies under the Medicaid 1115 waiver opportunity, which requires a rigorous independent evaluation. To support peer learning around the evaluation of delivery and payment reform, SHADAC has reviewed publicly available evaluation plans related to one waiver type, the Delivery System Reform Incentive Payment (DSRIP) programs, and created a resource summarizing state evaluation approaches.  

Delivery System Reform Incentive Payment Programs[1]
Delivery System Reform Incentive Payment (DSRIP) programs are a subset of Medicaid 1115 waiver initiatives that aim to improve a state’s healthcare delivery system through performance-based payments to hospital systems and providers. The Centers for Medicare and Medicaid Services (CMS) approved the first DSRIP waiver in 2010.[2] At their core, DSRIP waivers provide incentive payments to hospitals and other providers to support various projects aimed at improving care quality and patient outcomes and reducing cost. To participate in the program, healthcare organizations pick from a menu of projects (crafted by CMS in collaboration with the state) to implement and evaluate within their system. Payments are made to providers based on milestones that are met and quality improvement achievements. Under this construct, CMS and states assume that the value added by systems, infrastructure, and payment reforms will outweigh the costs of implementation and incentive payment.

DSRIP Evaluation Activities and Requirements
There are both federal-led and state-led evaluations of DSRIP programs. At the federal level, CMS has contracted with Mathematica Policy Research to produce meta-analyses, case studies, and implementation reports for many 1115 waiver strategies, including specific evaluations of DSRIP programs.[3] Notably, CMS published two separate but related analyses of the implementation of DSRIP progress and outcome measurement and of key DSRIP demonstration design strategies to reward performance, as well as an Interim Evaluation Report of existing DSRIP demonstrations. Most recently, CMS published its approach to the final outcomes evaluation for DSRIP programs approaching conclusion.[4] In addition, CMS has developed state-level independent evaluation guidelines for states pursuing a variety eligibility, coverage, and delivery system reform waivers. States, including those with DSRIP waivers, are required to adhere to these evaluation guidelines and report regularly on both their monitoring and evaluation efforts.  

SHADAC’s Catalogue of DSRIP Evaluation Plans
With CMS’s heightened expectations and improved guidance for program evaluation, SHADAC felt that reviewing state evaluations of DSRIP programs could serve as a useful learning opportunity for states interested in delivery reform or other systems change efforts.[5] We have summarized these activities in a resource that includes program characteristics as well as evaluation design characteristics being used in the seven states for which evaluation plans or reports are publicly available. Evaluation design characteristics include:

• Program Implementation Timeframe
• State and Federal Funding
• Timeline for Evaluation Deliverables
• Evaluation Characteristics (Budget, contractor, logic model)
• Evaluation Methods (Qualitative Analysis, Randomized Controlled Trial, etc.)
• Evaluation Comparison Group Levels (Beneficiary, Program, Provider, Project, etc.)
• Beneficiary Comparison Groups
• Secondary Data Sources
• Primary and Qualitative Data Sources

States can use this resource in the following ways: 

• Search for programs implementing similar reforms and learn more about evaluation approaches.
• Identify whether there are any available evaluation reports.
• Learn independent contractor budgets.
• Determine whether states are proposing control groups in the event that a state is interested in reaching out to learn more details about the success of the approach.
• Determine whether states are proposing randomized control trials.

SHADAC used the following approach to selecting states for inclusion in the catalogue. We cross-referenced MACPAC’s summary of existing DSRIP or DSRIP-like waivers[6] with CMS’s federal evaluator’s list of DSRIP programs. We also confirmed that these states were in the Kaiser Family Foundation’s Medicaid Waiver Tracker.[7] To investigate how states are evaluating their DSRIP initiatives, we looked for evaluation designs published by the DSRIP states of interest on both CMS’s state waiver repository and state government websites. According to federal guidance, evaluation plans should be submitted to CMS within 180 days of waiver approval, and posted publicly within 30 days of evaluation plan approval. Of the DSRIP and DSRIP-like states, we identified seven states with publicly accessible and approved evaluation designs—California, Massachusetts, New Hampshire, New York, Oregon, Texas, and Washington. The work of three states (MA, CA, and TX) pre-dates the implementation of the Affordable Care Act in 2014.

If you have feedback on the usefulness of this catalogue, or have an interest in seeing a catalogue like this for other program types, please contact SHADAC at shadac@umn.edu.