Contracting for Managed Substance Abuse and Mental Health Services: A Guide for Public Purchasers
Technical Assistance Publication (TAP) Series 22

CHAPTER VI--Quality Management

Key issues in this chapter:
  • Framework for evaluating quality: structural, process, and outcome measures
  • Accreditation of MCOs and other providers
  • Report cards on MCOs
  • Measures of consumer and family satisfaction
  • Quality management systems for MCOs
  • Although cost containment has been the dominant catalyst behind the move to managed care systems, there is now strong and growing interest in evaluating and improving the quality of care. Quality of care has been defined as the degree to which health services for individuals and populations increase the likelihood of desired health outcomes and are consistent with current professional knowledge (IOM, 1990a).

    Driving the new interest in quality of care are improved methods for assessing quality; increased competition for contracts; concern about the effects of incentives in managed care contracts for undertreatment and restricted access; and a growing demand for accountability by Federal, State, and county governments (IOM, 1996; Meyer, 1997; Rosenbaum et al., 1997).

    Quality standards can vary, but there is a consensus emerging about the aspects of service delivery and treatment outcomes that should be measured to assess the system of care. The measures and mechanisms chosen in different managed care initiatives to assess and improve the quality of care and increase the accountability of managed care organizations (MCOs) will vary substantially depending on the circumstances (IOM, 1996).

    A purchaser of managed care can use a well-written contract to establish what standards of quality it expects from an MCO and to specify how quality will be defined, monitored, and managed. One of the most notable developments in purchaser contracting practices in recent years is the increased reliance on establishing minimum standards of MCO performance and the development of measures to determine whether MCOs are meeting those standards. The following language from a contract between the State of Massachusetts and a health maintenance organization (HMO) illustrates how a purchaser may provide an MCO with detailed contract language on both the standard it expects the MCO to meet and the measure that the purchaser will use to determine whether the MCO is meeting the standard (Rosenbaum et al., 1997):

    Standard: "The HMO shall maintain an ongoing formal process to develop and adopt clinical practice guidelines for conditions which have traditionally exhibited high cost and/or high variation among Provider treatment methodologies. Guidelines should combine the best available scientific evidence, outcomes, and expert opinion in the specialty for which the guideline is being developed and should be developed in conjunction with the Provider network to assure maximum acceptance. The HMO shall further demonstrate that such protocols have been implemented and that measurement of compliance with the guidelines is occurring.

    Measure: The HMO shall select two clinical practice guidelines, including at least one of the following: asthma management, prenatal care, substance abuse or pregnancy, and a second guideline of the HMO's choice. For each of the two clinical guidelines selected, the HMO shall document: (a) the process for the development and dissemination of clinical practice guidelines to participating Providers and members; (b) how the HMO incorporates scientific evidence, expert opinions, and the opinions of network providers and expert clinicians outside of the network into such guidelines; and (c) an ongoing evaluation process for the purpose of updating and revising the clinical practice guidelines, as indicated by current medical practice standards."

    This chapter presents a framework that may help purchasers in thinking about quality. It also discusses issues that purchasers should consider when developing provisions related to quality management in requests for proposals (RFPs) and contracts. The chapter covers the following topics:

    A. Framework for Evaluating Quality: Structural, Process, and Outcome Measures

    When purchasers are developing their strategies for using a contract with an MCO to ensure a certain level of quality, they may find Donabedian's conceptual framework for evaluating the quality of care useful (Donabedian, 1980, 1982, 1985). Donabedian identified three distinct categories of measures used to evaluate the quality of care: (1) structural measures; (2) process measures; and (3) outcome measures.

    Process measures are currently the dominant type of quality measures in managed behavioral health care contracts. As discussed below, however, there is growing interest in the use of outcome measures, and MCOs are increasingly allocating resources to outcome measures to maintain a competitive position in the industry. There is less emphasis on the use of structural measures of quality in managed behavioral health care contracts than on the use of either process or outcome measures. The use of each of these three types of measures of quality in managed behavioral health care is discussed further below.

    Framework for Evaluating Quality: Structural, Process, and Outcome Measures
    • Structural Measures

      Measures pertaining to the capacities, technologies, and infrastructure that make up the structure of care (e.g., management information system, number and types of staff, types of facilities, size and composition of an MCO's provider network)

    • Process Measures
    • Mesures pertaining to the administrative, clinical, and other processes by which care is provided

    • Outcome Measures

      Measures pertaining to the orucomes of care(e.g.,reductions in drug use, amelioration of symptoms of depression)

    • SOURCE: Donabedian,1980,1982,1985.



    1. Process Measures of Quality

    Process measures of quality--which in the field of behavioral health care are frequently referred to as "performance measures"--are generally of three main types: (1) measures of administrative processes (e.g., an MCO's customer service practices, and claims payment proficiency); (2) measures of clinical processes (e.g., an MCO's compliance with patient placement criteria, provision of relapse prevention training, continuity of care); and (3) measures of financial/utilization processes (e.g., population penetration rates, utilization patterns, and claims targets) (Oss, 1994).

    Process measures typically measure activities that are believed to lead to positive outcomes. The use of process measures is based on a belief that improving the process of care will yield improved outcomes. Purchasers should recognize, however, that not all process measures relate to, or even correlate with, outcome measures. Although the managed behavioral health care industry is attempting to develop a consensus on the most valuable process indicators to use (see the April 1997 issue of Behavioral Healthcare Tomorrow), little research has been directed toward the relationships between specific process measures and actual outcomes. Furthermore, little is known about the efficacy of process measures to predict outcomes consistently across patient groups (McLellan et al., 1996).

    Some of the potential benefits and limitations of process measures of quality are identified in Exhibit VI-1. To be most useful, process measures must be based on reliable, accurate, and complete data and relate clearly to the outcome being sought. If process measures are used properly, they can provide purchasers of managed substance abuse and mental health services, MCOs, consumers, and others with valuable information that can facilitate efforts to monitor, evaluate, and improve the quality of care.

    Exhibit VI-1.

    Process Measures: Benefits and Cautions
    Potential Benefits

    • Measures provide operational and measurable representations of performance.
    • Measures establish a base upon which to set standards to be attained by the MCO.
    • Measures increase overall accountability of the MCO.
    • Measures create opportunities to monitor and improve performance over time.
    • Measures provide a vehicle for implementing quality improvement initiatives or corrective action plans.
    • Measures provide a structure for financial incentives and sanctions.
    • Measures establish a framework for comparing performance in specified areas across health care systems.
    • Measures provide useful information to individuals and businesses to aid in their
    Cautions/Limitations
    • Measures can easily be based upon data that are not complete, accurate, reliable, and/or valid.
    • Measures based on data from an MCO's management information system (MIS) are subject to distortion, misinterpretation, and, in some cases, misrepresentation.
    • The validity of many measures may be questionable; some measures may be imperfect representations of the issues they attempt to address.
    • Case-mix risk adjustments, critical in accurately interpreting process and outcomes measures, are seldom included.

    2. Outcome Measures of Quality

    Outcome measures of quality have begun to receive much attention in the behavioral health care field. Although there is much evidence to support the effectiveness of prevention, treatment, and rehabilitation for substance abuse and mental health disorders (Hubbard, 1989; IOM, 1990a; McLellan et al., 1996; Simpson & Sells, 1990) evidence developed in other States or systems of care is often not perceived by State legislatures and the public as applicable to local systems of care. Consequently, the purchasers of behavioral health care services are often challenged to defend the expenditures to a skeptical legislature and public and to demonstrate that such services are resulting in positive outcomes and measurable societal benefits. Furthermore, some MCOs are beginning to propose outcome-based reimbursement arrangements as a means to circumvent what they perceive to be purchasers' overemphasis on price factors (Meyer, 1997).

    Neither the mental health nor the substance abuse fields have achieved a high degree of consensus about specific outcome measures to use and what indeed constitutes treatment success.(1) Thus, for example, the addiction treatment field continues to struggle with the controversial issue of whether treatment goals for persons with substance use disorders should be directed toward abstinence from all substance use or whether reduced use can be a viable goal for some populations.

    In the mental health field, a similar struggle is sometimes noted over whether symptom reduction or symptom recovery ("cure") should be the ultimate goal for those with mental illnesses. For an adult with severe mental illness, individually determined goals might include securing a job, regular housing, enough income to live on, and a social network. For a child with a serious emotional disturbance, goals might include an improved level of functioning in school, at home, and with peers. For an individual with severe substance dependence, goals may include decreased quantity and frequency of substance use, reduced symptoms, reduced involvement with the criminal justice system, lower medical costs, and improved vocational or employment status (as currently measured in the States of Minnesota, California, and Oregon) (McLellan et al., 1996).

    Exhibit VI-2 lists some research-based behavioral health care outcome measures from an Institute of Medicine (IOM) report Managing Managed Care (IOM, 1997). (Appendix G lists examples of other potential outcome measures in behavioral health care, along with several potential process measures.)

    Exhibit VI-2. Research-Based Behavioral Health Care Outcome Measures
    1. Percentage of individuals who show reductions in symptoms;

    2. Percentage of patients who show improved functioning;

    3. Number of patients who return to work;

    4. After return to work, average number of consecutive days worked without absences;

    5. For children and adolescents, number who return to school;

    6. For children and adolescents, average number of consecutive days in attendance at school;

    7. Number of clients returning to earlier levels of treatment;

    8. Number of clients who are able to live independently in the community;

    9. Number of clients whose substance-free status is validated through regular breath and urine testing; and

    10. Number of clients who increase participation in community activities.

    SOURCE: IOM, Managing Managed Care: Quality Improvement in Behavioral Health, 1997.



    The purchasers of managed behavioral health care should be prepared to include a well-defined plan for outcome measurement in the contract. Given that the development and use of outcome measures in managed behavioral health care systems are still in the early stages, a purchaser should use caution in selecting the outcome measures. Furthermore, the cost of obtaining outcome data must be weighed against the value of the data. Factors to take into consideration include the validity of the data and the reliability and accuracy of the information they provide, costs of data collection, its utility for making corrective decisions, and consistency with confidentiality requirements. In addition, outcomes must be selected that can be precisely measured, such as psychiatric hospitalization readmission rates, or the percentage of increase in school attendance among children, or a decrease in criminal justice system involvement. If available resources are very limited, the purchaser may wish to consider implementing the outcome measurement system in stages; building on it as resources become available and experience is acquired.

    A Resource for Information on Outcomes Monitoring

    Purchasers interested in developing outcome-measurement systems for substance abuse and mental health treatment services may wish to review the Treatment Improvement Protocol (TIP) developed by the Center for Substance Abuse Treatment (CSAT) titled Developing State Outcomes-Monitoring Systems for Alcohol and Other Drug Treatment (CSAT, 1995a). This TIP describes useful principles and techniques for designing outcomes-monitoring systems, establishing policies and viable infrastructures for monitoring and evaluating the outcome measures designed, and selecting a feasible number of outcome measures that best reflect the goals of the contract. Examples of outcomes-monitoring systems developed in various States are described in the TIP.



    In selecting outcome measures, the purchaser should be aware that different stakeholders--for example, the legislature, the courts, the health care system, consumer and family groups, various social and public health agencies--may take a substantial interest in the types of outcome measures used. Their goals and expectations, which often vary dramatically and may conflict, can result in strong pressures on the purchaser of managed behavioral health care services to devote finite resources to an expansive, all-inclusive set of measures that does not include a measure of whether individuals receiving care actually get better.

    Clinical outcomes for individual patients (e.g., improvements in cognitive functioning, reduction in cocaine use) may not be of primary interest to some stakeholders. In an analysis of outcomes desired by various stakeholders, McLellan and Weisner (1996) found that few stakeholders sought "direct" treatment outcomes; rather, they called for broader societal benefits such as "reduced crime, improved health status, prevention of human immunodeficiency virus infection, reduction of unsafe sexual practices, improved employment, and improved family functioning." Because measurement of outcomes is a complex undertaking and can require significant financial and staff resources, only a limited number of measures may be feasible. Alternatively, purchasers may want to limit the tracking of certain outcomes to specific population groups rather than all enrollees.

    3. Structural Measures of Quality

    Structural measures of quality refer to a wide range of "tangibles" that are part of the health care system and are believed to be positively related to outcome, or at least increase the chances that outcome will be positive. Structural measures of quality include capacities, technologies, and infrastructure that, when present, would be perceived by most to be related to higher quality processes and outcomes of treatment. Thus, for example, better facilities, telecommunications, office equipment, a management information system (MIS), and staffing in a hospital would seem to enhance the institution's capacity to provide high-quality services, all things being equal.

    Licensing of facilities usually relies heavily on structural measures of quality. Requirements concerning the size of rooms and hallways, sanitation, medical records, fire detection, staffing, etc. ensure that the physical facilities and clinical support systems are adequate to support the program. Although meeting these requirements does not ensure a good treatment outcome, not meeting them could potentially create conditions that would ensure a poor outcome, for example, an undetected fire. Accreditation standards (discussed later in this chapter) also consider structural features, such as staff training, staff size, medical record organization and content, infection control, and quality assurance/improvement systems. These standards are extremely important in encouraging the development of systems that will actually improve outcome.

    Facilities and programs are increasingly understanding the value of nicely landscaped grounds, good lighting, and decorative art. These structural features help create the perception, or an expectation, that the program has the potential to provide quality outcomes. These "placebo" effects may be quite effective at improving the potential for good outcomes.

    Purchasers should specify what structural measures will be required as part of the managed care system under development. For example, having comparable facilities for serving private- and public-pay clients, living in different areas, might be required.

    4. Quality Measures for Substance Abuse and Mental Health Services: Similarities and Difference

    It is important for purchasers of managed behavioral health care services to distinguish between quality measures that can be applied effectively to both substance abuse and mental health populations and those that are appropriate for one population or the other (as well as those that are appropriate for children). While many process and outcome measures can be applied both to mental health and to substance abuse treatment, an optimal set of measures for individuals with severe mental illness will differ in many ways from a set of measures for individuals with addictive illness. Similarly, the most appropriate measures for children and adolescents may be quite different from those used for adults.

    An IOM committee that studied managed behavioral health care identified the following general outcome areas that can be applied to a broad cross-section of consumers of both substance abuse and mental health treatment services (IOM, 1996): improvement in employment or vocational status, medical status, family and social functioning, legal problem status, cognitive functioning, and quality of life.

    Expected outcomes and the processes necessary to achieve given outcomes may differ depending on an individual's diagnosis, the severity of the problem, the drug(s) of abuse, age, and the stage of illness or recovery. Thus, for example, the purchasers should not expect all process and outcome measures to be equally applicable for severely mentally ill adults, individuals with long-term heroin addiction, adults with major depression, children with severe emotional disturbances, and adolescents with comorbid mental and addictive disorders.

    Structural, Process, and Outcome Measures of Quality. Purchasers may wish to address the following in RFPs and contracts:

    Include the input of consumers when establishing measures that are relevant to them; patient satisfaction data alone are insufficient and easily "gamed" by surveyors.

    Ensure that outcome measures are measurable and limited enough so they can be tracked and used to improve treatment.

    Identify, precisely define, and/or provide all performance measures that the MCO is required to develop and/or implement (e.g., claims paid, clinical records, satisfaction, and per capita cost analysis).

    Specify performance measures and standards to which incentives or sanctions are attached.

    Clarify purchaser quality monitoring activities, including reviews of performance measures, clinical records, grievance and appeal data, enrollment/disenrollment data, terminations, utilization and financial data, and management systems and procedures.

    Require that the MCO have or develop the technological capacity to fulfill all contractually stipulated responsibilities related to the collection and measurement of performance indicators and outcome measures.

    Define the process by which disagreements, misinterpretations, or ambiguities about measures will be resolved.

    Delineate the MCO's responsibilities about the integrity of the data and reporting requirements.

    Establish standards for validation and verification of reported performance data.

    Require compliance with State and/or Federal reporting requirements.

    Specify the processes and mechanisms by which the quality management staff and structures of the MCO and purchaser will communicate.

    Require that funds be set aside by the MCO to conduct outcome studies.

    Specify the required capabilities, flexibility, and resource intensity of the MCO's outcomes measurement system.

    Require the MCO to collect, analyze, and develop periodic reports with data on specified outcomes for specific subpopulations.

    Specify appropriate outcome measures for different populations (e.g., persons with long-term mental illness, individuals who are homeless and abuse substances, children with severe emotional disturbances).

    Require that the MCO collaborate with the State, county, and/or local mental health and/or substance abuse authorities in outcome-related evaluation initiatives.

    Require that the MCO make staff, data, and other relevant resources available to the evaluators representing the purchaser.

    Establish guidelines for phasing in outcome measurement, using established baseline measures wherever feasible.

    B. Accreditation of MCOs and Other Providers

    The purchaser of managed behavioral health care services must decide whether to require an MCO to have specific accreditation to be eligible to bid for the contract and whether to require accreditation for all or some types of network providers. A number of organizations systematically assess the clinical and administrative operations of individual providers, MCOs, and other health care delivery systems and institutions and accredit them if they comply with predetermined standards. Many of the standards used in accreditation are structural measures of quality (see earlier discussion in this chapter). Public and private payment programs often require accreditation as a condition of payment for covered services.

    The activities of four of the most well-known national accrediting organizations that review behavioral health care organizations--the Joint Commission on Accreditation of Healthcare Organizations (JCAHO), the National Committee for Quality Assurance (NCQA), the Commission on Accreditation of Rehabilitation (formerly the Commission on Accreditation of Rehabilitation Facilities, or CARF), and the Council on Accreditation of Services for Families and Children (COA)--are described in Exhibit VI-3.(2) In many cases, accreditation by these organizations has become a necessary, but not sufficient, requirement to be competitive in a managed care environment. However, very few MCOs serving persons with substance abuse and mental health disorders have been accredited by these organizations.



    Exhibit VI-3.

    National Accreditation Organizations



    Joint Commission on Accreditation of Healthcare Organizations (JCAHO): Accreditation Programs and Services: Behavioral Health Care Accreditation Program


    One Renaissance Boulevard
    Oakbrook Terrace, IL 60181
    Phone: (630) 792-5791
    Fax: (630) 792-5644
    Website: www.jcaho.org



    • Accredits staff model delivery systems in organizations providing mental health, substance abuse, and mental retardation/developmental disabilities services.
    • Developed accreditation guidelines for wide variety of network-based services systems, such as MCOs, health maintenance organizations (HMOs), preferred provider organizations, and provider-sponsored networks.
    • Currently evaluates and accredits more than 15,000 health care organizations.

     

    National Committee for Quality Assurance (NCQA): Behavioral Health Accreditation Program


    2000 L Street, NW, Suite 500
    Washington, DC 20036
    Phone: (202) 955-3500
    Fax: (202) 955-3599
    Website: www.ncqa.org



    • Accreditation focuses primarily on the highest organizational levels of an MCO's structure.
    • Develops quality standards and promotes improvement in quality of care provided in managed care plans and has recently issued accreditation standards for behavioral health care.




    Exhibit VI-3.

    National Accreditation Organizations (cont'd.)



    Commission on Accreditation of Rehabilitation (CARF) (Formerly the Commission on Accreditation of Rehabilitation Facilities)


    4891 East Grant Road
    Tucson, AZ 85712
    Phone: (520) 325-1044
    Fax: (520) 318-1129
    Website: www.carf.org



    • Accredits individual substance abuse programs, mental health programs, and community-based rehabilitation programs treating individuals with chronic mental and addictive disorders.
    • Built on strong consumer-centered philosophy that encourages active consumer involvement in all activities, including collaborating in the development of individual treatment plans.
    • Developed program standards regarding access issues.
    • As of June 1997, over 13,000 programs accredited.


    Council on Accreditation of Services for Families and Children (COA)


    120 Wall Street, 11th Floor
    New York, NY 10005
    Phone: (212) 797-3000
    Fax: (212) 797-1428
    Website: www.nn4youth.org



    • Created quality standards for more than 50 types of services more closely related to behavioral health care and social services than to the medical model of delivery.
    • Services include individual outpatient programs, day treatment programs, developmental disabilities services, day care/foster care for children, and numerous others.
    • Accredits about 3,000 social service programs and 1,000 behavioral health programs

    .



    Apart from national accrediting organizations, States accredit and license organizations, such as those based on the PACT (Program for Assertive Community Treatment) model, consumer-run clubhouses, and those providing partial hospitalization, intensive outpatient, and intensive case management services for persons with severe mental illnesses and substance abuse disorders. Accreditation of these organizations generally involves systematic assessments of clinical and administrative structures. In some cases, jurisdictions have moved to or are experimenting with national accreditation as a replacement for State or local licensing (e.g., Michigan, North Dakota pilot program, and the Veteran's Affairs mandate for CARF accreditation for all behavioral health and vocation programs under their Prescription for Change initiative).

    Mandating the accreditation of MCOs and/or service providers may have both benefits and drawbacks for a purchaser of managed substance abuse and mental health services. The obvious advantage of requiring MCOs and/or service providers to obtain this "seal of approval" is that it provides reasonable assurance that the clinical and administrative operations of the accredited organization have met basic standards of quality in terms of process and outcomes. However, several problems related to current accreditation practices have been noted (Horvath and Kaye, 1995; IOM, 1996; Bazelon Center for Mental Health Law, 1997):

    The costs of going through an accreditation process can be prohibitive, especially when personnel costs and time are considered and requiring accreditation could eliminate viable MCOs or programs from bidding on the contract.

    There is redundancy among accreditation standards from different accrediting agencies. National MCOs, managed behavioral health organizations (MBHOs), and other health care delivery systems often obtain accreditation from several different agencies to meet the requirements of various purchasers, which results in costly duplication of effort.

    Private accreditation standards are frequently vague and do not take into account many issues particularly important in the public sector (e.g., access, grievance procedures, and enrollee rights); additional standards may need to be developed.

    Accreditation standards often duplicate State licensing requirements.

    Current accreditation standards for behavioral health care are generally less refined than general health care measures and often emphasize structural and administrative process issues that do not adequately address the quality of clinical services.

    In response to these and other issues, the Institute of Medicine (IOM, 1996) has questioned the utility and validity of accreditation in the current health care environment and has encouraged accreditation agencies to focus their standards on the most relevant issues, to examine the use of "deeming" (accepting another entity's standards and/or review process in place of one's own in some or all areas), and to consolidate the multitude of accreditation standards from various organizations to reduce overlap and redundancy.

    Accreditation. Purchasers may wish to address the following in RFPs and contracts:

    Specify whether the MCO, network providers, and/or subsets of network providers must be accredited.

    Specify the type or types of accreditation required.

    Establish any timeframes within which an organization(s) must obtain accreditation, if not already accredited.

    Specify the consequences of losing accreditation, having the accreditation downgraded, or not achieving it within a specified period.

    Ensure compliance with relevant State licensing requirements and other applicable standards for MCOs, HMOs, and substance abuse and mental health treatment providers.

    Specify policies regarding "deeming," and clarify how providers can request that an existing accreditation be deemed acceptable.

    C. Report Cards on MCOs

    Purchasers are increasingly using "report cards" to monitor the performance of MCOs. Report cards present systematically organized data on standardized sets of measures, often with associated minimum standards, about MCOs and/or health care providers. Purchasers, MCOs, consumers, and others can then examine and compare objective information about the clinical and administrative processes of different MCOs.

    Widespread use of report cards on MCOs promises to increase standardization of measures and data-gathering procedures across a variety of behavioral health care systems. It should be noted, however, that most report cards are designed for adult services rather than services for children or adolescents.

    Report cards can be developed by any purchaser or organization. Four nationally recognized report cards that are increasingly being used by purchasers to assess the quality and performance of those MCOs managing behavioral health care services are these (see Exhibit VI-4):

    • The Health Plan Employer Data Information Set (HEDIS 3.0) prepared by the National Committee for Quality Assurance (NCQA);
    • Performance-Based Measures for Managed Behavioral Health Care Programs (PERMS) prepared by the American Managed Behavioral Healthcare Association (AMBHA);
    • County Behavioral Healthcare Measures prepared by the National Association of County Behavioral Health Directors (NACBHD); and
    • The Mental Health Statistics Improvement Program (MHSIP) prepared by the Center for Mental Health Services (CMHS).

    Industry norms such as HEDIS and PERMS are developed on the basis of data from the existing administrative database (e.g., telephone answering rates) that does not necessarily include data on certain types of outcomes that may be important to purchasers. Improvements in social functioning or employment, for example, are not part of an administrative database. Should a purchaser want to obtain data on these types of outcome measures, it must be willing to invest in both infrastructure development and the collection of data. The acquisition of standardized and reliable information about these outcomes would require a significant commitment and investment on the part of the purchaser.

    Exhibit VI-4.

    Four Nationally Recognized Report Cards



    Health Plan Employer Data Information Set (HEDIS 3.0)

    National Committee for Quality Assurance (NCQA)


    2000 L Street, NW, Suite 500
    Washington, DC 20036
    Phone: (202) 955-3500
    Fax: (202) 955-3599
    Website: www.ncqa.org





    • The most widely used set of general health care performance standards.
    • Incorporates measures designed for Medicaid and other populations.
    • Includes 71 measures in the reporting set of measures (to be reported in 1997), plus 33 measures in a "testing set".
    • Categorized into eight domains: accessibility, treatment effectiveness, stability, satisfaction, cost, degree of informed choices, use of services, and descriptive information.
    • A handful of behavioral health measures are in the reporting set and more are in the testing set, including:

    • - Outpatient followup after hospitalization,
      - Number of providers accepting new patients,
      - Utilization of different specified levels of care,
      - 90-day inpatient readmission rates,
      - Several related items in a health plan member satisfaction survey.
    • Precisely defined measures and standards.


    Performance-Based Measures for Managed Behavioral Healthcare Programs (PERMS)

    American Managed Behavioral Healthcare Association (AMBHA)


    700 13th Street, NW, Suite 950
    Washington, DC 20005
    Phone: (202) 434-4565
    Fax: (202) 434-4564
    Website: www.ambha.org


    • Includes 23 industry-standardized measures designed specifically for substance abuse and mental health treatment.
    • Measures access, consumer satisfaction, and quality.
    • Quality measures include systems effectiveness for substance abuse treatment and continuity of care for patients in treatment and detoxification.
    • Precisely defined measures.


    Exhibit VI-4. (cont.)

    Four Nationally Recognized Report Cards



    County Behavioral Healthcare Measures

    National Association of County Behavioral Health Directors (NACBHD)


    6000 Lamar Street, Suite 130
    Mission, KS 66202
    Phone: (913) 384-3535
    Fax: (913) 591-5653
    Website: www.naco.org/affils/afflpres.htm#16


    • Developed to monitor and improve the quality of substance abuse and mental health services.
    • Designed to be particularly relevant to service delivery at the county level.
    • Categorized into five domains of three to five measures each, with recommended tools to obtain those measures:

    • - Access,
      - Consumer satisfaction,
      - Consumer outcomes,
      - Intersystem outcomes, and
      - Utilization.



    Mental Health Statistics Improvement Program (MHSIP)


    Center for Mental Health Services (CMHS)


    Parklawn Building, Room 15-105
    5600 Fishers Lane
    Rockville, MD 20852
    Phone: (301) 443-0001
    Fax: (301) 443-1563
    Website: www.mentalhealth.org



    • Measures for comparing and evaluating access, appropriateness of treatment, outcomes, and prevention in the delivery of mental health services to persons with severe mental illness.
    • In testing and development in 20 States.
    • Instrument reflects significant consumer input into design and content.


    D. Measures of Consumer and Family Satisfaction

    Measurement of consumer and family satisfaction with behavioral health care services is increasingly becoming an integral component of quality monitoring efforts. Purchasers are therefore including these measurements as a quality-monitoring tool in contracts with MCOs. Indeed, the voices of consumers, their families, and others who support consumer recovery can provide valuable information to the purchaser about the strengths and weaknesses of an MCO and its network of providers. Consequently, a growing number of purchasers are attaching financial incentives or sanctions to MCO performance in this area (Petrila, 1996; Ruggeri, 1994).

    Assessment of consumer satisfaction can empower consumers and send a strong message to providers that such a focus is valued. Yet despite the value of consumer satisfaction measures for assessing quality, the purchaser should be aware that many challenges are associated with such measures and that findings related to satisfaction may need to be interpreted with caution. Some evidence suggests that even when MCOs primarily serving Medicaid beneficiaries have weaker ratings on performance measures for quality and access, enrollees are still as likely to indicate satisfaction with these plans as are individuals enrolled in private insurance plans that have significantly higher performance ratings (Rosenbaum et al., 1997).

    A recent literature review of the measurement of consumer satisfaction summarized the challenges currently associated with measurement of consumer satisfaction. These challenges are identified in the box below.


    Challenges in Measuring Consumer Satisfaction

    • There is no widespread consensus about the definition and appropriate measurement of satisfaction.
    • Very complex relationships exist between consumer satisfaction and quality of care, consumer demographics, provider profiles, and treatment outcomes.
    • Design and implementation of surveys for special populations (such as those with substance abuse and mental health problems treated in the public sector) are far from exact sciences.
    • Tools used to measure satisfaction with behavioral health treatment have been less rigorously evaluated for reliability and validity than tools for general health care.
    • Satisfaction surveys often yield high and undifferentiated levels of satisfaction.
    • Consumer responses are sensitive to the method of administration and can be affected by social desirability (the tendency of respondents to answer in a way that would please the person administering or issuing the survey).
    • Survey questions and results are often more a reflection of basic access (e.g., answering phones) and administrative efficiency of the MCO than of the quality of clinical services.
    • Surveys require sufficient brevity to generate adequate response rates, yet they must be specific enough to suggest actions that may be taken to improve the particular situation.
    • Some researchers have found no relationship between consumer satisfaction and quality of care and some have even found an inverse relationship.


    Given the challenges of conducting consumer satisfaction surveys, many purchasers are now including alternative mechanisms to assess the experience and satisfaction of enrollees with the MCO and its network providers. These include focus groups, in-depth interviews from samples of enrollees, and face-to-face surveys using consumers or family members as interviewers. Consumer satisfaction teams can also be used. These teams, which are consumer-run programs, serve as a monitoring and feedback mechanism in the system and can be given the authority to provide feedback to upper management of the MCO and/or purchaser. In addition, there is a current effort to adapt for behavioral health care the Consumer Assessment of Health Plan Study of the Federal Agency for Health Care Policy and Research (AHCPR).

    The purchaser may also wish to consider periodically assessing, or having the MCO assess, the satisfaction and opinions of key providers in the MCO's network as well as other organizations with which the MCO has an ongoing relationship. In the private sector, such mandated provider surveys are generally conducted by external parties. Surveyed organizations may include network providers; State and county mental health, substance abuse, social service, and other government agencies; consumer and family advocacy groups; and any other organization whose perspective would help illuminate the strengths and weaknesses of the MCO's performance. If the purchaser wants the MCO to conduct the surveys, the purchaser should define its role in approving the surveys and controlling how survey results can be used in marketing initiatives.

    Measuring Consumers' Satisfaction. Purchasers may wish to address the following in RFPs and contracts:

    Require the MCO to include input from providers, consumers, families, and other stakeholders in the development, implementation, evaluation, and refinement of satisfaction measurement systems and tools.

    Require that all satisfaction measurement tools and processes have the approval of the purchaser before implementation.

    Ensure that any surveys used, and processes followed, are methodologically sound (e.g., use adequate sampling and weighting techniques).

    Determine the frequency of surveys based on the intended use of the information.

    Specify that a standardized base of questions, data elements, and measurement processes (including guidance on sample size, timeframes, and response rate) must be used on surveys and other tools to facilitate comparisons across all levels of care (additional questions and data elements for different levels can be added to this base to allow programs and health care systems to individualize their tools).

    Require the MCO to supplement any surveys it conducts with additional, possibly more meaningful, mechanisms for understanding consumer views, such as focus groups or in-depth interviews conducted by family members or consumers.

    Require the use of nationally standardized report cards (e.g., HEDIS, PERMS) or the use of selected consumer-oriented measures, to allow comparison to other health care systems.

    Require periodic assessment of complaints, disenrollments, and requests to change facilities and providers.

    Require systematic followup of survey findings, including identification and investigation of sources of dissatisfaction, development and implementation of a corrective action plan, dissemination of findings to providers, enrollees, and legislative oversight committees, and a reevaluation of the survey process.

    Require regular reviews of the MCO's performance by conducting unbiased surveys of providers and other stakeholders.

    Ensure that providers and enrollees who provide negative feedback about the MCO are not threatened or penalized in any manner for their action.

    E. Quality Management Systems for MCOs

    A promising strategy for establishing and maintaining a managed care system based on quality principles is to use the RFP and contract to ensure that the MCO has a well-supported internal quality management (QM) system. The resources supporting and sophistication of an MCO's internal QM efforts vary tremendously among MCOs. However, MCOs are increasingly adhering to NCQA's comprehensive standards for QM. NCQA's accreditation summary reports provide detailed guidance to MCOs who seek accreditation.

    1. Internal Quality Management

    All Medicaid managed care contracts require the MCO to have an internal QM system. Rosenbaum et al. (1997) found that most State Medicaid programs set out extensive specifications for the structure and performance of such systems, not leaving it to the MCO's discretion.(3)

    Purchasers of managed behavioral health care should make sure that the RFP and contract clearly state the expectations, capabilities, and responsibilities of the QM system, even in cases where the details for developing and/or refining the MCO's internal QM system are primarily the responsibility of the MCO. Provisions requiring mandatory compliance with NCQA's standards will give the purchaser leverage to require changes in the MCO's internal QM system if needed. Other contract provisions may require the establishment of an MCO QM team; require an independent quality review council composed of providers, consumers, and other stakeholders; or require the MCO to develop and maintain a continuous quality improvement program.

    Internal Quality Management. Purchasers may wish to address the following in RFPs and contracts:

    Require the MCO to create a quality management team, accountable to the governing body of the organization.

    Develop, implement, and systematically refine a comprehensive quality management plan that is consistent with the RFP and all applicable State and Federal requirements.

    Incorporate a process for continuous improvement of quality across QM activities.

    Establish a system to monitor the completeness, accuracy, and appropriateness of service authorization decisions and ensure compliance with the utilization control requirements of the U.S. C.F.R. 456.

    Develop a comprehensive set of procedures for network providers and specify the MCO's responsibilities for management and reporting of serious incidents (i.e., deaths, suicide attempts, injurious assaults on provider premises, use of seclusion or restraints, medication errors, felony arrests, and convictions).

    Develop and implement systematic procedures for monitoring, managing, and improving the quality of individual network providers and of the provider network as a whole.

    Require the MCO to establish, facilitate, and empower a community-based monitoring council that includes providers, consumers, and family members.

    Require the MCO to actively cooperate with any external quality monitoring team as it develops, implements, and monitors quality improvement goals, objectives, and activities for the provider network and the MCO.

    2. External Quality Management

    Some purchasers may opt to supplement the MCO's QM program with an external agency that specializes in monitoring and auditing quality in health care settings to increase credibility and accountability (Huskamp, 1996).(4)

    HCFA requires that all States administering Medicaid managed care waiver programs under Section 1915 or Section 1115 of the Social Security Act provide for external quality assurance oversight.

    1. In September of 1997, the American College of Mental Health Administration (ACMHA), concerned that the rapid proliferation of process/performance measures and strategies is counterproductive for the behavioral health care field, held a summit in Santa Fe, New Mexico, to address outcome measurement. Representatives attended from such organizations as the Substance Abuse and Mental Health Services Administration (SAMHSA), the Institute of Medicine (IOM), the National Alliance for the Mentally Ill (NAMI), the Commission on Accreditation of Rehabilitation (CARF), the National Committee for Quality Assurance (NCQA), the American Managed Behavioral Healthcare Association (AMBHA), and the National Mental Health Association (NMHA). Participants were divided into working groups reflecting the following domains: prevention, access, process/performance, outcomes, and structure. Over the course of 2 days, each group produced a set of core values for each domain, with recommendations for next steps. Participants are actively collaborating with key agencies to refine the work initiated at the summit.

    2. For more information, see the Institute of Medicine report (1996).

    3. For a detailed analysis of this issue, see Table 5.1, Rosenbaum et al. (1997), which presents State Medicaid managed contract provisions addressing quality assurance.

    4. The Health Care Financing Administration (HCFA) mandates that all State Medicaid waiver programs provide for external quality assurance programs to assist in managing, collaborating with, and/or monitoring the MCO's quality improvement systems without any vested interest in the outcome.


    << Back | Table of Content | Next >>

    Back to Top