Treatment Improvement Protocols (TIP) 14 |
Methods of data collection for an outcomes monitoring system (OMS) have been discussed in the previous chapter. Content of an OMS is discussed in this chapter. While these two aspects of OMS design are truly inseparable, there are some advantages in considering the "what" separately from the "how." Overlap will be evident, however, when content depends on the duration of time investigated, and when accuracy of content depends on the timing of data collection. These factors, while mentioned in this chapter, were discussed in detail in the preceding chapter.
Determining the content of an outcomes monitoring system depends upon a number of factors:
While there is no single, definitive set of variables to include, an OMS needs to describe both patients and the treatment they receive. To date, most outcomes studies have placed more emphasis on differences in patients than differences in treatment; many have excluded treatment variables altogether. In recent years, a major shift has occurred and several well-designed studies have been launched to examine the relationship of aspects of treatment to patient outcomes (Ball and Ross, 1991; McLellan et al., 1993; Moos et al., 1990).
The number of patient variables that can be included in outcomes studies is virtually unlimited. Because each additional variable translates into additional costs, a basis for making decisions must be established. Three questions can be asked with respect to every patient variable under consideration for inclusion in the OMS:
The first question is important because changes in patient status are the essential components of an OMS. The second question is important because variables that are known to predict outcomes are crucial to characterizing and distinguishing treatment populations. Predictor variables include, for example, the chronicity of substance use, social stability, and criminality. Predictor variables must be taken into account in interpreting the outcomes for any particular group of patients. In research terms, predictor variables are called independent variables, and outcome variables are called dependent variables.
In research terms, predictor variables are called independent variables, and outcome variables are called dependent variables. |
The third question relates to utility. Even when the answer to one of the first two questions is yes, the variable may be of limited utility. For example, suppose it had been established that people of one particular religious background had better outcomes than another. While religion could then be considered a predictor variable, it may have no practical implications for patient assessment, treatment placement, or treatment design. The same might be said of a finding that patients with life-threatening health conditions had better outcomes. This hypothetical finding would be interesting and not necessarily unexpected, but it would have little practical application. These examples also illustrate the need for caution when interpreting correlations. Correlates are not necessarily causally linked, although they may be. (More research would be necessary to make such a determination.) Even when the relationship may be cause and effect, utility can still be limited. In the two situations illustrated, for example, no one would suggest, based on the findings, that patients be encouraged to change religions or develop life-threatening conditions.
With respect to outcome measures, Chapters 1 and 2 discussed the critical importance of including outcome measures that have political and pragmatic utility, such as those that document that treatment produces cost savings in areas such as healthcare utilization and the criminal justice system.
The questions related to predictive value and utility can also be applied to proposed treatment variables. Is this variable to be included because it is known or believed to be related to patient outcomes? If it is related, what might be the implications of findings for patient placement, program design, or State policy?
Before discussing variable selection in more detail, it will be helpful to present and explain the way variables are categorized in this chapter. As shown in Exhibit 5-1, patient variables are categorized as administrative, predictor, and baseline/outcome. Treatment variables are classified as administrative and predictor. This classification scheme is primarily for ease of presentation; the distinctions are not necessarily so clearly cut. For instance, some variables classified here as administrative or baseline can also be considered predictor variables.
The two collections of patient variables currently in most widespread use are the Federal Client Minimum Data Set, also referred to simply as Client Data Set (CDS), and the Addiction Severity Index (ASI). They are explained in detail below.
The Alcohol and Drug Abuse Client Minimum Data Set was developed by the National Institute on Alcohol Abuse and Alcoholism (NIAAA) and the National Institute on Drug Abuse (NIDA) to standardize States' reporting of patient data for Federal reporting purposes. This data set is required for all treatment providers that receive any State alcohol and/or drug agency funding (including Federal block grants), and must be collected on all patients served by these programs, regardless of their individual payment sources. The CDS and related data reporting are now under the auspices of the Substance Abuse and Mental Health Services Administration Office of Applied Studies.
As shown in Exhibit 5-2, the Client Data Set includes patient and program identifiers, demographics, information about AOD use, and other patient and program information. Because its use is mandatory for many providers, its development was collaborative, and the variables and response choices have already been standardized. The CDS will be presented here for consideration as a legitimate starting point for OMS content. It makes no sense to develop a competing set of variables and then require treatment providers to use different sets for different purposes. However, suggestions will be offered about how to expand, modify, or supplement the CDS to meet local needs without violating the integrity and consistency of data reported to Federal agencies.
The Addiction Severity Index assesses seven areas of patient functioning: alcohol use, drug use, medical status, psychiatric status, employment and financial support status, legal status, and family and social relationships. Originally developed in 1980, it was revised most recently in 1992 (McLellan et al., 1980; McLellan et al., 1992b). Each of the dimensions covered by the ASI includes lifetime measures, which can serve as predictor variables, and past-30-days measures, which can serve as baseline/outcome measures. The ASI also includes clinical and patient-reported ratings of problem severity in each of these areas, as well as the patient's rating of the need for more treatment or assistance in each area (The fifth edition of the ASI has been reprinted in full in a previous Treatment Improvement Protocol (TIP) Screening and Assessment for Alcohol and Drug Abuse Among Adults in the Criminal Justice System.)
The validity and reliability of the ASI have been well established for a variety of diverse populations (Longabaugh, 1991). Because the ASI provides a standardized way to measure problem severity across many dimensions, treatment outcomes across programs and in varied geographical areas can be systematically compared.
According to NIDA, which has supported its development, the ASI has achieved widespread acceptance around the world among AOD treatment personnel and researchers. It is in use in more than 1,000 programs in the United States and is increasingly being mandated by State and local governments to assess government-funded programs (Mathias, 1994). The ASI has the additional advantage of having been translated into nine languages.
| Exhibit 5-1 Patient and Treatment Variables |
||
|---|---|---|
Patient Variables |
||
| Administrative | Predictors | Baseline/Outcome |
Patient identifier Referral sources Payment source |
Demographics Education Vocational history Social history Alcohol and other drug (AOD) use history AOD treatment history Medical history Psychiatric history Legal problems Motivation Treatment readiness |
AOD use frequency AOD use amount Mode of drug administration HIV risk behaviors AOD dependence symptoms Physical health Psychological health Employment Financial stability Legal problems Family/social relationships |
Treatment Variables |
||
| Administrative | Predictors | |
Program identifier Admission date Discharge date Length of stay Charges |
Setting/level of care Therapeutic modality Context Treatment components Staffing Discharge status Patient satisfaction Posttreatment services |
|
No single patient assessment instrument will ever achieve universal acceptance; the AOD field is too diverse for that. The ASI, like any instrument, has its limitations and its critics. Some of the limitations and ways to compensate for them are discussed in the sections that follow. While consensus panel members were reluctant to recommend any one instrument as the standard for State outcomes monitoring systems, it was proposed that the ASI or its constituent variables be given serious consideration because of the strong base of research supporting its validity, reliability, and utility. Furthermore, its content addresses most of the areas recommended for consideration.
| Exhibit 5-2 Federal Client Minimum Data Set (1990) |
||
|---|---|---|
Required Items |
Optional Items |
|
PROVIDER IDENTIFIER PATIENT IDENTIFIER DATE OF ADMISSION DATE OF BIRTH SEX Male RACE Alaska Native ETHNICITY Puerto Rican HIGHEST SCHOOL GRADE COMPLETED EMPLOYMENT STATUS Employed full time Homemaker Student Retired Disabled Inmate of institution PRINCIPAL REFERRAL SOURCE An individual (or self) Other formal adjudication process Probation/parole Other recognized legal entity DUI/DWI Diversionary program Prison PRIOR AOD TREATMENT ADMISSIONS 0 |
PRIMARY AOD PROBLEM Alcohol TERTIARY AOD PROBLEM AGE OF FIRST USE (OR ALCOHOL INTOXICATION) Smoking Inhalation Injection Other 1-3 times past month 1-2 times per week 3-6 times per week Daily Free-standing residential detoxification Hospital rehabilitation (acute care) Short-term (<30 days) nonacute residential Long-term (>30 days) nonacute residential Intensive outpatient (2+ hours per day 3+ days per week) Nonintensive outpatient Outpatient detoxification No PATIENT VERSUS CODEPENDENT OR COLLATERAL RECORD |
MARITAL STATUS Now married Separated Divorced Widowed Dependent living Independent living Public assistance Retirement/pension Disability Other None Medicare Medicaid Private insurance Blue Cross/Blue Shield Health maintenance organization Other Unknown Workers compensation Medicare Medicaid Other government payments Blue Cross/Blue Shield Other health insurance No charge Other Unknown PSYCHIATRIC PROBLEM No No No |
As explained above, patient variables can be historical one-time measures (predictor variables), or they can measure status prior to treatment at different points along the treatment continuum or following treatment (baseline/outcome measures). For purposes of an OMS, data collected at intake or shortly thereafter should include both historical and baseline measures.
While consensus panel members were reluctant to recommend any one instrument as the standard for State outcomes monitoring systems, it was proposed that the ASI or its constituent variables be given serious consideration because of the strong base of research supporting its validity, reliability, and utility. Furthermore, its content addresses most of the areas recommended for consideration. |
Patient predictor variables include characteristics generally referred to as demographics such as gender, age, and race/ethnicity, as well as social/vocational factors such as marital status, employment status, level of education, and income level. Predictor variables indicate special treatment needs. These variables also include aspects of a patient's history that may include such factors as a history of physical or sexual abuse, childhood conduct disorder or antisocial behavior, age of onset of substance use, number of previous treatments and admissions for detoxification, psychiatric hospitalizations, chronic medical conditions, and many other factors. These will be discussed in detail later in this chapter.
| Exhibit 5-2 Federal Client Minimum Data Set (1990) |
||
|---|---|---|
Required Items |
Optional Items |
|
PROVIDER IDENTIFIER PATIENT IDENTIFIER DATE OF ADMISSION DATE OF BIRTH SEX Male RACE Alaska Native ETHNICITY Puerto Rican HIGHEST SCHOOL GRADE COMPLETED EMPLOYMENT STATUS Employed full time Homemaker Student Retired Disabled Inmate of institution PRINCIPAL REFERRAL SOURCE An individual (or self) Other formal adjudication process Probation/parole Other recognized legal entity DUI/DWI Diversionary program Prison PRIOR AOD TREATMENT ADMISSIONS 0 |
PRIMARY AOD PROBLEM Alcohol TERTIARY AOD PROBLEM AGE OF FIRST USE (OR ALCOHOL INTOXICATION) Smoking Inhalation Injection Other 1-3 times past month 1-2 times per week 3-6 times per week Daily Free-standing residential detoxification Hospital rehabilitation (acute care) Short-term (<30 days) nonacute residential Long-term (>30 days) nonacute residential Intensive outpatient (2+ hours per day 3+ days per week) Nonintensive outpatient Outpatient detoxification No PATIENT VERSUS CODEPENDENT OR COLLATERAL RECORD |
MARITAL STATUS Now married Separated Divorced Widowed Dependent living Independent living Public assistance Retirement/pension Disability Other None Medicare Medicaid Private insurance Blue Cross/Blue Shield Health maintenance organization Other Unknown Workers compensation Medicare Medicaid Other government payments Blue Cross/Blue Shield Other health insurance No charge Other Unknown PSYCHIATRIC PROBLEM No No No |
The distinction between predictor variables and baseline/outcome measures is more an issue of form than substance. Some variables can fit both categories, depending on when they are measured and whether the question is repeated at treatment followup. For example, the number of lifetime detoxification admissions can be collected at intake simply as an indicator of chronic intoxication. But admissions to detoxification could also be used as a baseline/outcome measure if compared over a defined interval before and after treatment (for example, the number of detoxification admissions during the 6 months before treatment compared with the number during the 6 months after treatment). Similarly, a psychiatric diagnostic variable could be used as a predictor variable, while measures of recent psychological distress could be used as baseline/outcome measures.
A patient identifier is critical to linking various forms for the same patient and for linking followup questionnaires to treatment questionnaires. It can also be used to match the records of multiple admissions for the same patient.
There are basically two types of patient identifiers—those that can be traced back to an individual and those that cannot. Examples of identifiers that can be traced to an individual include name, social security number, driver's license number, and permanent numbers assigned to recipients of public assistance. The concern about using traceable identifiers in an OMS is that their use is a breach of patient confidentiality. Identifiers that cannot be traced to an individual include cryptograms comprised typically of some combination of the following: letters from the first, middle, and/or last name; date of birth; digits from the social security number; or other identifying elements that do not change over time. The drawback of nontraceable identifiers is that they cannot be used to match records with other information systems that do not use the same type of identifiers (such as Medicaid, child protection, arrest records, and the like). Whichever type of patient identifier is used, extreme caution must be taken to protect patient confidentiality. All patient identifiers can be "scrambled" upon data entry so they are decipherable only to a few designated individuals who know the code. Computer databases can also be protected by several layers of security. The legal issues related to protecting confidentiality are discussed in Chapter 6.
Whichever type of patient identifier is used, extreme caution must be taken to protect patient confidentiality. |
The Federal Client Data Set includes a required item for principal referral source. One alternative would be to expand the response choices (for instance, separating self and family and adding more specific choices under community and healthcare referrals) and then collapse them for Federal reporting purposes. Another alternative would be to have the option "check all that apply" in addition to a "principal" referral. The intended use of the data should guide the final decision.
An optional item in the Federal Client Data Set lists response categories for "Expected primary source of payment for this treatment episode." In most States, it would probably be more useful to identify the actual payment source or sources. Since this item may not be determined at intake, it may more practical to include this question on a discharge form.
Demographic information such as age, gender, and race/ethnicity has obvious descriptive importance in an outcomes monitoring system. To the extent that differences in outcomes might be observed among major demographic groups, this information must be taken into account in planning services.
These variables serve a number of useful functions. Aggregated across programs at a State level, they provide a descriptive profile of who is entering treatment. They can provide comprehensive profiles of patients being served by the treatment system or any portion of it. By comparing this information with population base rates and AOD incidence/prevalence rates, it is also possible to use these data to determine who is not being served by AOD treatment programs.
The patient demographic profile can also be used to track trends in treatment admissions over time. Changing patterns might suggest the need for more programs designed to serve special populations. For example, if an increasing number of women are entering treatment, this increase may indicate a need for more programs to serve this population. Similarly, if a decreasing number of adolescents are receiving AOD treatment, the capacity for this group may need to be decreased.
The patient demographic profile can also be used to track trends in treatment admissions over time. Changing patterns might suggest the need for more programs designed to serve special populations. |
Demographic information can also be cross-tabulated or correlated with other variables such as AOD use patterns and treatment discharge status to determine whether demographic factors are independent predictors of outcomes or interact with other variables to predict outcomes. Controlling for demographic factors in data analyses can also indicate when demographic predictor variables are merely artifacts of other predictor variables.
Frequently in social science research, a demographic factor is significantly associated with an outcome. For example, race may be significantly associated with likelihood of relapse. However, race may also be significantly associated with a primary drug problem. For instance, the rate of treatment admissions for cocaine addiction may be higher among blacks than whites in the geographic area under study. An analysis that controlled for drug of choice might find that once drug of choice was controlled for, there was no significant difference between the outcomes of blacks and whites; that is, whites and blacks with alcohol problems had comparable outcomes, and whites and blacks with cocaine problems had comparable outcomes.
Typically, the kinds of variables included under demographics are the easiest to turn into specific questions on data collection instruments. They are usually formatted as categorical variables, where one response is selected from among a list of responses, and thus are much easier to design than measures of problem severity. Nonetheless, some important considerations should not be overlooked. These will be explained in the following discussion when pertinent.
Gender and age. Gender is an essential variable and one of the few that needs no discussion. Age is also extremely important as a predictor of outcomes and relatively straightforward to collect. In some systems that include birth date and date of admission, age is computed rather than recorded.
Race/ethnicity. Many instruments include the race breakdown used in the 1990 U.S. Census and the Federal Client Data Set. Five racial categories are used (Alaska Native, American Indian, Asian or Pacific Islander, Black, and White) with an extra response choice for "Other"; a separate variable addresses Hispanic ethnicity. Hispanic people can be of any race. While these formats have advantages in terms of standardization, they can pose some problems. Biracial and multiracial patients are sometimes categorized into one race by the interviewer based on appearance; sometimes patients are asked to pick which racial category best describes them; and sometimes biracial and multiracial patients are categorized as "other."
An alternative to the forced choice option is a "Check all that apply" response format. Several choices can then be formatted into single categories for Federal reporting purposes, but the value of more accurate and detailed descriptions would not be lost.
Another potential problem with existing standard racial/ethnic categories is that they may not provide sufficient information for some localities. For example, agencies in States with relatively large American Indian populations may wish to include tribal affiliation as a subset of the Indian category. States with diverse Asian American populations may find it helpful to distinguish Southeast Asian and other Asian subgroups. For example, forms used in California and Washington State include more than 10 specific Asian designations.
On the other hand, some persons argue against collecting data on race because results can be misinterpreted or used for political ends. Findings attributed to race actually often reflect cultural or socioeconomic factors. It is important to remember that even widely used variables and response choices should be examined in terms of whether they will provide information in ways most appropriate for local purposes.
Primary language/immigration status. Language difference is another variable that may help State planners design programs that are appropriate to the populations they serve. The Washington State system includes a question asking whether use of English is functional or limited, as well as an item identifying the primary functional language from among a list of 46 languages, including American Sign Language. Recent immigration status may also be important for cultural considerations and special needs, particularly in those States with large numbers of immigrants.
Some persons argue against collecting data on race because results can be misinterpreted or used for political ends. Findings attributed to race actually often reflect cultural or socioeconomic factors. |
The Federal Client Data Set uses the highest school grade completed as a measure of education level (a general equivalency diploma is equivalent to 12 years). Another option is a categorical variable that includes as response choices the highest degree attained: none, high school diploma, vocational technical certificate, bachelor's degree, master's degree, or professional degree (e.g., M.D., J.D., or other doctoral degrees).
Marital/relationship status, parenthood, sexual orientation. Marital status is an optional item on the Federal Client Data Set. The item as it exists does not provide a category for persons living in a committed relationship but not legally married. OMS planners might want to consider different response choices for this existing item or perhaps add an additional item.
Information on parenthood status and number of dependents can also help with program design and may be predictive of outcomes. The Washington State forms include questions about the number of children living with the patient (their own and others'), as well as children not living with the patient.
Sexual orientation can be a controversial item, and there may be doubt about the truthfulness of patients' responses because of their fears of discrimination. Some proponents argue that it is important to determine whether gay, lesbian, bisexual, or transgender individuals are more likely than others to have AOD problems. It is important to remember, of course, that treatment admissions are not an accurate reflection of AOD abuse prevalence. If underreporting were to occur, it would distort the findings anyway. As with all other variables under consideration, the key question is utility. Is the question asked routinely as part of individual treatment planning? Would the system be changed in any way as a result of findings in this area? If the data are not going to be put to practical use, it is probably not justifiable to ask patients to reveal such sensitive information.
Living arrangement. One simple breakdown is provided as an optional item in the Federal Client Data Set: homeless or transient, dependent living (as with parents or in a supervised setting), or independent living. These latter two response choices could be broken down into subcategories. For instance, dependent living could include separate response choices for living with parents, living with other relatives, board and lodging, halfway house, and so forth. Independent living could include separate choices for house, apartment, or mobile home. The utility of more detailed information should guide the decision process.
Veteran status. Veteran status is also an optional item in the Federal Client Data Set. The item is easy to collect and noncontroversial but may have limited utility overall. Minnesota modified the item so the "Yes" response was changed to "Yes, no combat" and "Yes, served in combat zone" to see whether combat experience might be a predictor variable. Iowa has modified this question to address military status, with response choices of "none," "veteran," "in reserves," and "active duty."
Socioeconomic status, income, health insurance. Socioeconomic status (SES) is a useful concept but not necessarily an easy one to operationalize. OMS planners may want to review available SES classifications to determine whether any would be useful for their purposes. Several optional items from the Federal Client Data Set can be used as rough SES indicators, for example, primary source of income/support, health insurance, and expected source of payment for treatment. Other options include personal or family income and occupational level. The State of Washington asks for monthly household income and monthly personal income. Iowa asks for taxable individual monthly income as well as occupation level (professional/managerial, sales/clerical, crafts/operatives, nonfarm laborers, farm owners and laborers, and service/household).
Health insurance variables can be useful since health insurance plays a part in dictating the kind of healthcare to which a person has access, which may have a significant relationship to outcomes. While any one variable in this general area has its limitations, a selection of several of these variables will probably provide sufficient information for categorizing patients.
Spirituality/religious affiliation. Another factor that can be considered as part of social history is religious affiliation and spirituality. Recovering AOD abusers often emphasize the importance of spirituality in their recovery. Involvement in church activities is another predictor of positive outcomes. Many AOD treatment programs use the 12-step approach, and issues of religiosity and spirituality may be important and can be measured (Tonigan et al., 1991).
The ASI addresses several aspects of family and social relationships. As lifetime measures, it determines whether close personal relationships existed with parents, siblings, partner(s), children, and friends. It also asks about periods of serious problems with these people as well as with other family members, neighbors, and coworkers. (This section, like all others in the ASI, also includes many 30-day measures for baseline/outcome comparison.)
Other variables could certainly be used. The key question, in terms of both treatment planning and predictive value, would be the availability and extent of social supports.
History of physical/sexual abuse. Many adolescents and adults in treatment report histories of sexual abuse and family violence. It is not yet known to what extent these factors may be predictors of outcome. Documentation of the extent of these problems among AOD patients, however, can be used for treatment planning and design purposes.
Many instruments, including the ASI, require a yes or no response to questions about abuse, victimization, and/or perpetration. For example, the Washington questionnaire asks whether the patient has ever been a victim of domestic violence and whether the patient is currently a victim of domestic violence. While this format allows for quantification of the extent of these problems, it may be inadequate as a predictor of treatment outcome. Many factors contribute to the impact of physical and sexual abuse on its victims: identity of the perpetrator, age of the victim, frequency and severity of abuse, and whether the victim sought or received help at the time or since the abuse. Assessment of all these issues, while appropriate for clinical purposes, would be beyond the scope of an OMS.
Abuse victimization and perpetration can also be used as baseline/outcome measures and will be discussed in this context later in this chapter.
Employment status. A variable that can present problems is employment status. The Federal Client Data Set includes a forced-choice response format: employed full time, employed part time, unemployed (and looking for work), or not in the labor force. This last category can be expanded into another forced-choice format that includes homemaker, student, disabled, retired, and correctional inmate. A forced choice results in the loss of information for patients who have multiple roles and responsibilities, such as the homemaker who is employed and goes to school, or the student who is also employed. Multiple role status in itself may be a predictor of outcomes and thus valuable information to retain. Forced choices by their very nature reflect a value as to which is more important when multiple responses apply, a value which may not be shared by patients or other OMS stakeholders. In the case of the Federal Client Data Set, "employed full time" would supersede choices listed below, and other applicable information would be lost.
Loss of information can be avoided with a "check all that apply" format. For Federal reporting purposes, the responses can be recoded into the hierarchy of response choices set forth in the Client Data Set guidelines. While this TIP cannot address the nuances of all variable options in this detail, the employment status variable, like the race variable, points out the pitfalls of assuming that demographics are easy variables to define or agree on.
Employment history. Current employment status does not provide information about occupational stability over time or give any indication of downward mobility that might be associated with AOD use. The ASI includes variables on length of longest held full-time job, most recent or current occupational status, and usual employment pattern over the previous 3 years. The Iowa system inquires about the number of months worked during the previous 6 months. These are examples of variables that inquire beyond employment status at admission.
Current employment status does not provide information about occupational stability over time or give any indication of downward mobility that might be associated with AOD use. |
Information about the history of substance use is important for proper evaluation of treatment outcomes because the type of drug and the severity of the AOD problem may dictate the intensity and specificity of treatment as well as predict outcome. People with a substance use disorder are a heterogeneous population with a variety of different use patterns; however, the effects of all these differences on outcomes are unclear.
Because reduction in substance use and its associated consequences is one of the primary goals of treatment, many substance use variables are appropriate as baseline/outcome measures and will be discussed later in this chapter. However, some substance use variables can be used as predictors as well, and some are solely one-time measures.
Substances used, primary drugs of abuse. Some indication of the substances used by individual patients is essential to an OMS. There are various ways to collect this information, and several combine type of drug (or drug class) with use frequency. The Federal Client Data Set requires coding the drug (or drugs) identified as primary problem, secondary problem, and tertiary problem for each patient. Clinical judgment is used to make the determination. For primary and secondary drugs, codes are also required for age of onset of use (first intoxication for alcohol), frequency of use, and usual mode of administration. This data set has some limitations, which will be discussed below.
Age of onset. Age of onset of use has been shown to be a predictor of substance use severity in both epidemiological and clinical studies (Buydens-Branchey et al., 1989). In the Federal Client Data Set, this variable is required only for primary and secondary drugs of abuse. If the patient initiated use with a third drug, the required information does not provide actual age of onset of any AOD use. This limitation can be resolved by either asking for this directly or including age of onset for all drug or drug categories reported by the patient.
Frequency of use. In the Federal Client Data Set, this variable is also required only for primary and secondary drugs of abuse. If the patient uses a variety of drugs, no information is obtained on the remainder. The variable is further limited by focusing only on recent use. A solution to both these problems is to add frequency of use as a variable for all drugs reported. It may be preferable, however, to exercise this option as a baseline/outcome measure so that frequency of use over a specified time interval can be compared before and after treatment. Use frequency will be discussed in greater detail in the section on baseline/outcome measures.
Mode of drug administration. "Usual mode of administration" is included in the Federal Client Data Set. Because it is required only for primary and secondary drugs of abuse, it does not serve to identify all current (or previous) injection drug users. It only identifies those clients whose usual mode of administration is injection. This information may be deemed inadequate, considering the risks for HIV infection associated with even infrequent use, or previous use dating back many years. Several States, including California, Iowa, Minnesota, and Washington, have included a more general question to address any history of injection drug use. This question need not be drug specific.
Age of onset of use has been shown to be a predictor of substance use severity in both epidemiological and clinical studies. |
AOD diagnoses. Substance use disorder diagnoses recorded at intake or discharge may be useful for purposes of population profiles. The Diagnostic and Statistical Manual of Mental Disorders, Third Edition, Revised (DSM-III-R) (American Psychiatric Association, 1987) codes are optional variables in the Federal Client Data Set. However, diagnostic categories alone are of limited value as predictor variables and should be supplemented by other indicators of problem severity such as frequency of use and use consequences. The full series of AOD abuse/dependence diagnostic codes should be listed with instructions to "check all that apply" in order to capture the full range of applicable diagnoses for polydrug-using patients.
Treatment history is an important patient variable to measure. It can provide an efficient indicator of the severity of a patient's alcohol or drug problem. The number of lifetime treatment admissions and number of lifetime detoxification admissions, for example, can assist with treatment placement decisions and may be useful predictors of outcome. The number of prior AOD treatment admissions is included in the Federal Client Data Set. Iowa limits the period of inquiry to 10 years.
In many studies, a relationship has been found between AOD use and injuries, physical disabilities, and other medical problems (Cherpitel, 1992; Cregler, 1989; Eckardt et al., 1981; Heinemann, 1993; Moessner, 1979). Whether to collect information on medical history and current medical conditions and the amount of detail to include will depend on the purposes of the OMS and the availability of resources. Background information could include such data as number of lifetime hospitalizations (excluding childbirth) or a general rating scale of overall physical health. On the other hand, if more detail is sought, questions could be asked about the nature and duration of illnesses or symptom severity.
While it is true that many illnesses are associated with substance abuse, the purpose of collecting exhaustive medical data in an OMS must be justified. What might be appropriate for a medical workup is not necessarily appropriate for an outcomes monitoring system. This is another area that could generate an almost unlimited number of potential variables, and decisions must be guided by the utility of the information in analyzing treatment outcomes. If the purpose is to determine whether ill health is related to outcomes, general indicators will probably suffice. If the purpose is to determine to what extent medical problems remit with treatment, a baseline/outcome measure would be needed.
The purpose of collecting exhaustive medical data in an OMS must be justified. What might be appropriate for a medical workup is not necessarily appropriate for an outcomes monitoring system. Decisions must be guided by the utility of the information in analyzing treatment outcomes. |
The ASI includes questions about lifetime medical hospitalizations, chronic medical problems, and regular prescription drug use. The Washington system includes questions covering numbers of emergency room visits, outpatient/clinic visits, inpatient hospitalizations, and days of hospitalization in the previous year. Washington also has separate questions that address whether or not the patient is currently under medical care for infectious disease, traumatic injury, continuing illness, or dental problems.
Pregnancy status. Pregnancy status is an optional and controversial item in the Federal Client Data Set. In some States, legislation has been passed that mandates reporting certain illicit drug use by pregnant women to local authorities. This requirement may motivate women to deny pregnancy. Depending on whether this item was completed at intake or shortly thereafter, pregnancy may also be underreported. Some women may discover they are pregnant at a later point during treatment (or not know at all).
Psychiatric disorders are found in AOD-abusing populations at rates higher than in general populations (Helzer and Pryzbeck, 1988; Hovens et al., 1994; Regier et al., 1990; Rounsaville, 1990; Stowell and Estroff, 1992). Psychiatric disorders may predate AOD abuse, be a consequence of AOD abuse, or coexist with AOD abuse. Psychiatric disorders include depression and bipolar disorder (manic-depression), anxiety disorders, schizophrenia and other thought disorders, posttraumatic stress disorder, dissociative disorders, and eating disorders and other compulsive behavior disorders. They also include personality disorders such as antisocial personality disorder. (For more information about dual diagnosis of psychiatric disorders and AOD abuse, refer to another TIP in this series, Assessment and Treatment Planning for Patients With Coexisting Mental Illness and Alcohol and Other Drug Abuse.)
As with physical disorders, it is important to be selective about the amount of information collected about psychiatric disorders and emotional distress. Collecting and recording data for an OMS is not the same as clinical assessment or documentation. Again, the guiding principle in OMS planning is to select those variables that may be significantly associated with treatment outcome or that serve as baseline/outcome measures. For example, the amount of subjective distress experienced by a patient may have a greater relationship to outcome than to the specific nature of the psychiatric diagnosis, or a combination of the diagnostic code and clinical severity rating may provide the most useful information. Ultimately, for purposes of data analysis, large numbers of variables must be distilled into a smaller number of useful ones. A review of existing studies and outcome monitoring systems may guide the distillation process. Incorporating a variety of psychiatric assessment instruments into the OMS may seem like a safe way to be sure all the bases are covered, but it is also likely to pose an insurmountable obstacle in terms of staff and patient time and data analysis.
The amount of subjective distress experienced by a patient may have a greater relationship to outcome than to the specific nature of the psychiatric diagnosis, or a combination of the diagnostic code and clinical severity rating may provide the most useful information. |
The Federal Client Data Set includes an optional variable that merely records the existence of a psychiatric problem. This question is probably too general to be of any predictive value and too vague to provide useful interpretation of findings. Washington includes an item to record whether or not a psychological evaluation was conducted and a psychiatric diagnosis was made, as well as yes/no questions on the current use of mental health services and psychotropic medication.
The ASI includes questions on the number of lifetime psychiatric hospitalizations and courses of outpatient treatment. The ASI also asks about lifetime episodes of depression, anxiety, hallucinations, memory or concentration difficulties, difficulty controlling violent behavior, suicidal thoughts, and suicide attempts, as well as the use of psychotropic medication. The Minnesota OMS added a question to the ASI series to address other compulsive behaviors such as eating disorders and gambling. Similarly, Iowa asks a general question about addictions other than AOD dependence.
Patients enter treatment for many reasons. Factors that influence the decision may be internal or external, and for many patients a variety of factors may impinge on the decision. For some, the admission is involuntary; they are court ordered or committed to treatment. For others, coercion may be more subtle: the threat of a spouse's or partner's leaving or the loss of a job, a professional license, or custody of children are often significant factors. Patients may also enter treatment because they are sick, broke, or weary of maintaining their addiction.
An OMS may include a measure of the coercive influences and motivations to enter treatment in order to determine their relationship to treatment outcomes. Findings in this area could have important policy implications and could be used to improve patient outcomes. A rating of patient readiness for treatment could also be included. If patient readiness were found to be a strong predict or of positive outcomes, it might indicate a change in patient placement strategies or the need to search for ways to improve patient readiness.
In Minnesota, the OMS includes a question that addresses the primary condition surrounding admission to treatment. Response choices include treatment versus incarceration or as a condition of probation or parole, to avoid loss of children or to regain their custody, to prevent loss of a relationship or living situation, to regain a driver's license, to keep a professional license or job or to stay in school, to retain eligibility for government benefits, or to escape financial pressures related to continued alcohol and drug abuse.
The key to valid and meaningful baseline/outcome measures is a comparable interval of measurement before and after treatment; Chapter 4 addresses this issue in detail. The advantages of short windows, such as 30 days, are more accurate recall and finer precision of measurement. A drawback is the lack of information for a longer duration. In fact, this lack is one of the criticisms of the ASI, which measures AOD use frequency only for the previous 30 days. These questions, asked 6 months after discharge from treatment, would not reveal whether the patient had used any alcohol or other drugs since treatment. In contrast, a 6-month interval covers a longer duration but lacks precision in terms of measurement. The recommended solution is a combination of both short-term and long-term intervals. This combination will be explained more in the discussion of specific variables that follows.
The most obvious baseline and outcome measures are those associated with substance use and abuse before and after treatment. Measures of frequency of AOD use are almost always included in treatment outcome studies. Frequency of use is important to measure because it is associated with a range of biopsycho-social problems. Reduction of use frequency is a useful outcome measure even in the absence of total abstinence. A number of instruments incorporate effective ways to ask about frequency of AOD use. The ASI includes use frequency for a variety of drugs and drug categories but limits the inquiry to the previous 30 days.
Measures of frequency of AOD use are almost always included in treatment outcome studies. Frequency of use is important to measure because it is associated with a range of biopsychosocial problems. |
Minnesota's OMS design addressed the limitations of both the Federal Client Data Set and the ASI in measuring AOD use frequency. For all the Client Data Set drug categories, use frequency responses are:
The Minnesota OMS replaces the "no use in past month" response choice with three other choices:
This design captures the information as required for the CDS but also allows for a record of 6-month and lifetime use. (The lifetime response choice is used only at admission, not at followup.)
The amount of drug use per typical occasion of use can be an indicator of the severity of the AOD problem and the individual's ability to maintain control over use. Amount of use is not used as often as frequency of use because it is more difficult to quantify and typically less valid and reliable. For example, while quantity of alcohol consumed during a typical drinking occasion may be a useful measure for social drinkers, its value diminishes for persons who abuse or are dependent on alcohol. Alcoholics may drink from the bottle, so a measure of number of "drinks" becomes meaningless. Even when persons drink cans of beer, glasses of wine, or mixed drinks, their accuracy in reporting quantity is likely to decline with increased consumption (Webb et al., 1991).
Amount of use of illicit drugs is even more difficult to quantify. There is no standard measure of drug use comparable to that for alcohol. Potency is not controlled by governmental regulation. Amount of consumption (number of joints, hits, lines, or fixes) may provide some indication of the compulsiveness of the drug-taking behavior, but it does not indicate how much of the drug was actually ingested. Drug quantity is not recommended as an OMS measure.
The amount of drug use per typical occasion of use can be an indicator of the severity of the AOD problem and the individual's ability to maintain control over use. |
Mode of drug administration is an important variable for a number of reasons. Injection drug use is an indication of problem severity and also important in terms of the risk of transmission of HIV and other infectious diseases. The Client Data Set includes "usual mode of administration" at admission and the same questions could be asked at followup. Another option is to specifically ask for the frequency of injection (the mode of administration of greatest concern) for an identical time interval before and after treatment.
Injection drug use has already been addressed. In Minnesota, another question addresses the frequency of using "clean works" among patients who report injection drug use. High-risk sexual behaviors have also been associated with AOD use, either because of poor judgment under the influence or the use of sex as a commodity to obtain drugs. Questions can address the use of condoms during sexual intercourse and the number of sexual partners over a specified interval before and after treatment.
These are obviously highly sensitive topics. In Minnesota, where questions about using clean works are included in the OMS, these questions are the ones most likely to draw objections from former patients during followup telephone interviews. Nonetheless, they may prove useful measures of behavior change.
Measures of symptoms of dependence are useful in determining the severity of a relapse following treatment. Dependence symptoms include:
According to the fourth edition of DSM (American Psychiatric Association, 1994), a remission has been achieved when abuse and dependence symptoms have been absent at least 1 month. Early remission covers the first 12 months without symptoms, and sustained remission covers the period beyond 12 months. Remission can be further categorized as partial or full. In full remission, no abuse or dependence symptoms are present; in partial remission, full criteria for dependence are not met, but at least one criterion is present either intermittently or continuously. Including DSM-IV criteria symptoms in the OMS will allow for classification of patients' posttreatment status in terms of partial or full remission (or partial or full recovery).
The Federal Client Data Set includes diagnostic codes from the third revised edition of DSM as optional items. (In DSM-IV, the latest edition of DSM, the diagnostic codes for abuse of and dependence on particular substances are the same as in the third revised edition.) However, diagnosis is likely to be established in a clinical setting as a routine part of the assessment process. Including all the relevant questions to establish diagnosis at followup may be too time consuming, relative to the benefits of this particular element. The Minnesota OMS limits inquiry in this area to a question on the number of times the patient experienced withdrawal symptoms in a 6-month period before and after treatment.
AOD use can have direct and indirect effects on physical health. Direct effects include, for example, infections from injection drug use and damage to the liver and other organs from alcohol. Indirect effects result from poor nutrition, unsanitary living conditions, lack of sleep, and other health risks associated with AOD use and AOD-related lifestyles, such as traumatic injury.
It is sufficient for purposes of an OMS to measure health status before and after treatment, without necessarily linking health status to AOD use. For example, the ASI asks, "How many days have you experienced medical problems in the past 30 days?" as well as "How troubled or bothered have you been by these medical problems in the past 30 days?" and "How important to you now is treatment for these medical problems?" Reduction in problem days or subjective severity rating can be used to show improved status after treatment.
It is sufficient for purposes of an OMS to measure health status before and after treatment, without necessarily linking health status to AOD use. |
Other options are questions that record the number of days in the hospital or visits for emergency care during, for example, a 6-month period before and after treatment.
General medical care may actually increase for a time during or after treatment as patients attend to long-neglected health issues. For this reason, measures of outpatient physician or clinic visits do not always show a posttreatment reduction. Inpatient and emergency care, in contrast, typically show dramatic decreases (California Department of Alcohol and Drug Programs, 1994; Harrison and Hoffmann, 1989).
As with physical health, AOD use can have direct and indirect consequences for psychiatric status and emotional well-being. AOD use can exacerbate existing mental health problems or bring on depression and difficulties with memory or concentration. Living conditions and disrupted interpersonal relationships associated with severe addiction can also cause emotional distress. Finally, difficulty controlling violent behavior, suicidal thoughts, and suicide attempts can also be associated with AOD use.
The ASI asks about a variety of psychological symptoms over the previous 30 days, information which provides useful baseline/outcome measures. The psychological questions parallel the medical questions regarding number of days (in the past 30) on which emotional problems were experienced, a subjective rating of problem severity, and a rating of the need for treatment.
Psychiatric hospitalization over a 6-month period or longer could be used as a baseline outcome measure; however, base rates are relatively low, limiting the utility of this item. Outpatient visits may actually increase as patients follow through on treatment plans that include attention to mental health as well as AOD use.
Current employment status, a Federal Client Data Set item, can be used at admission and at the posttreatment contact. Months of employment in the preceding 6 months is another option.
The ASI uses a 30-day measurement window and several variables: days paid for working, net employment income, days experiencing employment problems, subjective rating of employment problem severity, and rating of the need for counseling for problems related to employment.
Relying solely on employment variables to assess financial stability is inadequate because a large number of patients may not be in the work force (such as students, homemakers, retired, and disabled persons). Measures of financial stability can be used to indicate change in status before and after treatment. The ASI includes 30-day income measures for unemployment compensation, public assistance, and pensions or social security.
The Minnesota modification of the ASI incorporates questions about financial stability comparable to the employment questions: days experiencing financial problems, subjective rating of financial problem severity, and rating of the need for assistance with financial problems.
Legal problems can be quantified by using measures such as number of arrests over a specified interval before and after treatment. Researchers with experience in this area often prefer arrests to actual crimes because arrests are a matter of public record, making it more likely that patients will give an honest accounting. A recent California study was very successful at quantifying actual crimes (California Department of Alcohol and Drug Programs, 1994), but the typical OMS is not likely to have comparable resources available.
Legal problems can be quantified by using measures such as number of arrests over a specified interval before and after treatment. |
The ASI uses two 30-day measures: number of days detained or incarcerated and number of days engaged in illegal activities for profit. It also includes the patient's subjective rating of severity of legal problems and need for counseling or referral for those problems.
Marital or relationship status can be used at baseline and at outcome, but this information can be used to indicate whether the status is an improvement. Living arrangement may also be of some limited utility. It may be more helpful to measure satisfaction with marital/relationship status and living situation.
The ASI includes 30-day questions about interpersonal conflicts that provide outcome measures, but the absence of conflict may mean merely lack of social contact rather than improved relationships. The ASI also includes 30-day measures of sexual, physical, and emotional abuse—whether or not the patient has been abused or has abused anyone else.
The Minnesota modification of the ASI adds 30-day measures regarding family stressors other than conflict. It also includes a rating of how troubled the patient has been by loneliness or being alone, as well as questions about living with people who use alcohol or other drugs or who have AOD problems.
The limitation of these instruments is that neither the ASI nor Minnesota's modification addresses changes in positive aspects of family or social relationships, especially the establishment of new relationships by patients who had previously been isolated or estranged from others.
Much remains unknown about how outcomes relate to the type and quantity of treatment received, and this relationship is an issue of concern to the single State agencies. In outcomes research, treatment variables have typically been neglected (Sobell et al., 1987), but they have important implications for understanding outcomes. Certainly they have important cost ramifications, as in the areas of setting and staffing. If different treatment modalities or programs can deliver comparable quality of care as measured by comparable outcomes, then cost alone might be the basis for choosing one over another.
If only baselines and outcomes are considered, then treatment itself is the "black box," with what goes on within treatment remaining a mystery to outcomes evaluators and policymakers. For an OMS to function effectively, there must be a level of specificity about what goes on between patient admission and discharge. In the past, for example, researchers have been preoccupied with comparing the results of outpatient versus inpatient programs (Cummings, 1991; Miller and Hester, 1986), without looking at the actual content of these programs. Issues that must be considered include differences in forms of treatment, such as individual, group, and family therapy; differences in settings and populations; differences in lengths of stay and frequency and intensity of treatment sessions; and differences in the actual components of treatment.
If only baselines and outcomes are considered, then treatment itself is the "black box," with what goes on within treatment remaining a mystery to outcomes evaluators and policymakers. |
An identifier is necessary to distinguish one treatment provider from another. This is a noncontroversial item.
For inpatient and residential treatment programs, length of stay can be computed if the admission date is recorded on an intake form and the discharge date on a discharge form. For outpatient programs, however, this information only indicates length of time from start to finish of treatment and gives no indication of number of contacts during that interval. Hours of involvement or contact days could be recorded on a discharge form. Service amounts will be discussed in more detail later.
Level of service is a required item in the Federal Client Data Set. This breakdown or an expansion of it should be included in an OMS. The CDS also requires an indication of whether an admission is the initial admission in a treatment episode or a continuation of treatment resulting from a transfer (such as inpatient to outpatient care). Level of care can also be considered a predictor variable.
Information on costs associated with treatment is essential to an OMS focused on accountability for taxpayer dollars. Information could be collected as a package cost for the whole program or as itemized costs for specific services. Costs are not readily available to the program staff who complete patient forms, however. As some States move to managed care based on capitated rates, costs for individual treatment admissions may be difficult or impossible to compute.
As some States move to managed care based on capitated rates, costs for individual treatment admissions may be difficult or impossible to compute. |
Little work has been done in determining the effect of therapeutic focus on patient outcomes for different kinds of patients. For instance, while therapeutic communities differ from 12-step programs, there is still not enough information available to predict which patients would do best in which modality.
Social climate may also affect patient outcomes. For example, a well-known program may be extremely similar in structure and focus to many other programs, but its high profile may lead patients to have greater expectations of success.
While a measure of therapeutic focus might be important, it might be more revealing to include a variety of measures of the specific services that comprise the treatment regimen. Most programs actually consist of a composite of some of the following components:
Some treatment components may be specific to cultural heritage and traditional healing practices.
Decisions must be made not only about which treatment components should be measured, but also how to measure them. While instruments that attempt to define and quantify treatment components are relative newcomers in the field compared with those that assess patient characteristics, they should be examined for their usefulness in this area. For instance, a simple checklist could be used to indicate whether or not a patient had received a service, while a more sophisticated measure could be developed to describe the frequency, intensity, and duration of the service. It would be important to know, for example, whether or not a certain minimum amount of a service were necessary to achieve any beneficial effect, or conversely, whether or not a threshold were reached beyond which there were little or no measurable benefit. While no OMS could measure all the potential treatment variables that could be generated, attention to selected key variables may produce useful information for allocating limited resources, placing patients in the most appropriate settings, and giving them the services most likely to produce favorable outcomes.
It would be important to know whether a certain minimum amount of a treatment service were necessary to achieve any beneficial effect, or conversely, whether a threshold were reached beyond which there were little or no measurable benefit to the patient. |
The Treatment Services Review (McLellan et al., 1992a) provides a standard measure of the nature and number of services provided during the treatment process. Services are categorized along the same dimensions covered by the ASI and are recorded weekly. For the Minnesota OMS, the Treatment Services Record—a similar instrument—was developed (Harrison, in press). This instrument also uses the ASI dimensions to categorize services and quantifies hours in treatment and in specific components of treatment (educational sessions; individual, group, and family counseling sessions; support group meetings; and informal contact with other patients). Both the Treatment Services Review and the Treatment Services Record measure contacts that are part of formal treatment as well as ancillary services, since both may affect outcomes. The Iowa OMS also quantifies time involved in specific services, recorded on a monthly basis.
Staffing patterns can be described by considering several objective and subjective qualities of staff members. These include the following:
Few data exist on the relation of staffing patterns to patient outcomes. Selectivity in measuring the staff qualities listed above would be imperative if staff qualities were to be included as measures in an OMS. Most patients are likely to have contact with a variety of staff, so they may be exposed to a variety of levels of expertise, AOD use histories, and therapist characteristics. In addition, staff characteristics may not be static and may interact in different ways with individual patient characteristics. Although staffing characteristics and staffing patterns may be related to treatment outcomes, a decision might be made not to include these measures in an OMS because of lack of data about the benefits of collecting this information in relation to the time and costs involved.
Programs vary tremendously in terms of the populations they serve. Special populations can be defined by personal characteristics such as gender, age, race/ethnicity, cultural background, or sexual orientation. Because of Federal block grant mandates, many States are currently designing more programs for pregnant women and women with children. Special populations can also be defined by drug use patterns; some programs are designed specifically for people addicted to heroin or cocaine. Programs can also be designed for individuals with coexisting disorders such as mental illness, eating disorders, or compulsive gambling. If an OMS were to find that certain patients did better in programs designed to serve their special needs, compared with other settings, these results could be used to justify increased funding for such targeted services, despite the increased costs these might entail.
The patient's treatment completion status is essential information to collect in an OMS. Successful treatment completion can be used as an intermediary outcome measure (Wickizer et al., 1994), whereas leaving against staff advice may predict a poorer posttreatment outcome. Other reasons for failure to complete treatment may provide useful information to improve treatment retention.
The patient's treatment completion status is essential information to collect in an OMS. Successful treatment completion can be used as an intermediary outcome measure, whereas leaving against staff advice may predict a poorer posttreatment outcome. |
Patient satisfaction with treatment as a whole or with specific services received can be measured during and after treatment. Satisfaction ratings by service type are included in the Minnesota weekly Treatment Services Record as well as in the followup interview. The Iowa system also includes patient ratings of the benefits of certain treatment components collected at followup. While patient ratings of treatment may not be related to other treatment outcomes, they may be important to consider in an era that is becoming more attuned to customer satisfaction. Satisfaction may also be related to treatment retention, so a correlation between patient dissatisfaction and early dropout from treatment may provide useful information about program changes necessary to keep patients in treatment.
The use of community resources following discharge from treatment may also have an impact on patient outcomes. Patients are typically referred to peer support groups and other available resources to assist with their recovery following discharge from formal treatment. Learning more about the relation of these services and successful treatment outcomes can assist with discharge planning and resource allocation.
The content included in existing State outcomes monitoring systems provides a basis for review by States planning a new system or revising their current systems. Several State systems, in varying stages of development, are described in Appendix B. A contact resource is listed for each, to aid readers in obtaining a set of current instruments and other relevant information. Because of their length and because revisions occur periodically, individual forms are not reproduced in this TIP.
All the discussion of OMS content in this chapter is based on experience to date. In this rapidly evolving field, recommendations will change as new information emerges from existing systems and ongoing studies. Some of the data elements currently in widespread use may be found to be of limited utility in evaluating outcomes. Others, in trial stages or not yet developed, may prove to offer more benefits. While this TIP can provide a useful overview of outcomes monitoring systems and general guidelines for content, decisions for an individual State will depend on local needs and an up-to-date review of continuing improvements in this field.