Treatment Improvement Protocols (TIP) 14 |
Outcomes monitoring is meaningful only if the information it produces is put to good use. A major responsibility of the single State agency (SSA) in implementing an outcomes monitoring system (OMS) is to analyze, interpret, and disseminate OMS data in ways that will satisfy the needs of potential users. In addition, the SSA must be able to use the results of outcomes monitoring for its own purposes, including planning and policymaking. This chapter discusses some of the issues related to this dual responsibility for the uses of outcomes monitoring data.
|
Who Uses Outcomes Monitoring Data, and How?
The potential users of outcomes monitoring data include most of those organizations and groups identified in Chapter 2 of this Treatment Improvement Protocol (TIP) as important to the planning and development process of the OMS. Treatment providers will also have been involved in the process of data collection. These data will be forwarded to the SSA for compilation and interpretation, and the findings will be formatted and disseminated to the providers as well as various other users.
In addition to the SSA and providers of treatment services, other users of OMS data may include the governor's office, the legislature, local or regional planning areas, and other health and human service agencies. Policy decisions made by these users on the basis of outcomes monitoring data will ultimately have an impact on the funding of programs and services to consumers. In this way, treatment outcomes data filter up and down within the State structure for use in different ways.
Treatment programs will use data for the following:
Local regional planning agencies and administrative units may use data for the following:
Internal data is used by the SSA:
A well-designed OMS will require adequate attention to data analysis. Data analysis should be performed by personnel who have the appropriate statistical training and expertise. The data analysis plan should be designed to answer the questions identified by OMS planners as most important for their audiences.
Determining treatment outcomes involves analysis of patient characteristics and services received by the patient along with outcome measures. Adjunct services outside the AOD treatment system may also have an impact on outcomes and should be included in the analysis. The multivariate analyses required will be quite complex. The improper application of statistical procedures or unwarranted interpretations of statistical results can result in inaccurate conclusions, potentially jeopardizing the integrity, acceptance, and usefulness of the OMS.
The improper application of statistical procedures or unwarranted interpretations of statistical results can result in inaccurate conclusions, potentially jeopardizing the integrity, acceptance, and usefulness of the OMS. |
The most often heard complaint about large data systems is that the data go in but never seem to come out. Unfortunately, this criticism is often justified. Frequently, the bulk of resources goes into data collection, and there are not enough funds and staff available for data analysis and reporting of results.
Since multiple stakeholders will have different interests, the data analyses should be conducted to meet those multiple needs whenever possible. It is also important that the results of the analyses be communicated to interested parties in a timely, meaningful, and relevant manner.
In order for an OMS to be useful and to obtain substantial buy-in and participation, OMS results must be made available on a timely and regular basis. Thus, it helps to have automated reports. A schedule of reports should be developed and disseminated to OMS participants and other interested persons. Guidelines for data delivery should be established as part of the OMS design parameters, just as data delivery guidelines from providers are incorporated into the system design.
A balance must be sought between the demand for quick results and the realities of analyzing data and preparing reports. Some audience needs can be accommodated by designing a series of reports that release findings incrementally. For example, program participants could get quarterly, semiannual, and/or annual summaries of patient characteristics at admission. Tabular summaries can be programmed so that automated reports are generated at predetermined intervals. Washington State and Minnesota have both automated reports. Data can be presented program-by-program or aggregated statewide or by county. Data can also be aggregated by program type (for example, inpatient or outpatient).
Programs can also obtain patient discharge summaries on a regular basis, including such data as program completion rates, payment source, and length of stay. Admission and discharge summaries can be extremely useful for planning purposes. Such patient profiles can help determine whether programs are serving their target populations, whether patterns exist in retention or dropout rates, and whether services delivered are appropriate to the needs described by patients. Regular data summaries are also useful for general audit purposes.
Treatment service data are likely to be more complicated to analyze and present and may require a more conservative report schedule. Followup data demand the greatest lag time of all. Suppose, for example, that designers of an OMS wanted to include data on 1996 admissions, and the design included a 6-month followup. Programs would start collecting data in January 1996. Depending on the length of treatment, patients admitted in December 1996 might not be discharged until May 1997. Their followup interviews would be conducted in November or December 1997. By then, almost 2 years would have elapsed since the beginning of data collection before followup data were even ready to be analyzed. Realistically, at least 6 more months might be required for analysis and reporting.
However, it may not be necessary to wait for followup to be completed on the entire year's admissions to begin to generate followup results. If the sample size is very large, preliminary results might be reported—for instance, admissions in the first quarter or first 6 months. Nonetheless, it is important that all personnel involved with analysis of OMS data understand that posttreatment followup, by its nature, extends the data collection and reporting process.
Comprehensive results of the OMS should be presented in a document that describes data collection methods, limitations of the data, analyses, findings, and recommendations. Tennessee and Colorado have created such documents. In addition, more selective reports, tailored to specific users, should be prepared. A brief summary of major findings may be appropriate for policymakers. Treatment providers will be interested in the benefits to them, and the data must be transmitted in a form that allows practical application by program staff. Media outlets can be sent press releases and charts or graphs of findings.
Comprehensive results of the OMS should be presented in a document that describes data collection methods, limitations of the data, analyses, findings, and recommendations. In addition, more selective reports, tailored to specific users, should be prepared. |
When the documentation of the OMS is developed, all participating groups should be invited to review and comment. Preliminary findings that may have important implications for these groups should be shared when appropriate. Some groups may want to develop their own action plans based on the findings. They may also want to develop a consensus statement in support of the findings. To encourage this kind of support, the groups should have the opportunity to comment before the final report and to find commonalities around which consensus can be developed. As a result, areas on which consensus is reached may move up in terms of their priority.
If findings do not win acceptance with one or more groups, the single State agency should conduct an analysis to determine why. Does the lack of acceptance reflect reservations about the validity of the findings, or is it the result of political differences? The explanation for any failure to endorse OMS findings has important ramifications for how the SSA will want to respond.
In preparing OMS findings for dissemination, and in all efforts to gain support for recommendations based on those findings, the emphasis must always be on improving outcomes for patients, their families, and society. This underlying principle should guide all policy decisions based on outcomes monitoring.
In preparing OMS findings for dissemination, and in all efforts to gain support for recommendations based on those findings, the emphasis must always be on improving outcomes for patients, their families, and society. |
A wide range of policy changes could be indicated by the findings of outcomes monitoring:
A review of the hypothetical findings described in Chapter 1 and the examples displayed here will be helpful in illustrating the kinds of treatment system changes that might be envisioned.
Most policy changes have important budgetary implications that must be kept in mind when setting priorities for policy change. Any efforts to bring about policy change should be undertaken only after careful assessment of the budgetary implications and the development of realistic proposals for addressing them. A policy mandate that does not carry with it the assurance of additional monies will necessitate reallocation of funds from other areas. Cost-sharing with other agencies may need to be developed through interagency agreements.
Implementation of desired policy changes identified through outcomes monitoring will require effective working relationships among policymakers at various levels within the State. If these policymakers were involved—or at least informed—in the planning stages of the OMS, then the groundwork would be laid for gaining their further support for new and revised policies that would improve treatment outcomes.
Hypothetical Example: Culturally Specific Programs Found
A statewide analysis compared outcomes for patients of color served in special population programs with those served in general population programs. American Indian patients who were admitted to general programs were compared with those admitted to programs specially designed in the traditions of Native American cultures and staffed by American Indian staff. In this hypothetical analysis, no significant differences at admission were found between the two groups of American Indian patients. Although abstinence rates 6 months following treatment were comparable, patients in the American Indian programs reported better physical, psychological, social, and occupational functioning. The improvements were sufficient to justify a greater effort to make culturally distinct programming available to other communities of color. State funds were used for scholarships to promote minority counselor training and program development. |
Cultivating good relationships with both the governor's office and the legislature has always been an important part of the job performed by most SSAs. Many SSA staff members have been responsible for shepherding proposals of various types through the legislative process and are skilled in negotiating that process to achieve policy change. They understand the need to identify and take into account factors operating within the political environment, factors that can have a decisive effect on legislative outcomes. This kind of understanding and experience will be needed if outcomes monitoring is to serve its intended purpose.
The support of legislators who are in a position to serve as advocates for policy regarding AOD treatment programs is imperative. This support should be cultivated throughout the process of developing and implementing the OMS. If possible, legislators or members of their staff should be represented on the steering committee that develops the OMS, although in some States regulations may prevent their formal participation. Even if legislative staff cannot formally participate on the committee, they should be invited to attend meetings as observers.
In developing legislative support for recommendations resulting from outcomes monitoring, it will be helpful to identify a legislative champion. Who in the legislature cares about substance abuse issues and will provide leadership for gathering the support of colleagues? Such a champion or advocate may not necessarily be a lawmaker on an oversight committee, but someone with a more personal interest, perhaps a person recovering from a substance use disorder or the parent of a child who uses chemicals.
It is likely that someone within the SSA has already established relationships with legislators who have this type of personal interest in supporting the agency and its mission.
If no legislative advocate is readily identifiable, then the SSA must take steps to cultivate and develop such an advocate. The aid of an intermediary with close ties to a key legislator might be enlisted to make the initial contact with that legislator.
Hypothetical Example: No Cost Benefits Found With New Vocational
|
Hypothetical Example: State Finds Its Services for Persons With
|
Such an intermediary might be a legislator's spouse with an interest in substance abuse issues, or someone in the State office of the National Council on Alcoholism or some other appropriate constituency group. In California, for example, the Women's Commission on Alcohol and Drug Dependence is a potent advocacy group with close links to the legislature and a good choice as an intermediary for the SSA.
Once relationships with legislators and key committee staff have been established, it is essential to keep them informed. Techniques to accomplish this include regular briefings, interim reports, and presentation of an overview of projected findings and recommendations in a preliminary report. An informal presentation by the SSA director and several key staff members might be an effective way to make this report. The report might be given 3 months ahead of the scheduled publication of a final document.
Such a presentation should be carefully planned, prepared, and polished. If the presentation includes information that the legislators or others in attendance may not want to hear, SSA staff should prepare them for that fact in advance. The presentation will be enhanced by effective audiovisuals that graphically depict the findings. It may also be enhanced by the participation of representatives of legislative constituencies, influential organizations, consumers, and families. There is some disagreement about the advisability of using consumer representatives to support agency positions and policies, and using them can carry some risk. But carefully selected consumers of services or family members, especially children, can provide powerful supporting testimony.
Followup with all contacts with the legislature, as with other potential supporters and collaborators, should be timely and consistent. Once the formal report is published, press releases and other marketing efforts can encourage the legislative action necessary to implement needed policy changes.
Every effort should be made to ensure having the governor's support for findings and recommendations emerging from outcomes monitoring. In many States, clearance from the governor's office will be required before the legislature is contacted or any reports are issued.
It is essential to the SSA's campaign for policy changes, that the SSA know the established lines of authority and the appropriate channels of communication. It is important that the governor be represented on the steering committee and that someone in the governor's office be kept informed of all developments as they occur.
Every effort should be made to ensure having the governor's support for findings and recommendations emerging from outcomes monitoring. |
There is probably already someone on the SSA staff who serves as liaison to the governor's office and who understands the process of effecting policy changes. This person may be the most appropriate choice to serve as liaison on behalf of outcomes monitoring, although the OMS project director may also assume this role. A key criterion for this assignment is the strength of the relationship with the governor's office.
In some instances, the policy changes proposed by the SSA may be in conflict with the policies or political philosophy of the governor. When this is the case, managing the process of policy change becomes more complex, and the SSA may have to rely on constituency groups to lead lobbying efforts in support of desired changes.
Providers of substance abuse treatment services will be responsible for translating the results of outcomes monitoring into improvements in patient services. The SSA should communicate and interpret these results to providers in a supportive and nonthreatening way. Particularly in cases where program changes are called for, this task may require the same tact and diplomacy required to win the support of State policymakers.
An OMS designed to address broad treatment service delivery questions will be less threatening to providers than one that compares individual program results. If OMS data indicate, for example, that outpatient counseling for certain patient groups does not work, then a different strategy must be tried. Programs offering outpatient counseling to these patient groups should know that the intent is not to eliminate them, but to find ways to make their efforts more effective. The emphasis is on improving patient outcomes, not casting blame or closing programs.
At the same time it is working to reassure providers about the impact of OMS findings, the SSA must also be prepared to be directive to providers about programming in response to those findings and to assist them in making requisite changes. For example, OMS data might indicate that group counseling sessions are more effective than didactic education programs in improving patients' social functioning. A program already getting $50,000 to serve women with alcohol problems but that uses didactic strategies will need to revise its strategies based on such findings. Technical assistance from the SSA may be required, or it may be possible to refer the program director to another program serving similar patients but experiencing greater success through the use of group counseling.
The single State agency's best approach to aiding and facilitating needed program revisions will vary from State to State, depending in part on the SSA's relationship with providers. Some agencies do not have statutory authority over providers. If a regional or county authority exists, it may be necessary to communicate findings through that authority. Existing communication channels may need to be strengthened in order to ensure appropriate feedback to programs. Regional meetings or other communication strategies in addition to dissemination of reports may be considered.
|
It is important to bear in mind that outcomes monitoring should properly be considered an iterative process. Changes in outcomes variables, data collected, or analyses performed may be indicated following each data collection cycle. The steering committee may have a role in reviewing results and determining whether changes are needed in the OMS. The ongoing, evolving nature of outcomes monitoring must be understood by all participants in the OMS and all users of the resulting information. Similarly, the SSA should continue the process of education and cultivation of political allies.
The cost of the OMS and the responsibility of the SSA will clearly go well beyond that associated with the data collection effort. The agency must ensure that data are appropriately analyzed and used and that findings are accepted. If done well, however, the benefits from outcomes monitoring should justify the effort and costs involved.