SECTION I. THE CRITERIA PROBLEM

A major part of a program planning process is the attempt to estimate the contribution that each alternative program, or mix of programs, makes toward meeting fundamental governmental objectives. For the purpose of this paper, the terms "goals," "aims, "purposes," "missions," or "functions" may be substituted for "objectives." The need for evaluation criteria arises because funds and physical resources are scarce; there are not enough available to satisfy all needs and proposals. (The term "measures of effectiveness" is sometimes used by analysts instead of "criteria.") Thus the problem of choice arises, and evaluation of proposals is needed to make the best use of available resources. To perform this evaluation, it is necessary to identify specific criteria that can be used to evaluate performance against the governmental objectives.1

For example, if a governmental objective such as "to reduce crime" was identified, then it would be appropriate to use crime rates as the major criterion (but not necessarily the only criterion) for evaluating activities aiming at these objectives. That is, in comparisons between various proposals, each proposal's effect upon the anticipated future crime rates would need to be estimated.

As the example indicates, the selection of criteria depends upon the objectives that are formulated. Also the process of selecting the criteria will often suggest the need for revision of the objectives. Thus, the establishing of objectives and criteria are interacting processes. In this paper, the emphasis is on criteria; objectives are discussed and presented only briefly. Ideally, a thorough discussion of State and local government objectives would be undertaken first. An important characteristic of both "objectives" and "criteria" as used in this paper is that they are intended to be "end" oriented rather than "means" oriented. That is, they are intended to reflect what is ultimately desired to be accomplished and for whom, not ways to accomplish such objectives.

For example, the phrase, "to disperse cultural facilities rather than concentrating them in a single locality" is a means "to provide adequate cultural opportunities to all." Use of the former phrase as the statement of objective rather than the latter would lead to somewhat different criteria, such as "the number of cultural facilities." Program analysis would better compare dispersal programs with centralized programs as alternative means to providing adequate cultural opportunities.

1 The term "output measure" is also occasionally used instead of "criteria." However, when "output measure" is used, it often is used to encompass not only program evaluation criteria (the subject of this paper) but also indicators of the size of programs such as the number of cases handled, the number of fire stations, policemen, teachers, hospital beds, etc., which though of considerable interest are not major evaluation criteria in the sense used in this paper.

Also, the concept of objectives as used in this paper avoids inclusion of specific numerical magnitudes. For example, a statement of objectives such as "to reduce crime rates 10 percent" should be avoided. For program analysis it is seldom appropriate to prespecify magnitudes. The specific amount of improvement that should be sought should generally not be determined until after the alternatives have been evaluated as to the costs and benefits of each and after these tradeoffs are understood.

The criteria for program analyses ideally should have the following general properties:

(1) Each criterion should be relevant and important to the specific problem for which it is to be used. (This will depend upon the fundamental objectives to be satisfied.)

(2) Together the criteria used for a specific problem should consider all major effects relative to the objectives. Enough criteria should be evaluated to cover all major effects. The use of insufficient criteria can be very misleading.

For example, programs to improve housing conditions should in general consider not only the number of acres of slums removed but also the effects upon the persons removed (perhaps by including a second criterion: the number of persons still living in substandard dwelling units).

Although it would make the evaluation considerably easier to have only one criterion, or at least very few criteria, the important thing is to avoid excluding major considerations from an analysis.

As indicated in the previous example, probably any single objective if emphasized too much without considering other needs, could lead to excesses and result in even worse conditions. Other examples are: sole consideration of safety in moving traffic could result in excessive trip delay times; in the law enforcement area, sole concentration on crime rates might lead to programs that result in excess control of individual movement.

With all the criteria expressed in terms of one unit (such as the dollar) or two units (such as the dollar and some nonmonetary unit), neat, analytically optimizable solutions would usually be possible. However, forcing the analysis into oversimplified forms may hide many major considerations. Use of multiple evaluation criteria seems, in general, to be unavoidable.

(3) Each of the criteria ideally should be capable of meaningful quantification. This involves two major problems. The first is the measurement of the current and historical magnitudes of each of the criteria. This measurement is needed to give a clear picture of the magnitudes of the problem, to determine how well the jurisdiction is actually doing toward meeting its objectives, and to provide a basis for making projections into the future.

For the housing example used above we would want to be able to measure how many acres of slums and how many people are living in substandard dwelling units there currently are, and how many were living in such units previously.

The second problem is the estimation of the future magnitudes for these criteria for each of the alternative programs being considered. Projecting into the future is always hazardous. One of the most, if not the most, difficult problems in program analysis is the estimation of the effects on the criteria of the various courses of action. Historical

data are important both for measuring progress and for making inferences as to what has caused any changes that have occurred. This latter information is very important for preparing estimates of the effects of future courses of action.

In practice, it is very difficult, and probably impossible, to meet perfectly all three of these ideal properties of criteria. The list in section III is a first attempt to identify the major criteria that are likely to be pertinent for governmental programs. An explicit attempt has been made to make the list conform with the first two properties (that is, relevancy and coverage) given above for ideal criteria. However, the list is certainly far from definitive in either depth or coverage. It is also somewhat idealistic; the analysts' ability to estimate meaningfully the effects of alternative program upon the criteria (the third property given above) will undoubtedly be limited in many instances particularly with current information systems.

On occasion, it may be necessary to utilize purely qualitative criteria such as, "In reducing crime, alternative A is more effective than alternative B but less effective than C." This ranking procedure might be partially quantified by having experts apply their judgments to some type of ranking scale. This would result in such a result as, "In reducing crime, alternative A has a value of 80 on the specially prepared ranking scale, B has a value of 65 and C a value of 85."

Thus in practice, even though criteria are not completely capable of being satisfactorily quantified, criteria that have the other two properties may still be useful.

The list of criteria in section III is hoped to be a reasonable starting point from which individual governments would develop a sound set of criteria appropriate to their own specific problems and governmental objectives. Many of these criteria are already in use. For an individual problem, the analysts will need to determine the specific criteria appropriate to that problem. The list in section III may help to suggest the appropriate ones. Each interested reader is encouraged to think through and work out what he feels to be an improved list.

With few exceptions, only nonmonetary criteria are listed in this paper. It is assumed that, in general, all problems will need to consider the actual monetary effects of each alternative course of action proposed. That is, one objective in all problems will be to keep monetary costs as low as possible for any level of program effectiveness aimed for. However, it is a premise of this paper that in the past too much emphasis has been placed upon attempting to translate all program effects into dollar terms. It is true that if this could be done meaningfully, the evaluation of alternative and final program selection would be eased considerably since the quantitative evaluations would all be expressed in the same unit-the dollar.

Realistically most governmental problems involve major objectives of a nondollar nature. Not only is it very difficult for analysts to assign dollar "values" to such nondollar objectives, but it is also questionable whether it would be desirable even if it could be done. Thus, questions of the value of such effects as reducing death rates, reducing illness incidences and severities, improving housing conditions, and increasing recreational opportunities should not become simply a problem of estimating the dollar values of these things.

80-976-67- -3

The analysts should rather concentrate upon the estimation and presentation, for each alternative, of full information as to the actual dollar effects and the effects upon the nonmonetary criteria. This is the primary function of program analysis-and of "cost effectiveness," "cost benefit," "cost utility," or "systems analysis," terms which for the purpose of this paper are all assumed to be equivalent. Attempts to force the criteria into commensurability are in most cases not worth much effort. It should be left to the decisionmakers to provide the value judgments needed to make the final program decisions.2

2 However, if the analysts can uncover some clues as to the worth that the jurisdiction's public does assign to such nonmonetary criteria, this information should also be provided to the decisionmakers (but not substituted for the basic information on the nonmonetary effects) to assist them in making their judgments. For example, various surveys of the public might give some information as to the degree to which persons currently might be willing to exchange money for changes in the nonmonetary criteria magnitudes. Highway tolls, for example, do indicate that the persons still using the highway are willing to pay at least the price of the toll for the advantages provided by the highway over alternate routes.

SECTION II. DISCUSSION AND QUALIFICATIONS

The criteria listed in section III are subject to a number of substantial qualifications and warnings; these are discussed below:

1. Criteria must relate to governmental objectives

As has been already indicated, the problem of selecting the appropriate criteria is dependent upon the problem of specifying objectives correctly. Thus, for a traffic-control problem, if the problem had originally been stated solely in terms of "reducing the number of traffic accidents," and if the analysts had limited themselves solely to this objective, the only criteria would have been the number of traffic accidents. Alternatives which, for example, restricted traffic flow such as by slowing down traffic considerably, would still tend to be the most "cost effective" since the rapidity of traffic movement was not implied in the statement of objectives and therefore was not included in the criteria.

For each major program area identified in the list, a brief statement is first given which summarizes the assumed objectives of the major program area. The criteria listed for the major program area should ideally provide a specific basis on which to evaluate the contribution that each alternative course of action makes to these objectives. If the reader prefers different statements of objectives, he is also likely to be led to somewhat different criteria.1

The specific objectives of a jurisdiction also depend upon the jurisdiction's own concept of the extent of the government's role in each program area. In many instances, there are likely to be considerable differences of opinion as to the proper role of the government. However, in general, such functions as law enforcement, fire protection, and water supply are usually assumed to be primarily governmental functions. Such other functions, however, as health, intellectual development, job opportunities, and leisure-time opportunities may rely heavily upon private sectors. Nevertheless, governments do have some role in most of these, usually at least having a part in helping the "needy" to reach certain minimum standards.

A related problem is that of the many and periodically changing ways in which government's role is divided among the various levels of government such as among city, county, State, and National, and, of increasing importance, special regional organizations.

The specific role played by the government in each individual jurisdiction must be considered in selection of the appropriate criteria.

In this paper no consideration is given to the question of "national objectives" such as national prestige and national security. It would

1 Persons with different perspectives, different cultures, would probably develop a different set of objectives--thereby implying somewhat different criteria. For example, the caveman would probably insist upon a major program area entitled "Food Supply," and one labeled "Mate Procurement." (The latter would be a tough one for State and local governments.)