Originally published Vol.
2, Issue 4 (September 2004)
Measuring the Success of Conservation Programs
Though farmers may be
induced by conservation program payments to change
their farming practices, it is difficult to link
their actions to outcomes, because they take place
within a larger set of complex interactions.
Katherine
Smith, Marca
Weinberg
|
This
article is drawn from . . . |
Economics
of Water Quality Protection from Nonpoint
Sources: Theory and Practice, by Marc
O. Ribaudo, Richard D. Horan, and Mark E.
Smith, AER-782, USDA/Economic Research Service,
December 1999.
“Beyond
Environmental Compliance: Stewardship as Good
Business,” by Jeffrey Hopkins and
Robert Johansson, Amber Waves, USDA/Economic
Research Service, April 2004.
“Have
Conservation Compliance Incentives Reduced
Soil Erosion?” by Roger Claassen,
Amber Waves, USDA/Economic Research
Service, June 2004.
|
Defining and measuring success
is easy—if you are Rube Goldberg. A widely
acclaimed 20th century cartoonist, Goldberg depicted
outlandish inventions that accomplished simple tasks
through an intricate series of linked steps, each
one triggering another until a desired outcome was
reached. Success, in Goldberg’s world, was
clearly defined and could be attributed directly
to the completion of several sequential, though
highly improbable, cause-and-effect actions. Success,
in the real world, even when it is clearly defined,
is not so easily measured. Gauging the success of
government programs, in particular, can be downright
complicated, even when the principles used in designing
them are rather simple.
Most conservation programs, for
example, are designed to improve the environment
by offering incentive payments to farmers, who are
thereby induced to change their farming practices.
Those changes in farmers’ practices—be
they reducing pesticide use, adopting conservation
tillage, or constructing a riparian buffer—should
then lead to enhanced environmental quality. But,
unlike the chain of events in a Goldberg invention,
the actions involved in a conservation program take
place not in isolation, but, rather, within a larger
set of complex interactions, making it difficult
to link programs to actions
to outcomes.
The first step in measuring the
success of agricultural conservation programs—and
other programs designed to address agri-environmental
issues—is linking a change in farmers’
stewardship behavior to the program being evaluated.
Because many other factors (including other government
programs) influence farmers’ choices, it is
critical to determine the extent to which it was
a given conservation program incentive that stimulated
some farmers to do something that they would not
otherwise have done. A second step requires assessment
of how the portion of observed stewardship behavior
that can be linked back to conservation program
incentives then affects environmental quality—given
that other factors also affect the environment.
Gauging Farm Operators’
Responses to Program Incentives
Farm operators are the target
of conservation program incentives, even though
the program itself aims to target one or more environmental
enhancements. Thus, to evaluate the program, one
must determine exactly how program incentives induced
operators of farms of various types, sizes, or features
to “sign up” as program participants.
Then, for those who become program participants,
it is important to find out how the type and extent
of conservation practices they adopted relate to
the levels of incentives provided through the program.
Only by separating the influence of program incentives
from other factors that affect farmers’ conservation
choices can the program evaluator be confident that
it was the program being evaluated that had an effect,
not other circumstances.
A farmer may adopt conservation
practices for a myriad of reasons. He or she may
be an ardent environmental steward who would implement
a particular practice (like maintaining grassed
buffers between cropland and water sources) regardless
of program incentives. Alternatively, a farmer may
adopt an environmentally friendly practice wholly
or partly in order to increase profits. ERS research
on conservation tillage, for example, demonstrates
that good stewardship can also be good business.
Policy incentives aren’t usually required
to induce a farmer to adopt what he or she views
as good business practice; market forces should
do the trick in this regard.
In evaluating the effectiveness
of incentives to induce farmers to participate in
conservation programs, it is important to note that
conservation programs are not implemented in a policy
vacuum. Both the costs and benefits of participating
in a given program will vary as a direct result
of the confluence with other government programs.
For example, commodity programs influence some crop
prices, making it more or less economically advantageous
to manage the crops in ways that enhance environmental
quality. Input use is sometimes controlled through
quantity restrictions and use regulations. Input
prices may also be influenced by policies—including
labor laws, pesticide regulation, and subsidization
of irrigation water—that influence relative
input prices and, thus, the financial costs or benefits
of conservation practices that shift input use patterns.
Finally, technological change, economy-wide variables
(such as interest rates and unemployment rates),
and farm household constraints (such as the role
of off-farm work in farm household income) are also
likely to influence farmers’ decisions about
farming practices—whether or not a conservation
program incentive is added to the mix.
Because farmers may adopt conservation
practices for reasons unrelated to the conservation
program, simply identifying changes in farmers’
practices (let alone environmental quality) is an
insufficient basis for judging the success of a
conservation program. One has to be able to
determine what proportion of farmers’ practices
can be attributed to a particular program before
the success of the program can be assessed.
Isolating the effects of program
incentives from the effects of other factors potentially
influencing farmers’ observed conservation
practices demands a lot of data of particular sorts.
A necessary requirement is the collection of data
that enable statistically reliable comparisions
of farming practices by farmers before and after
program implementation, or by farmers who did and
did not participate in the program in a given year
or years. Statistical analysis of such data can
support or refute a correlation between farm practices
and conservation program provisions.
However, supporting or refuting
simple correlation is not sufficient because that
correlation may be spurious and because it does
not prove causality. A “before-and-after”
comparison, for example, might miss the strong influence
of a new program on participants’ behavior
if other factors, such as unusual weather conditions,
prevented a large number of the participants from
following through on their program-induced good
intentions. Similarly, a “with and without”
comparison could falsely attribute observed conservation
practices to the conservation program if all farmer
participants in the program were pre-inclined toward
voluntary environmental stewardship even without
the program, and nonparticipants were disinclined.
More information is needed than simply who participated
and what practices they employed if a strong case
is to be made that the program was the stimulus
for farmers’ adoption of observed practices.
Additional data are necessary
to separate the effect of a conservation program
incentive from the effects of concurrent changes
in market prices, weather, other policies, and technology.
Identifying the farmers for whom program incentives
induced adoption of conservation practices requires
data on the characteristics—types and locations—of
both participating and nonparticipating farmers,
the circumstances under which they made a participation
decision, the amount of the incentive to which they
did or did not respond, and regional and other variables.
A close look at outcomes associated
with the Conservation Compliance provision of the
1985 Food Security Act reveals the importance of
isolating the effects of the program in order to
measure its success. The provision requires agricultural
producers to implement soil conservation systems
on highly erodible (HEL) cropland to remain eligible
for farm program payments. Annual soil erosion on
U.S. cropland declined by 40 percent between 1982
and 1997, suggesting that compliance mechanisms
encouraged greater conservation effort. However,
erosion also declined on cropland not subject to
compliance requirements, demonstrating that other
factors must also have played a role in reducing
soil erosion. On farms for which conservation practices
could have increased net returns to farming, for
example, adoption may have eventually occurred regardless
of effects on soil erosion. In fact, after accounting
for other factors, such as erodibility, commodity
program payments, and land use changes, ERS research
shows that only about 25 percent of overall erosion
reduction between 1982 and 1997 could be directly
attributed to Conservation Compliance. Even on the
HEL lands targeted by the provision, about 11 percent
of erosion reduction during that period was due
to factors other than Conservation Compliance.
Linking Farmers’
Choices to Environmental Quality
Measuring changes in farmers’
practices that result directly from conservation
program changes tells only part of the story. Conservation
programs are not designed simply to induce a change
in conservation practices, but to change those practices
in order to improve water quality, air quality,
wildlife habitat, or a host of other environmental
attributes. More and more frequently, conservation
programs aim to improve all of those environmental
attributes at once.
Connecting the dots that link
a program’s incentives to success in achieving
that program’s environmental goal(s) is difficult
in general, but can be especially challenging when
evaluating conservation programs. Most of these
programs address “nonpoint” sources
of pollution, such as the nutrients, sediments,
pesticides, and salts that enter water diffusely
in runoff. In comparison to “point”
sources, such as factories and municipal plants,
which discharge through a pipe, ditch, or smokestack
on which a meter can be installed, nonpoint sources
are not so easily measurable and have an environmental
effect only in the aggregate.
For example, the goal of a particular
conservation program might be to address water quality
problems caused by agricultural production. Evaluating
a program based on that objective would require
data on the entire set of actions and outcomes associated
with agricultural production. Farmers control their
inputs and crop production practices. Their management
decisions, including which crop is produced on which
field and with what combination of inputs, can affect
water quality, but gauging whether or not and how
much they actually do affect water quality is a
difficult task. Farmers’ decisions may lead
to field-level emissions (through runoff or leaching)
of potential pollutants, such as sediments, nutrients,
and chemicals, which are difficult to monitor. Depending
on the location of the field and other physical
and environmental factors, an emission may or may
not find its way to the target water body.
But even that sequence of events
is only part of the story. The last piece involves
the underlying objective: What is it about water
quality that concerns us? Is the goal to reduce
nutrient concentrations in drinking water? Is it
to provide improved fish habitat, perhaps to increase
recreational fishing benefits? Once a (potential)
pollutant reaches an environmental sink, such as
a river or aquifer, it may or may not have ecological
or human health implications, depending upon its
toxicity, the number of other sources emitting the
same pollutant, interactions with other pollutants,
and the total emissions simultaneously reaching
the environmental sink. While scientists know much
about the relationship between nitrogen runoff and
tillage practices, and the effects of nitrogen levels
on biological functions, less is known about how
nitrogen is transported from a myriad of individual
fields to specific water bodies or other sinks.
In evaluating the effects of a
conservation program on environmental quality, the
nonpoint source issue is compounded by the exceptional
site specificity of many agri-environmental events.
Soil losses (or other pollutants) at one location
may have a different effect on the environment than
an identical level and type of soil loss in another
location. Furthermore, similar levels of environmental
effects vary in value among locations depending
upon the proximity of human populations or economic
activity to the site of the damage. For example,
if a program objective is to help restore a recreational
fishery, water quality improvements that increase
fish populations closer to cities and where interest
in fishing is particularly high will be higher valued
than equivalent changes in fish populations in regions
of the country that are sparsely populated or where
interest in fishing is low. Estimating monetary-equivalent
values for environmental improvement is a particularly
difficult task that, while not necessary for judging
whether or not a conservation program met its goals,
is essential to determining how efficiently those
goals were met.
Models Simulate What We
Cannot Observe
Environmental process models can
help overcome the nonpoint source and site specificity
complications of conservation program evaluation
by substituting predictions from models for direct
observations of effects. For example, site-specific
changes in (in-field) soil erosion due to particular
erosion control practices can be estimated using
the Universal Soil Loss Equation and the Wind Erosion
Equation. Both models provide reasonably accurate
results and require only minimal data (a total of
six variables) describing climate, topography, soil,
and cropping information at the field level. In
contrast, models of nutrient and pesticide runoff
are far more complex, simulating multiple environmental
effects from the transport and fate of multiple
pollutants into environmental sinks. These “fate
and transport” models require a lot of data,
often necessitating the use of dozens of variables.
Any one process model has unique
advantages and disadvantages, depending on the indicator
of interest, but relatively few are capable of simulating
the environmental effects of changes in agricultural
practices on a national scale. (See box, “Some
Agri-Environmental Process Models.”)
Some Agri-Environmental Process
Models
A myriad of agri-environmental
process models exist, ranging from simple
linear calculations suitable for a handheld
calculator to extraordinarily complex computer
programs requiring high-powered machines and
extensive training to operate, and from those
calibrated to a single watershed to models
developed to provide national-scale estimates.
Three process models with acceptance among
a wide range of analysts include one that
is particularly comprehensive and predicts
emissions at “edge of field” and
two that attempt to link practices to water
quality.
• USDA’s Erosion-Productivity
Impact Calculator (EPIC)—a mechanistic
simulation model used to examine long-term
effects of various components of soil erosion
on crop production. The model has several
components: soil erosion, economic variables,
hydrologic conditions, weather, nutrient composition,
plant growth dynamics, and crop management
(www.brc.tamus.edu/epic/).
• USDA’s Soil
& Water Assessment Tool (SWAT)—a
river basin scale model developed to predict
the water quality impact of land management
practices in large, complex watersheds. Required
input data include weather, soils, crops,
pesticides and nutrients (www.brc.tamus.edu/swat/"l).
• U.S. Geological
Survey’s SPAtially Referenced Regressions
On Watershed Attributes (SPARROW)—a
statistical model that relates in-stream water-quality
measurements to spatially referenced characteristics
of watersheds, including contaminant sources
(such as farm fields) and factors influencing
terrestrial and stream transport (http://water.usgs.gov/nawqa/sparrow/). |
A final complication: Model results
are unlikely to match real world observations because
farming practices aren’t the only things that
affect environmental quality. Floods or drought
can damage the environment even under the very best
management practices. A given level of runoff may
cause no environmental damage in a wet year but
may significantly harm fish and wildlife in a dry
year when streams have insufficient flows to dilute
the runoff to nonharmful levels. Likewise, a single
watershed may well experience pollutant discharges
not only from agriculture, but also from industrial
sources, municipal water treatment plants, urban
runoff, aerial deposition, and even natural seepage.
Thus, the influence of unmodeled events needs to
be extracted to reconcile simulation results with
measurements made on the ground.
Identifying Appropriate
Environmental Indicators
Just what is the best indicator
by which to measure environmental quality change
in the policy evaluation context? Regardless of
whether it will be measured directly or simulated
with an agri-environmental process model, the indicator(s)
by which a given program will be evaluated must
be carefully selected. Reflecting broadened public
concerns, conservation programs increasingly target
multiple environmental quality goals. Along with
reductions in soil erosion, potentially measurable
goals have expanded to include improved water quality
and conservation of wetlands and wildlife habitat.
Newer program objectives may include preserving
open space, managing nutrients from fertilizers
and livestock waste, reducing pesticide runoff,
improving air quality, reducing greenhouse gas emissions,
or sequestering carbon in soil.
|
|
The appropriate indicator for evaluating
a program’s success must map to an aspect
of environmental quality that the program aims to
address. But that’s not enough. It must also
link directly to those changes in conservation practices
induced by the program. For example, a measure of
ambient downstream water quality, such as nitrogen
concentration, may appear to be an ideal indicator
of the success of a conservation program that aims
to improve water quality. But if agriculture is
only a small part of the aggregate water quality
problem, ambient water quality may be getting worse,
even with a wildly successful conservation program
in place. The ambient water quality indicator may
not measure the factor of interest, which, in this
example, is agriculture’s contribution
to water quality, and thus is not a good choice
for evaluating this agri-environmentally oriented
program. In this case, a less direct measure of
water quality, such as pounds of nitrogen discharged
into the water body from farm fields, may actually
be a better indicator.
Appropriate indicators are:
-
Policy relevant—provide
a direct link to both the environmental attributes
of concern and the behavioral changes
associated with the evaluated program incentives;
-
Measurable—based on
sound science and make use of data that are
available or could feasibly be collected;
-
Reasonably priced—cost-effective
in terms of data collection, processing, and
dissemination; and,
-
Easy to interpret—communicate
essential information to policymakers and other
stakeholders.
Putting It All Together
The voluntary nature of most U.S.
conservation programs, the human factors involved
in farmers’ decisions to participate (and
to what extent), the complexity of farm household
decisionmaking, and the nonpoint source and site-specific
nature of agri-environmental problems combine to
make evaluation of conservation programs a data-intensive
and technically challenging process. To be successful,
program evaluations must answer both of the following
questions explicitly, through estimated, simulated,
or directly measured means.
1. How do different farm operators in different
circumstances decide what production and conservation
practices to implement, in the presence and
absence of the conservation program being evaluated,
at different levels of incentives provided by
that program?
Isolating the unique effect of conservation
program incentives on farmers’ practices
requires analysis to extract the influence of
other (policy, household, general economic, etc.)
factors that affect farm-level decisionmaking.
This, in turn, requires evaluators to collect
data on the full set of factors potentially affecting
farmers’ decisions, in sufficient volume
and across diverse farm and land types and locations,
to allow statistical segregation of program-related
effects from those of other influential factors.
2. How do the farm practices attributable to
conservation program incentives affect environmental
quality?
Isolating the unique effect of
farm practices on environmental quality requires
program evaluators to determine where, and under
what resource conditions, practices implemented
in response to the program are located, and to designate
appropriate agri-environmental indicators for measuring
program success. Process models that simulate the
complexities involved in the transport of agricultural
runoff from multiple fields to environmental sinks
may help link environmental performance with farm
practices. But even then, additional analysis is
required to reconcile model predictions with real
world observations.
The complicated series of cause-and-effect
relationships associated with conservation program
evaluation seem beyond even the imagination of Rube
Goldberg. Many factors must be accounted for to
determine the portion of environmental enhancements
directly attributable to program incentive-induced
changes in farmers’ practices. Still, carefully
designed survey and monitoring programs encompassing
each of those relationships in a coordinated fashion
make such evaluation not only feasible, but well
within reach.
|