The validity of UNAIDS/WHO estimates of the burden of HIV/AIDS is rightly questioned by politicians, scientists, and activists--especially since the 2003 estimates to be released in July, 2004, will show substantial drops in the burden of HIV/AIDS in several countries, and increases in others. However, the estimates are based on an explicit attempt to meet criteria we believe should guide the generation of international morbidity and mortality figures. These criteria extend beyond the quality of the input data to include features of the estimation process such as transparency and participation. The 2003 estimates now include plausible ranges for estimates rather than a single best estimate. This reduces the chance that insignificant differences in estimates from different sources are given importance. Here, we describe the levels of uncertainty associated with the UNAIDS/WHO estimates of HIV/AIDS. We explain the reason for moving to the use of plausibility bounds, the factors that determine the width of the bounds, and the implications for policy makers and programme managers. UNAIDS and WHO have produced country-specific estimates of HIV prevalence and its demographic effect every 2 years since 1997.1-4 These estimates have provided important information about the level and scope of the HIV pandemic, and served as a basis for advocacy. Effective advocacy requires that the audience has confidence in the reported estimates. Questions about the validity have arisen because the prevalence estimates have been adjusted over time, because different values can be used to support particular political stances, and because various organisations have generated radically differing data. The new HIV/AIDS estimates for end of 2003 will be lower than those reported in 2002.3,4 This difference is mainly the result of changes to estimates in sub-Saharan Africa. The scale of changes in the estimates varies by country, but figures are notably lower in Cameroon, Ivory Coast, Ethiopia, Kenya, Rwanda, Zambia, and Zimbabwe, and notably higher in Senegal.3,4 On the whole, these changes are largely the result of data inputs and estimation methods rather than real changes in prevalence. Such fluctuations can undermine public confidence and allow competing estimates based on less valid methods and data to be received equally. We believe, however, that fluctuations in the UNAIDS/WHO estimates are the result of improvements in the validity of the estimates. These improvements are sufficiently important to offset the negative effects that they might have on public confidence and the ease of estimating time trends. A set of criteria can be used to assess the quality of the estimation process. We propose that good estimates are: (1) based on all relevant data whose methods of collection are transparent; (2) derived using transparent processes and evidence-based assumptions that are sufficiently documented to permit replication; (3) open to change over time, as new data become available to refine both the methods and assumptions; (4) systematically compared, when possible, with other data sources or approaches; (5) useful in determining trends; and (6) accompanied by estimates of uncertainty and bias and discussion of their sources. Here we review the process, data, methods, and assumptions used to produce the UNAIDS/WHO estimates of HIV/AIDS and describe the accompanying uncertainties to illustrate the attempts made to meet these criteria. We describe the approach we have used to produce plausibility bounds and discuss the implications of the uncertainty around the estimates of HIV/AIDS for public health policy makers and HIV/AIDS programme managers. The methods
that WHO
and UNAIDS have used to produce previous rounds of country-specific
estimates
of HIV/AIDS have been described elsewhere.5-7 Methods and
assumptions
used to model the epidemics and arrive at the estimates have evolved
during
this time, guided by an external group of scientists and researchers,
The
UNAIDS Reference Group on Estimates, Modelling, and Projections.8
To date, reports have used different approaches for so-called
generalised
epidemics, in which national prevalence in pregnant women exceeds 1%,
and
low-level or concentrated epidemics in which HIV is concentrated in
groups
with high-risk behaviours.
In countries with generalised epidemics (mostly in sub-Saharan Africa), where spread of HIV is assumed to be predominantly caused by sex between men and women, estimates of adult prevalence are based on data from women in antenatal care. Epidemic curves for geographically or epidemiologically distinct populations (eg, urban and rural epidemics in different states or provinces) are fitted to time-series HIV prevalence data. The prevalence of HIV, as calculated from antenatal-clinic data, is adjusted to reflect the ratio between the prevalence in antenatal clinics and that in women of reproductive ages and in all adults, as judged by regularly updated reviews of studies comparing HIV prevalences in specific populations. These adjustments account for changes in fertility of women infected with HIV and differing rates of HIV-infection between women and men during the course of the epidemic. The combination of these curves is then used as the basis for the national estimate of adult HIV prevalence.8,9 In countries with low-level or concentrated epidemics, transmission is assumed to occur mainly in groups at high risk of HIV infection (such as sex workers and their clients, injecting drug users, and men who have sex with men) and their sexual partners. Here, estimates are made for each group and then summed to obtain an estimate of the number of adults with HIV. As for generalised epidemics, this process can be applied separately for different regions within a single country. This process is repeated for several years' data and then a curve is fitted to these points to create a national epidemic prevalence curve.10 For every country, these prevalence curves are translated into estimates of adult HIV incidence and AIDS-mortality, and child HIV incidence, prevalence and AIDS mortality, based on estimates of how long adults and children survive after infection with HIV and the probability of mother-to-child transmission.11 The role of national AIDS programmes Over time, involvement of individual countries has increased from merely commenting on provisional estimates to encompassing preparation of estimates, the following of local training in estimation methods, and the provision of software implementing these methods.1 Between the 2001 and 2003 estimates, UNAIDS and their technical partners--WHO, UNICEF, and the US Centers for Disease Control and Prevention--ran a series of training workshops in which epidemiologists from more than 130 countries were trained in UNAIDS/WHO methods.6-8 Such increased involvement has allowed country representatives to prepare provisional estimates, which were then reviewed at national and international levels, leading to agreement on a final country-specific model and a resultant set of estimates for 2003. This process increased involvement by national programmes, national statistics offices, and other government and academic organisations, and resulted in improved quality of estimates through the use of additional data and the application of local knowledge. Sources of uncertainty and plausibility bounds A major limitation of prevalence estimates for HIV (and for most other diseases) is that input data are often drawn from populations other than those for which the epidemiological parameters are being estimated. For HIV, prevalence is most often measured in facility-based surveys that are unlikely to include people fully representative of the broader population. National estimates of the HIV prevalence and incidence and AIDS-associated mortality are derived from these prevalence inputs, together with several additional assumptions. In countries with generalised epidemics, estimates of HIV prevalence, to date, have been based primarily on residual blood samples from pregnant women in antenatal clinics.12 In the absence of population-based surveys that include tests for HIV antibodies, sentinel surveillance of women attending antenatal clinics provides the best available estimate of HIV prevalence in the population. Studies have shown that high proportions of women in areas of sub-Saharan African badly affected by HIV have access to these services.13 In addition, women visiting antenatal clinics are healthy, avoiding the sampling bias inherent in testing sick individuals. Validation studies have shown that estimates based on antenatal sentinel surveillance provide a good approximation of HIV prevalence in adults aged 15-49 years (men and women combined) in the local community.11 However, with the exception of South Africa, clinics that participate in sentinel surveillance do not represent a random sample of all clinics in a country. Surveillance systems often select sentinel sites in clinics located in urban or periurban areas, both for ease of access and because these clinics serve a large number of pregnant women and can, therefore, yield sufficient sample sizes during data collection. Often, this sampling pattern leads to few data being available from pregnant women in rural areas. In the last two rounds of UNAIDS/WHO estimates,2,3 an adjustment was made to correct for this bias by lowering prevalence from non-urban areas by 20%. However, this adjustment was not always sufficient. For example, in the Ivory Coast, the 20% adjustment was applied to all non-urban sites for 1999 and 2001 estimates. Since that time, however, Ivory Coast has expanded the number of rural surveillance sites. Based on these new sites, the median rural prevalence was estimated to be 5% instead of the 8% used in the 1999 and 2001 estimates. Incorporation of these new data, together with new assumptions discussed later in this article resulted in a 2003 estimate of adult HIV prevalence that is almost 25% lower than the 2001 estimate.4 A comparison of the concurrent prevalence in new sites with that in sites where prevalence was also available in previous years allows earlier, less representative estimates to be adjusted retrospectively, and for more appropriate trends to be generated. For countries with low-level or concentrated epidemics, facility-based samples are also frequently used in sentinel surveillance of key populations at high risk of exposure to HIV. For example, prevalence estimates for injecting drug-users may be based on clients of harm-reduction services and estimates of rates of men who have sex with men are derived from men presenting at genitourinary clinics who report having anal sex. These methods yield convenient samples for populations that cannot be validated reliably using population-based samples because self-reports of risk behaviour are likely to be biased. Estimates of other epidemiological parameters used in UNAIDS/WHO modelling are based on a thorough review of available data. Unfortunately, many of these data are also based on non-representative samples, and local differences in factors such as breastfeeding and birthing practices limit the validity of regional values applied in some countries. Plausibility bounds In earlier UNAIDS/WHO reports, we
provided point
estimates and represented uncertainty as one of three ranges (plus and
minus 20%, 28%, or 35%), depending on the quality of serosurveillance
data.2,3
Although this approach captured some sources of error, it did not
reflect
the varying levels of uncertainty in estimates for different countries.
Furthermore, presentation of a point estimate rather than a range
encouraged
a false perception of precision in advocacy messages.
The most important determinants of the size of the plausibility bounds are described in the panel and examples of plausibility bounds for various estimates are presented in the table. Primary factors that determine the size of the plausibility bounds around HIV estimates
Additional sources of uncertainty Additional sources of error or bias can
be difficult
or impossible to quantify and, although not captured in the calculation
of plausibility bounds, they should not be ignored. Errors resulting
from
problems with laboratory testing have not been considered in our
calculation
of plausibility bounds for estimates of HIV. Although there are many
studies
showing that the sensitivity and specificity of the commonly used HIV
tests
are very high,17 these values do not capture the effects of
poor handling and storage of samples, or the use of out-of-date tests
and
poor quality reagents. Although quality assurance programmes are in
place
in most countries to reduce the likelihood of this source of error, we
cannot assume that errors have been eliminated. For example, in
Zimbabwe,
samples for the year 2000 were retested using a more specific test kit
and HIV prevalence was found to have been overestimated by around 14%.18
Strategies for reducing uncertainty A systematic assessment of uncertainty surrounding UNAIDS/WHO estimates of HIV is long overdue. Nevertheless, there has been a continuing process involving reviews of additional data sources and validation studies in an effort to improve the methods and assumptions used in making estimates of HIV. The ability to do these types of validation studies will increase over time as new sources of data, such as from household surveys and expanded provision of prevention and care activities, become available. Triangulation A process of triangulation, drawing in other sources of independent data, can be used to check the plausibility of estimates. In countries with generalised epidemics, estimates of deaths derived from vital registration systems or censuses can be used to look at changes in the age patterns of mortality over time, resulting in independent estimates of deaths from AIDS. This approach has been used in Zimbabwe19 and South Africa,20 and has contributed to improved estimates of HIV. Likewise, to check for bias and to improve estimates of HIV prevalence and AIDS mortality, estimates of the number of children orphaned by AIDS from sentinel surveillance have been compared with estimates from household questionnaires.21 Use of new data sources The availability of effective interventions to delay progression to AIDS and death has also allowed improved measurement of prevalence. In recent years, it has become acceptable to include testing for HIV in large-scale household surveys. These surveys have the potential to improve the accuracy of HIV estimates because they can provide country-wide data on HIV prevalence for both sexes, and include samples from remote rural areas rarely covered by sentinel surveillance systems. Data from several of these studies (eg, in the Dominican Republic, Kenya, Mali, Niger, South Africa, Zambia, and Zimbabwe) have recently become available, and have led to important refinements in the national estimates for 2003 and in the calibration of assumptions about differences between urban and rural populations and between sexes. However, even results from population-based sample surveys are subject to bias, especially when there are high rates of absence from the household or refusal to be tested. An important challenge is to understand and measure these biases.22 Country-level review of assumptions and data sources A third process that is leading to improved estimates and reduced uncertainty is the country-level review of assumptions and data sources used in making estimates. For example, the rate of HIV infection captured by a recent household survey in Kenya23 was lower than expected in view of previous estimates from sentinel surveillance of pregnant women at antenatal clinics.23 Local review suggested that in Kenya, prevalence in pregnant women was actually somewhat higher than in the whole adult population due to a female-to-male HIV-prevalence ratio of almost 2:1. This review led to major adjustments in the national estimate for Kenya. A similar exercise in Zambia found a close correspondence between adult prevalence in the household survey and that recorded for women in antenatal care.24 However, researchers noted that several urban sites had been misclassified as rural in previous estimates. Correct classification of sites resulted in a much lower estimate of HIV prevalence in rural and national areas of Zambia. The figure shows the effect of these changes on the epidemic curve for HIV/AIDS.  Estimated number of people living with HIV/AIDS
in Zambia over time (1980-2003): end-2003 model vs end-2001
model Increased involvement of national programmes has resulted in more correct definitions and epidemic curves for subnational geographic areas. In previous estimates, UNAIDS/WHO used single urban and rural curves for most countries in Africa. Now, many countries develop three or more curves to describe different epidemics based on location or population. This process has been facilitated by the increased number of surveillance sites in many countries. Implications for trend analyses Estimates that are improved through the incorporation of better methods or data make it difficult to estimate trends over time. In addition, there is a danger that these adjustments in the estimates can be confused with actual epidemic trends. However, when changes are made, new estimates for the entire course of the epidemic are made (eg, figure for Zambia) allowing a full analysis of the trends in the epidemic. UNAIDS and WHO now publish estimates not only for the most recent year (2003) but also for the year of the previous estimate (2001), to clearly show the estimated trend in the epidemic. As shown in
the table,
the 2003 plausibility bounds are large for some HIV estimates, which
begs
the question--are estimates adequate to support sound public-health
decision-making?
We believe the answer is yes.
One reason why questions about validity focus heavily on estimates of HIV compared with those for other diseases might be that earlier UNAIDS/WHO estimates concentrated on point estimates, albeit with ranges intended to alert readers to the imprecision. This reporting may have led readers to infer an unwarranted level of certainty about the estimates. We have tried to correct this in the end-2003 estimates4 by replacing ranges with plausibility bounds, and by alerting readers to this issue through articles such as this. Inappropriate emphasis on point estimates has been a common practice, not limited to HIV. The WHO, in its yearly World Health Report, has traditionally presented estimates of cause-specific mortality at the global level as single points, without accompanying ranges or other forms of uncertainty estimates. The magnitude of uncertainty for these estimates, however, can be surmised by reviewing the variability in selected point estimates over short periods of time. In the World Health Report, the proportion of all deaths due to acute lower respiratory infections in 1998 was noted to be 6·5% (3·5 million deaths),25 and in 1999, 7·1% (4 million deaths).26 Estimates of other diseases have varied even more widely. In the 1996 and 1997 World Health Report, the number of deaths attributed to malaria was 1·5 and 2·7 million, respectively,27 but 2 years later had dropped to 1·1 million.25 A good part of these swings of 10% or more reflects error--or stated more positively, the incorporation of new and improved information, models, and assumptions, rather than true changes in the burden of disease. All disease-specific estimates have associated error, which can be quite large. Such error applies to the UNAIDS/WHO estimates of HIV as well as to epidemiological estimates for other diseases. However, not all estimates and
projections are
equally valid. Based on the set of criteria presented in the opening of
this paper, we believe that HIV estimates are valid indicators of the
level
and scope of the AIDS epidemic. The UNAIDS/WHO estimates of HIV and the
processes used to generate overall estimates in the past 7 years
reflect
an ongoing effort to meet these quality criteria. Other programmes
within
the WHO have adopted the concept of an independent reference group
similar
to that described here for HIV/AIDS. The best example may be the Child
Health Epidemiological Reference Group, which has developed systematic
and transparent methods for the estimation of the major causes of child
mortality.28
At a global level, the corrections
reported in
2004 should be welcomed. Although changes in estimates do raise some
concerns
about the validity of previous estimates, the willingness to change and
update estimates as new methods and data become available also shows
that
a self-correcting process is in place for making estimates of HIV/AIDS.
Conflict of interest statementAcknowledgments We acknowledge the contributions of the many researchers and scientists who have participated in the UNAIDS Reference Group on Estimates, Modelling and Projections. Their contributions have been invaluable in improving the quality of estimates and in setting the agenda for future work. We thank the many national programme managers and epidemiologists who have worked diligently to monitor the HIV epidemic and who have shared their expertise and national data to help improve the estimates of HIV and its effect; Elisabeth Zaniewski for data management; and Catherine Hankins and Jennifer Bryce for comments and suggestions on previous versions of this manuscript. The authors have received
funding
from UNAIDS, The Royal Society, and the UK Medical Research Council.
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||