Table of Contents

Appendix 12: Response to Comments provided by a Review of the Modeling Techniques Used

In February of 2004 a review of the modeling of the risk assessment was conducted by two reviewers, one internal and one external with expertise in @RISK and Monte Carlo simulation. See copy of Carrington (2004) and Donahue (2004) for the full review.The VPRA team requested the reviewers to focus on the following issues:

1. The appropriateness of the general modeling approach adopted (e.g., the regional/seasonal "segmented" structure, no temporal structure within each region/season segment) and whether or not the level of model detail is consistent with the quality and quantity of data that was identified.
2. The appropriateness of assumptions made with respect to modeling and specification of variability and uncertainty distributions.
3. The appropriateness of selected parametric models used for summarizing available datasets, the methods of estimation used, and whether or not effects of model uncertainty are adequately addressed and discussed.
4. The appropriateness of the selected statistical methods of analysis for sensitivity assessment of influential variability and uncertainty factors.
5. The appropriateness and correctness of implementation of the model specification in @Risk (e.g., possible coding errors).
6. The appropriateness of selected sample sizes for Monte Carlo simulations (1,000 uncertainty samples versus 10,000 variability samples).

Several substantive comments were received from the reviewers with respect to these (and other) modeling issues. Below is a summary of the reviewer's major comments and FDA's response to these comments.

Comment 1

Geographical and seasonal variation currently described by segments (or scenarios) could be described (coded) in correlated distributions, which would facilitate evaluation of the effect of intervention strategies on an annual and national basis.

FDA Response to Comment 1

A separate simulation of correlated distributions for a national estimate of public health impact of baseline risk and mitigations is possible. We simulated the region-seasons separately because we wanted to see the impact of mitigations on a regional-seasonal basis. Since we had simulated the region-seasons, it was simple to get a national estimate from these data as opposed to a separate simulation for a national estimate. While the suggested approach of this comment is helpful in looking at national estimates apart from regional seasonal impacts, we concluded that implementing the suggestion at the present time was not necessary in relation to achieving the stated goals of the risk assessment.

Comment 2

With respect to appropriate specification of the effect of uncertainties, the assessment does not include the range of all plausible interpretations of the data and this is particularly evident with respect to uncertainty of the dose-response and the growth rate model. In particular, the assessment evaluates three possible dose response models but the identified uncertainty is not carried forward in an integrated fashion.

FDA Response to Comment 2

We attempted to identify and appropriately include all relevant uncertainties in a consistent and balanced manner. With respect to uncertainty of the growth rate model we were limited in how this could be addressed because the raw data was not available (including effect on different strains). Predictions were therefore based on the summary model fit information provided in the cited reference (a log₁₀-linear primary growth model with a secondary model of the square-root type). The extent to which use of alternative models would produce substantially different predictions depends on the degree of extrapolation away from the range of the experimental conditions and there is relatively little extrapolation away from the time-temperature range of these data. The primary extrapolation is from broth cultures to growth conditions within the oyster, with a relatively large uncertainty being specified for this extrapolation. As to the identification of dose-response uncertainty we did not carry forward model uncertainty for two principle reasons. First, of the three models considered, the Beta-Poisson is the only one which is low-dose linear; a characteristic which is reasonable a priori based on mechanistic considerations (FAO/WHO, 2003). Second, after anchoring each model (in turn) to the epidemiological data it was found that the residual uncertainty of risk predictions for Gulf Coast summer (the region/season with the largest number of attributed illnesses) was comparable across these three different models (Appendix 4). Anchoring each model separately was considered appropriate since, in this instance, the epidemiological estimate of average annual illness burden is effectively being utilized as a "datum" for the purpose of estimation.

Comment 3

With respect to sensitivity analysis, a method that examines the correlation between input percentiles (rather than values) and the output variable may be preferable. Any appropriate method applied to the uncertainty dimension is useful for planning research but in the variability dimension such analyses may not be useful unless targeted at distributions (or portions of distributions) that can be controlled.

FDA Response to Comment 3

The observation that sensitivity analyses may not be useful when applied to variability factors that are not controllable is a valid point. Here we have used sensitivity analysis as applied to variability factors as a means of summarizing the behavior of the model rather than limiting the analysis to just the controllable factors per se. Thus, while we have identified total V. parahaemolyticus/g in individual servings as an important variability factor we do recognize that this not controllable on a serving by serving basis. More refined sensitivity analyses limited to controllable variability factors could be developed at a later time. As to preference of a method comparing the output to percentiles of the input it is our understanding that this is most relevant when there are pronounced thresholds and discontinuities (e.g., growth/no-growth boundaries). With the exception of some low temperature region/seasons, such threshold behavior is atypical of the present model.

Comment 4

With respect to appropriateness of selected sample sizes for Monte Carlo simulations, use of the median rather than the mean as a central estimate of the distribution of uncertainty in output variables would mitigate any concerns that the central estimate is driven by potentially erroneous expression of the tail of the uncertainty distribution.

FDA Response to Comment 4

We have not looked at the effect of anchoring the dose-response with respect to the median as opposed to the mean of the uncertainty distribution. Future work with the risk assessment will examine this issue.

Comment 5

The general segmented structure of the model (region/season) is justified based on the data that modelers had to work with but the justification of this region/season approach could be better documented in the technical document.

FDA Response to Comment 5

We have amended the document to better justify the region/season approach.

Comment 6

With respect to appropriateness of selected parametric distributions used for modeling, the distributions (i.e., Normal) used to model the water temperature are not as accurate and precise as they could be and this may impact on the predicted densities of total V. parahaemolyticus and the number of pathogenic V. parahaemolyticus. As shown in Appendix 4 these data are (typically) skewed and this fact, compounded with the uncertainty arising from selection of only one point (or buoy) to represent the temperature of an area may have a significant impact on the modeling results. Other models of water temperature (such as a bounded Beta variate) may be appropriate given that the sensitivity analysis in Appendix 5 shows that the water temperature parameters are significant.

FDA Response to Comment 6

Although a parametric distribution could be utilized that better represents the skewness of the temperature data, there is a trade-off between fidelity of representation of the data and utility of the model. The choice of the Normal distribution to summarize the water temperature data for the model simulations was based on the judgment that the discrepancy of predictions resulting from use of a fitted Normal rather than the empirical distribution of the data was a relatively minor "cost" to pay for more utility or ease of use (e.g., interpretability). On a practical level, the model would be much more cumbersome if the empirical distributions of water temperature data (or bounded Beta variates) were used rather that the Normal approximation. With respect to utility and interpretability, the potential effect of year-to-year variations of temperature distributions (i.e., extreme temperature events such as El Niño or La Niña) was initially identified as a potentially important factor to be considered in the assessment. Appropriate assessment of the effect of year-to-year variability of temperature distributions requires an effective summary of year-to-year differences in the temperature data. It is unclear how this could be effectively accomplished based on either empirical distributions of a limited number of years of temperature data or the parameters of bounded Beta variates fitted to these data. As to the magnitude of the impact of using a Normal approximation rather than the empirical distribution of the temperature data, simulations where conducted using the NBDC Gulf Coast temperature data and the maximum likelihood estimate of the V. parahaemolyticus/g versus water temperature regression relationship. The simulations indicated that the alternative specifications of water temperature distributions result in predictions of mean log₁₀ V. parahaemolyticus/g at time of harvest which have a relative difference of <1% across all years and seasons of the temperature data. Relative differences in mean V. parahaemolyticus/g at time of harvest are larger with a range of up to a 10% relative difference for some of the year and season specific data; however, the average relative difference was only 2%. Thus any infidelity of representation of the skewness of the water temperature data (within a given year and season) does not appear to have a substantial impact, and this is further validated by the comparison of model simulation output to data on V. parahaemolyticus/g at time of consumption.

Comment 7

The estimation of the dose-response deserves further attention. As illustrated by the sensitivity analyses, the impact of the dose-response uncertainty is substantial. As such, other sources of dose-response data should be considered. In the absence of better data or modeling methods, the impact of this uncertainty (as a weakness of the model) should be better identified in the technical document and interpretive summary.

FDA Response to Comment 7

We have amended the document to better explain our use of dose-response data and the impact of the dose-response uncertainty on the estimates of risk.Since the goal of the risk assessment was to (1) examine the factors that contribute to the risk, (2) examine the differences between different regions/seasons, and (3) evaluate the impact of potential mitigations, the dose response curve is not something that is varied in any manner during the risk assessment. Accordingly, it can almost be viewed as a constant for the risk assessment that was not changed, so the uncertainty was a constant for all factors and "what-if scenarios".

---

Table of Contents