Summary of Comments Received on BMDS and EPA Responses

• 2007 External Review of New Quantal Models
• 1999 External Review of Original BMDS Models

2007 External Review of New Quantal Models

New quantal dose-response models having alternative background parameters were externally reviewed by experts in June, 2007. These models are available for testing in the beta test version of the new BMDS 2.0 beta. Reviews were generally positive and confimed that the functioning of the computer code has been rigorously tested. The reviewers' comments and EPA's responses are provided here, with the model development and testing report.

The new models are modifications of the current quantal models in BMDS. They were developed by modifying existing source codes, changing the code only as necessary. Addition of these new models to the suite of models in BMDS expands the range of alternatives available for fitting dose-response curves using generally accepted types of models.

Five new models implement an alternative form of background term, one that is additive to the dose. These are the multistage, cancer, log-probit, gamma, and Weibull models. The current versions of these models have a background term that is directly additive to the response probability. Models with background terms additive to dose were proposed and analysed in peer-reviewed papers by leading experts over 20 years ago, so they are not new to risk analysis. One type of model, log-logistic, will not be provided in a version with background additive to dose, because of technical difficulties causing unreliable convergence on solutions.

Two quantal models, probit and logistic, that currently have no explicit background response parameter will become available in that form.

Thus, with the exception of the log-logistic, all the BMDS quantal models will be available in BMDS 2.0 beta in two versions, differing in the background term. Current versions of quantal models keep the names used in BMDS 1.4.1c. New versions with an alternative background parameter are distinctively named to identify them clearly, e.g., "Weibull-BgDose", or "Probit-BgResp".

1999 External Review of Original BMDS Models

Numerous positive comments were received by reviewers impressed with the capabilities, potential and overall user friendliness of BMDS Version 1.1b. In addition, a number of comments were received that pertain solely to benchmark dose assessment methods (guidance) issues.  For the sake of brevity, these types of comments are not addressed here.   Questions and concerns relating to benchmark dose methods should be submitted during public review of the Agency's "Benchmark Dose Technical Guidance Document," which received an EPA internal review in February, 2000.  The following table summarizes the major criticisms received from more than two dozen reviewers of BMDS version 1.1b organized by major topic area and EPA responses.  It is not intended to be a comprehensive list.

NOTE:  Several positive comments and suggestions were received concerning the graphics (plotting) facilities in BMDS.  Please note that BMDS uses a self-contained plotting program called Gnuplot (BMDS version 1.2 uses MS-Windows 32 bit version 3.7, patch level 1).  During installation, the files wgnupl32.exe and wgnuplot.hlp are copied to a subdirectory of the main BMDS directory called "gp371w32."  All plotting features and the online help system of Gnuplot program are under the control of the authors (Thomas Williams, Colin Kelley and many others; Copyright(C) 1986 - 1993, 1998, 1999). 
A list of Gnuplot frequently asked questions (FAQ) is available from http://www.ucc.ie/gnuplot/gnuplot-faq.html. 
Comments and requests for help with Gnuplot should be sent to  info-gnuplot@dartmouth.edu. 
Bugs, suggestions and mods should be sent to bug-gnuplot@dartmouth.edu.

#	Comment (Approximate Date Received)	Response
1.	Create/Edit Screen Comments	Create/Edit Screen Responses
a.	"Mapping of data type to column should be saved when data is saved so that on reloading the same data, the selections are known" (1/1/99)	Program revised.  The Create/Edit Screen of BMDS has been improved in several other areas as well, including allowing for use of keyboard and mouse commands similar to commercially available spreadsheet software, and  cutting and pasting from inside and outside BMDS.  Associating columns with model variables is also easier, since variable choices can now be linked via user defined column headers.
b.	The program reacts in odd ways when "the user accidentally assigns the same column to more than one data type"  (1/1/99)	Program revised.
c.	"Please have the model type saved on the INPUT SCREEN with the data set so the user does not have to re-enter it each time."  (3/8/99)	Program revised.
d.	"When the user goes from the model screen back to the edit data set screen the set values should remain.  It is annoying to have to reset values each time."  (3/8/99)  "... the program resets the choices to columns 2 and 3, respectively, making it necessary for the user to select choices yet again."  (3/31/99)	Program revised.
2.	File Management/Data Import Comments	File Management/Data Import Responses
a.	"I tried to create a data set with a long file name – the system said it was invalid and asked if I wanted to ABORT, I then went through WINDOW SCREEN hell..."  (3/8/99)	The interface now allows for long file names and is stable with respect to the creation and use of files in directories outside of the main BMDS directory.
b.	"I found it was too easy to overwrite an existing data set when entering a new data set & selecting an existing file name to modify."  (3/29/99)	Several checks and improvements have been added to the BMDS version 1.2 file management system to ensure flexibility and avoid confusion.  However, any confusion/difficulties associated with  windows 95/98 file management systems are largely unavoidable.
c.	"Although the BMDS is more user friendly than commercially available software, it is unfriendly in how data files are brought into the package.  If the data are in an ASCII file, the first variable in each row must be in the first column and only one space is allowed between columns in any row...It would greatly enhance the usefulness of this package if the data import facility were improved."  (3/31/99)	Several improvements have been made to the file import feature of BMDS, including these suggested revisions.  In addition, data from commercially available spreadsheet software programs (Excel and Lotus) can now be cut and pasted directly into the BMDS spreadsheet.
3.	Model Run Screen Comments	Model Run Screen Responses
a.	"Options on Run screen always reset to defaults when a new Run screen is loaded...default values should be customizable by the user" (1/1/99)	User selected options for each model run screen are now retained, even if the user closes the screen and returns to it later.  A "Set to Defaults" button has been added so that users can revert back to the "factory settings" if desired.
b.	"Changing selection of model within a type should be possible on the Run screen" (1/1/99)	This may be considered for future versions of BMDS, but was not deemed a priority at this time.
c.	The Append function only works for nested models.  (1/1/99)	The Append option has been removed.  (see response to 3.g. below).
d.	"...there should be a way to restrict the rho parameter to greater than or equal to one but such a restriction is not on the Run screen".  (1/1/99)	This was a mistake in the design of the model run screen for certain models and has been fixed.
e.	"For the continuous model screen, the program should figure out if the effect causes an increase or decrease in response and set that as the default (upper right hand corner of MODEL screen).  The user should be able to change it, but not have to remember to set it each time."  (3/8/99)	An "Automatic" option has been added which allows the user to let the computer decide which direction (up or down) is adverse.  Note:  This choice only impacts the BMR, BMD and BMDL estimations, not the estimated shape of the dose response curve.
f.	"Could “initializedâ€� values for some parameters be used and default values be used for others?"  (3/31/99)	No.  Writing a program to determine an appropriate "set" of initialization values, given a subset of the values is not an easy task, not a high priority and not doable in the time frame available.  However, EPA will gladly consider any suggested approach, particularly if it is presented in the form of an algorithm.
g.	"Actually the append is a good option, it just should not be the default."  (7/20/99)	The Append feature was removed because it was felt that its disadvantages outweighed its advantages.  A big concern was that it was leading to mistakes caused by users forgetting to scroll to the bottom of an output file, thereby associating the results at the top of an Append file with their most recent model run.  It also caused problems in trying to match textual output data with the associate graphical plot file.  The simplest resolution was to overwrite in all cases, and allow users to backup overwritten .out and .plt files to the level of backups that they desire.
4.	General Model Comments	General Model Responses:  see Results of EPA Quality Assurance Tests for more on improvements that have been made to BMDS models and bookmark the model update page to keep track of the latest model versions.
a.	A reviewer noted problems with the use of the hybrid and continuous models on epidemiological data and suggested adding an ability to handle covariate data.  (1/1/99)	Because it was designed primarily for the analysis of toxicological data, the current version of BMDS is capable of handling group data only (i.e., more than one individual per dose group) and does not allow for consideration of covariates (except in the case of the nested models).  Such tools for assessing epidemiological data will be given strong consideration as possible additions to future version of BMDS.
b.	BMDL Calculations:  "I get the following message on text results: 'Benchmark dose computation failed.  Lower limit includes zero.'  What does it mean?"  (2/1/99)	Version 1.1b of BMDS gave this type of message under two circumstances, when the BMDL associated with the BMD could not be estimated and when any of the BMDLs used in the plotting of the graphical BMDL curve (blue line) could not be calculated.  Version 1.2 now treats these circumstances differently.  In the latter case, BMDS model output now reflects the less serious nature of the computational failure, and the help manual suggests running the model without doing the BMDL curve calculations (not checking the “BMDL curve calc.â€� box).  In the former case, the user will still receive a "computation failed" message, but with changes that have been made to the BMDS models, this situation should be relatively rare.
c.	Threshold Parameter: "Why is the threshold model, THRESH R(d)=1-EXP[-q1(d-do)-....-qk(d-do)^k) not available in the BMDSoftware?"  (2/1/99) "EPA should provide a threshold model option in the BMDS"  (3/31/99)	In large part, the threshold parameter was not incorporated into the BMDS models to avoid the misconception that non-zero estimates for d0 (even if statistically significant) are evidence for the existence for a true threshold.  The threshold parameter d0 only estimates a (biological) threshold if the rest of the dose-response model is exactly correct.  This condition is virtually never satisfied, so the parameter d0 is generally just another parameter that helps the selected model fit the data, but reduces the degrees of freedom of the model.  In addition, additional parameters often have statistical properties that make the models harder to fit.   Nevertheless, EPA may have to add such parameters (we prefer to call them "x-intercepts" or "background-intercepts") to the BMDS models in the future, to accommodate otherwise difficult to fit data sets.
d.	"In several of the models when the estimation process does not seem to converge, BMD and BMDL estimates are presented anyway.  It is not clear how the estimates can be made from a non-converged model."  (3/8/99)	This is no longer the case.  In fact we have added a feature that prints out instructions at the end of a non-convergence output that describes/suggests how the user should proceed, in the advanced mode of BMDS, to obtain convergence.
e.	"Models are fit with parameters which are unstable (correlated at 1.0) so that the estimates can be biased to an unknown extent."  (3/8/99)	It is true that if parameters are perfectly correlated there is no unique solution for the parameter vector, however, the BMD and BMDL values are unique.
f.	"Some discussion of hormetic models would be helpful in the BMDS documentation."  (3/31/99)	BMDS models assume that dose response is monotonic.  The issue of hormesis and its significance is beyond the scope of this project.
g.	"Although the BMDS documentation contains some discussion of the need to assess model fit from several perspectives, it does not include criteria."  (3/31/99)	There is no discussion of model fit criteria within the BMDS documentation because that is not its purpose.  As is discussed above, fit criteria and  other BMD methods will be addressed in EPA's "Benchmark Dose Technical Guidance Document."
h.	"It would help if a cautionary note were printed when the fitted model is not monotone relative to dose."  (3/31/99)	As the reviewer points out, this information should be readily apparent from the graphical output of BMDS.  Writing a program to determine whether a fit is monotonic is not trivial, not a high priority and not doable in the time frame available.  However, EPA will gladly consider suggested approaches, particularly if presented in the form of an algorithm.
i.	"The dangers of extrapolating below the low dose should be explained more fully than is currently the case."  (3/31/99)	Again, this is an issue that is appropriately addressed in EPA's "Benchmark Dose Technical Guidance Document," not the BMDS software manual.  However, new information (e.g., Chi-square residuals) has been added to the model output pages of BMDS version 1.2 to assist users in determining the fit of the model in the low dose region of the curve.
5.	Dichotomous Model Comments	Dichotomous Model Responses:  see Results of EPA Quality Assurance Tests for more on improvements that have been made to BMDS models and bookmark the model update page to keep track of the latest model versions.
a.	"In the multistage discrete model the numerical values for added risk and extra risk estimates for the lower bound of the BMD are reversed..."  (3/8/99)	This was a bug in the user interface for BMDS version 1.1b that has been fixed.
b.	"In the multistage discrete model the degrees of freedom in the Analysis of Deviance Table shift around in non-standard ways as the degrees of the model changes."  (3/8/99)	The latest BMDS models (the models distributed with BMDS version 1.2) have been revised to correct these errors in the Analysis of Deviance Tables.
c.	In some cases, use of the Log transformed version of the Logistic model results in situations where "the estimate of benchmark-dose lower bound (BMDL) will be higher than the estimate of the BMD." (3/31/99)	The latest logistic and probit models (the models distributed with BMDS version 1.2) have been revised to correct these errors.
d.	"For discrete data, the choices are greater, though even here there are useful models not included, such as the negative binomial regression models of See and Bailer (1998)."  (3/31/99)	We appreciate your concerns and will consider your recommendations.  Future versions of BMDS will likely contain additional models that EPA determines would benefit risk assessors.  It is also possible that, as we gain experience in the use of this relatively new method, models found to be less useful will be revised or removed from the system.
e.	In the Weibull and Logistic models, the standard error calculations for estimated parameters do not appear to be correct.  (4/22/99)	The latest BMDS models (the models distributed with BMDS version 1.2) have been revised to correct these errors.
f.	Version 1.8 of the Multistage model produced "different p values for same deviance and df."  (7/16/99)	The latest BMDS multistage model (the one distributed with BMDS version 1.2) has been revised to correct these reporting errors.
g.	Version 1.8 of the Multistage model produced "no BMD estimates with background set to 0."  (7/16/99)	Setting background to 0 implies that it is impossible to have an affected animal in the control group.  If there is an affected animal in the control group the contribution of that group to the overall log-likelihood is -infinity (which comes from ln(0) that turns up in the expression).  In that situation, the value contributed by the control group is appropriately set to a large negative number, the likelihood of the fitted model is strongly affected by that arbitrary choice, and the  calculation of a BMD would not be appropriate.  Version 1.2 of BMDS currently issues a warning to the user when they try to fix background to zero in the presence of affected animals in the control group.  Fixing the background to zero should only be done when it is known with a fairly high degree of certainty that a control animal cannot have the affect being measured and modeled.
h.	"...standard errors for the estimates calculated by your package do not match the corresponding standard errors calculated in SAS."  (11/29/99)	The latest BMDS models (the models distributed with BMDS version 1.2) have been revised to correct these errors.
6.	Continuous Model Comments	Continuous Model Responses:  see Results of EPA Quality Assurance Tests for more on improvements that have been made to BMDS models and bookmark the model update page to keep track of the latest model versions.
a.	"In the linear continuous model a variance term is incorrectly labeled as a standard deviation" and the "numerical value of the standard deviation is questionable."  (3/8/99)	There was an error in our standard deviation calculation for some models.  The latest BMDS models (the models distributed with BMDS version 1.2) have been revised to correct this error.
b.	"In the model tests the likelihood ratio tests are done incorrectly (the wrong likelihood values are used in the equations..." and  "Some log-likelihood values are impossible."  (3/8/99)	The latest BMDS models (the models distributed with BMDS version 1.2) have been revised to correct errors in the continuous model fit tests and likelihood calculations.
c.	"At present, the models available for continuous responses are rather limited."  Several additional models were recommended.  (3/31/99)	We appreciate your concerns and will consider your recommendations.  Future versions of BMDS will likely contain additional models that EPA determines would benefit risk assessors.  It is also possible that, as we gain experience in the use of this relatively new method, models found to be less useful will be revised or removed from the system.
d.	"One particular problem that I've run into is running the continuous Hill model on this data set, Jurek.set [response increases quickly in low dose region, flattens or decrease at high doses."  (11/04/99)	The latest BMDS models (the models distributed with BMDS version 1.2) have been revised to do a better job of handling this kind of dose response data.
7.	Nested Model Comments	Nested Model Responses: see Results of EPA Quality Assurance Tests for more on improvements that have been made to BMDS models and bookmark the model update page to keep track of the latest model versions.
a.	A reviewer noted problems with the running of nested models on a data set that consisted of a large number of repetitions within dose groups and recommended that the software allow users to input a "repetition number" associated with a specific dose, response and litter size.  (1/1/99)	The problem was that the number of records (rows) in the data set exceeded the capabilities of the BMDS spreadsheet.  Version 1.2 of BMDS now allows users to define the number of rows and columns in the Create/Edit Screen spreadsheet (this was made a user option because very large dimensions can make spreadsheet operations significantly slower on some computers).  The enhanced spreadsheet capabilities of BMDS should reduce the necessity for a "repetition number" as rows of data can now be easily copied or transported from other (Excel or Lotus) spreadsheets programs.   Further, if the "litter-specific covariate" feature of BMDS is used for a covariate other than litter size, it is less likely that there will be repeating rows in a data set.  Nevertheless, if EPA continues to receive suggestions for a "repetition number" feature, it will be considered for a future BMDS version.
8.	Hybrid Model Comments	Hybrid Model Responses
a.	"BMDs agreed but our BMDLs were generally smaller than the ones predicted by BMDS"  and  "the cumulative Chi-square test sometimes does not return a p-value even when the chi-square statistic and degrees of freedom should allow the computation of a p-value."  (1/1/99)	The BMDS Hybrid model that has been packaged with BMDS Version 1.2 is still considered to be a beta model.  It did not receive much attention in 1999 (the original version distributed with BMDS version 1.1b was not significantly revised) and did not undergo the same Quality Assurance testing as the other BMDS version 1.2 models.  This is because EPA guidance/policy for the "Hybrid" assessment approach are not as fully developed at this time as for approaches involving the other BMDS models.  Thus, EPA has not corrected the problems identified in this model during the 1999 public review, but continues to request comments on both the procedure and the model.
9.	Cancer Model Comments	Cancer Model Responses
a.	"I would like to make sure that the models available in the BMDS are applicable for both carcinogens and noncarcinogens."  (2/1/99)	The models used in BMDS are in principle applicable to carcinogenic assessments, but EPA’s proposed cancer guidelines of 1996 specify the use of a particular model.
b.	"...have the EPA software identify the slope of the line associated with the lower bound dose to the point of origin, in addition to the slope estimated form the fitted curve."  (2/5/99)	EPA is in the process of developing a preferred cancer model for use with the new guidelines, and it will contain slope information useful for adherence to the new guidance.  Current plans are to make it available as a beta-test model when it is distributed for external review later in 2000.
10.	Model Output Comments	Model Output Responses
a.	"Having the input data echoed in the output would be nice"  (1/1/99)	The latest BMDS models (the models distributed with BMDS version 1.2) have been revised so the output page contains the information necessary for a user to repeat (verify) a model run and its results.
b.	In the multistage and Hill models, plots were only produced for the quadratic form of the multistage model. (3/8/99, 7/16/99)	The latest multistage and Hill models has been revised to correct this bug in the previous plotting programs.
c.	"I would much prefer to have the results at the beginning of the output, as the highly technical nature of the intervening output would be somewhat disconcerting to most of the users."  (3/29/99) The header of each report should include the "analysis performed (e.g. probit, Weibull, etc.),"  "parameters chosen by the user - log vs normal, background rates, etc." and "[BMR] value the user selected - e.g. the 1% response, 50% response, etc."  (7/20/99)	Information relating to model selected and options chosen appears at the top of text output, followed by goodness-of-fit information, then the BMD and BMDL values.  In developing BMDS interface, models and output, the Agency has tried to walk a fine line between making the program user friendly enough for toxicologists keen on maximizing their time, but not so user friendly as to mask attributes of the models and results that are important for them to understand before proceeding to use the results for establishing Agency policy.  In this case, it was deemed that users, particularly disconcerted ones, should be required to read through the "intervening output" (model specifications and goodness-of-fit information)  because of the important contributions it should be making to their overall decision making process.
d.	"When the dichotomous set example is run, the plot shows several BMDL levels in a connected curve, but there seems to be no way to select such an option from the program.  How was this done?  Would it be possible to select several BMRs at once?"  (3/31/99)	When the "BMDL Curve Calc." option is selected (Default) from the Model Run Screen, the graph resulting from the model run will display a blue BMDL curve.  The BMDL curve is estimated by calculating the BMDL for BMDs at BMRs of 1, 5, 10, 20 and 30%, and connecting these individual points via either a straight (Unique) or curved (Cspline) line depending on the Smooth Option chosen.  At the present time there is no way for the user to specify more than one BMR at a time, but this is something that a future version of BMDS will likely include.
11.	Hardware Compatibility Comments	Hardware Compatibility Responses
a.	BMDS does not seem to run on computers which utilize a CYRIX 6X86 processor chip architecture. (2/10/99)	We were not able to resolve this problem directly.  It is possible that certain computer configurations, particularly those involving the use of a CYRIX processor chip, will not be compatible with BMDS version 1.2.
b.	"Is it possible to make BMDS operational under Windows 3.1?"  (5/18/99)	BMDS is not backwards compatible to Windows 3.1 and to our knowledge there is no way to patch Win 3.1 so that BMDS will run properly under that operating system.
c.	"Is this newer version Windows NT compatible?"  (12/13/99)	Yes, version 1.2 of BMDS is Windows 95/98/NT compatible.
12.	Help System/Guidance Comments	Help System/Guidance Responses
a.	"In the explanation of the continuous model the description of ALPHA has the wrong symbols in the help description."  (3/8/99)	This has been corrected.
b.	"In the help file for calculation of the BMD for continuous data it should note the BMD is calculated based on the predicted, not the observed control mean."  (3/8/99)	The BMR, which forms the basis for the BMD, is now more explicitly defined as being based on the predicted control mean.
c.	"Documentation is limited... Perhaps a glossary of terms would help."  (3/29/99)	The help system contains an index of terms and a "find" function that allows for searching on a more expanded word list.  The reviewer may have been referring, at least in part, to the need for guidance in the choice of models and the application of BMD methods.  At this time the Agency's "Benchmark Dose Technical Guidance Document" is undergoing internal EPA review, but should be available sometime in 2000.
d.	"It would be very useful to have a manual in pdf format.  This should include actual examples of data analysis - a picture is worth a thousand words."  (7/20/99)	The help manual is provided in both Word Perfect and PDF formats.
13.	General Comments	General Responses
a.	"When I opened About BMDS and tried to find a way to close it, because I couldn't see it had an OK button, I tried closing it without clicking OK first, and got a flickering display, alternating with a "do cancel" button."	If you can not see all of the BMDS features, you may need to adjust the resolution of your monitor.  You should be able to do this by going to the Control Panel in Windows and clicking on the monitor icon.
b.	Because the BMDS already has widespread use, "future beta releases should have a built-in drop-dead date after which they will no longer work."  (3/8/99)	The concern over use of old versions of BMDS is going to be relevant whether BMDS is released as final or beta software because models within BMDS will always be in different stages of development.   A "drop-dead date"  is not felt to be necessary, however, because all model output is accompanied by the model version number.  EPA will verify all model results against the latest version of the model.
c.	"The framework of the program is excellent, but the current output has too many wrong answers...the next version should not be released without another full round of beta review similar to the first two reviews."  (3/8/99)	Since the public review of beta version 1.1b of BMDS, the BMDS system has been subjected to extensive internal testing and an independent QA assessment.  Subsequently, it was felt that most of the BMDS models were ready to be released as final, not beta, models.  Thus, the current overall BMDS system (version 1.2) is not considered beta.  However, certain models that either did not receive (Hybrid and Cancer models) or did not pass QA testing are being released within version 1.2 as beta models.  They are explicitly designated as such both up front (before they are run) and in their text output.  These models will undergo another round of beta review at a later date, as suggested.