The following comment letter written by Mary C. Christman, a statistician and private consultant was sent to the US EPA from the Florida Clean Water Network.  The letter speaks for itself and is now a part of the administrative record for Florida’s proposed Human Health Toxics Rule.  Here is Ms. Christman’s letter:

James Giattina, Director                                                                                                            U.S. Environmental Protection Agency – Region 4                                                                       61 Forsyth Street, S.W.                                                                                                              Atlanta, GA 30303-8960

10 October 2016 Dear Mr. Giattina:

Let me introduce myself. I am a biostatistician working on environmental issues. My background includes statistical support for environmental risk assessments, modeling of ecological and environmental phenomena, design and analyses of clinical trials, and design of analyses of sampling strategies to optimize information gathering for a variety of biological and environmental projects. I am also a current member of the USEPA Food Quality Protection Act (FQPA) Science Review Board (SRB). Since 2011, I have been principal in MCC Statistical Consulting LLC and for 11 years prior to that I was an Associate Professor of Statistics at University of Maryland and University of Florida.

This letter is prepared at the request of the Florida Clean Water Network in support of their comments on the human health-based surface water toxic criteria.

I have reviewed for statistical appropriateness, adequacy and completeness FDEP’s Technical Support Document (Technical Support Document: Derivation of Human Health-Based Criteria and Risk Impact Statement, June 2016, website: http://www.dep.state.fl.us/water/wqssp/health.htm), which describes the methodologies used to develop the updates to the human health water quality criteria in Rule 62-302.530, Florida Administrative Code (F.A.C.). This letter is a summary of my conclusions concerning the appropriateness and adequacy of the methods used to calculate updated criteria and whether the proposed criteria should be implemented.

The level of detail provided in FDEP’s Technical Support Document is insufficient to allow those with statistical expertise to adequately review, or certainly to approve, whether the proposed approach is better than the deterministic methodology currently used by USEPA. In fact, a request for more detail on FDEP’s statistical methodology was denied (see attached email correspondence with the Senior Assistant General Counsel for FDEP) with the explanation that no discussion of the revision is possible due to “pending litigation”. As a result, the proposed method cannot be adequately reviewed. The USEPA should prevent its implementation until such information is available and additional analyses are performed to verify that this approach is in fact a robust and defensible method for setting water quality criteria for human health.

Some examples of missing detail include:

  1. 1)  the calculations related to determination of risk under the proposed criteria, namely the exact methods by which hazard quotients for non-carcinogens and the risks for carcinogens were calculated;
  2. 2)  how the probability distributions for the exposure factors were identified as the “best” models of the distributions;
  3. 3)  whether any effort was made to determine that the exposure factors, such as body weight and drinking water intake, are completely independent among individuals; and,
  4. 4)  whether sensitivity analyses for adequacy of the proposed models were performed.

Reproducibility of research and analyses requires that information concerning the proposed methodologies be sufficient to allow another researcher to reproduce the same results. The explanations given in FDEP’s Technical Support Document are not sufficient to do that, especially for the hazard and risk assessments.

FDEP described the data used for developing probability distributions for the exposure factors, and reported the probability distributions that they fitted to said data. However, FDEP failed to provide any detail about the method for fitting the distributions nor did they provide any information on how they determined that, among possible alternative distributions, the one they reported was in fact the best model of the data. No measures of fit (e.g. mean squared error or Akaike Information Criterion) were provided, nor was a determination of whether the random inputs (fish consumption rates, body weight, drinking water intake) were or were not related to one another.

Since sensitivity analyses to determine effects of the model inputs and assumptions on outcomes were not described, one assumes that they were not performed. Instead, FDEP chose to report whether the outputs of the method were correlated with the random inputs, which provides little information about whether the probabilistic method is sensitive to the distributions of the inputs or whether the outcomes are robust to failure of the assumptions used (input probability distributions, independence of the exposure factors, etc.) or sensitive to the choices for the inputs to the equations used. A simple example of a check that could determine whether the conclusions (criterion value) would be the same under the full model (all exposure factors random) versus under a reduced model where one or more of exposure factors are held to a constant (such as the mean value). Neither sensitivity, other than the correlations, nor robustness was described in the Technical Support Document.

Although detail is lacking in the document, the detail that was provided indicates some potential problems with the approach. An obvious example of a problematic choice is the fit of a probability distribution to drinking water data provided in Table 3-23 of the USEPA (2011) Exposure Factors Handbook (EPA/600/R-090/052F). FDEP fit a probability distribution to only 8 values (percentiles) which resulted in a distribution with more than 10% probability that intake would be less than 0 mL/day. FDEP chose to modify the fitted distribution by constraining “the distribution to a minimum of 0 mL/day [which] resulted in an input distribution with a 10th percentile of 0 mL/day…” Converting negative values to 0s changes the distribution and its moments (mean, standard deviation, percentiles, etc) and likely leads to biased water quality criteria.

Another example of inappropriate introduction of variability into the probabilistic model is the method chosen to incorporate variability in fish consumption rates for subsistence fishers. FDEP chose to fix the total intake of a subsistence fisher at 142.4 g/day and introduced viability among individuals not by varying the total intake but by randomly allocating the total intake among the three trophic levels used in the probabilistic method. This has the result that unlike all other “average” Floridians, whose total fish consumption was assumed to vary among individuals and be equally distributed among the three trophic levels, all subsistence fishers were constrained to consume the “mean” intake of 142.4 g/day and varied only in what trophic level they obtain their fish. This likely leads to biased estimates of the proposed criteria and could result in greater risk and greater risk to subsistence fishers. Unfortunately, the effect cannot be assessed without additional information and until the other issues concerning the probabilistic method have been addressed.

A comparison of FDEP’s proposed criteria to two alternative methods was performed in order to assess whether the proposed criteria are reasonable given that I could not determine that from the technical information provided. For those chemicals that are currently regulated by the state I first compared the existing criteria (set in 1992) to the proposed criteria. FDEP argues that their proposed criteria are more protective than the existing criteria because the proposed values are less than the existing values for a substantial fraction of the 40 (Class III) or 43 (Class I) chemicals currently regulated by the state. On the surface this appears to be the case: for Class I, 20 proposed criteria are lower than the existing criteria, and for Class III, 26 are lower than the existing values. I then compared the proposed criteria to criteria values calculated using the deterministic method with USEPA values for all fixed inputs except for fish consumption rates. These were given fixed values at the 90th percentile of FDEP’s new calculations consumption rates for three tropic levels and three geographic regions. Among the 20 chemicals with proposed Class I criteria lower than the existing criteria, only two of them are also lower than the values obtained from the alternative deterministic method.  Among the 26 chemicals with proposed Class III criteria lower than the existing criteria, only one is also lower than the criterion calculated under the deterministic method.  These results suggest that either the FDEP proposed values are less protective than the values estimated under the deterministic method or that the probabilistic method is not being correctly implemented.

Lack of appropriate and complete vetting of the probabilistic approach used by FDEP indicates a need for the more precautionary approach of the EPA  deterministic method.  The probabilistic approach as implemented by FDEP was not peer-reviewed by bio-statisticians, but by toxicologists who have neither the statistical expertise nor experience in using the method to adequately evaluate the modeling and its underlying assumptions for probability distributions for the exposure factors.  As a result, it is difficult to determine if the proposed criteria are adequately protective for the majority of Floridians.

This probabilistic approach has not yet been used successfully by any other state in setting human health criteria and I recommend that Florida not be the first to be allowed to do so given the current information available for the approach used by FDEP.

Thank you for your consideration of these comments.

Mary C. Christman                                                                                                                    MCC Statistical Consulting                                                                                                  Gainesville, FL 32605


About the Author

Linda Young has been the executive director of the Clean Water Network of Florida since 1994. From 1989 to 1997, she founded and published a monthly statewide environmental newspaper. Over the past twenty three years, she has co-founded some of the most long-lasting and effective environmental organizations in the Southeast, including the Gulf Restoration Network, Gulf Coast Environmental Defense and C.A.T.E. She holds a B.A. in Communications from Southern Oregon University and a M.A. in Political Science/Campaign Management from the University of West Florida.



Subscribe to Blog via Email

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

Join 116 other subscribers


Follow me on Twitter