Accelerating earthquake predictability research: prospective, open-science and community-driven evaluations of earthquake forecast models

William Savran, Max Werner, Thomas Jordan, David Jackson, David Rhoades, Philip Maechling and the working group for the Collaboratory for the Study of Earthquake Predictability

Predicting the location, size, and origin times of future earthquakes is arguably the most fundamental problem in earthquake science. So much so, that nearly all research efforts are applicable to the topic of predictability in some form. Insights into understanding earthquake processes span numerous disciplines ranging from theoretical work on source physics to field studies mapping damage zones and historic ruptures to observational and statistical seismology. Compiling these multidisciplinary results into testable forecasting models will be a key effort in solving this problem. Future earthquake centers must emphasize a two-pronged approach that integrates both physics and observations to develop and evaluate testable earthquake forecasting and prediction models.

Within the Southern California Earthquake Center (SCEC), an example of this multi-disciplinary approach to build a comprehensive forecasting model is the Uniform California Earthquake Rupture Forecast (UCERF). The third iteration of this model (UCERF3) represents the state-of-the-art in addressing the medium-term forecasting gap as it combines long-term elastic rebound statistics with short-term earthquake clustering in the form of epidemic-type aftershock sequences (Field et al., 2017). This model provides probabilities of future earthquakes occurring – an important step for developing probabilistic hazard models that influence building codes and insurance markets. This multi-institutional and multi-disciplinary activity is an example of the type of product that a successful earthquake science center should be supporting. Future earthquake science centers should enable research and deliver products that must be developed through large-scale collaborative and focused efforts. The UCERF project successfully integrated governmental, academic, and private partners representing multiple disciplines – a true example of a system-level model. Nevertheless, the UCERF model, like all other models proposed for practical forecasting, needs to be subjected to formal testing to demonstrate its dependability and relative worth compared to other models based on different approaches.

Claiming success on the earthquake prediction problem requires a systematic understanding of earthquake physics to avoid localized predictions that do not generalize to other regions and time periods. A clear path toward providing confidence in predictions requires formal experiments that exceed the scientific rigor required to publish the ideas. Ideally, these experiments should be conducted by a neutral third-party. This requires two main steps (1) formulating ideas into testable models and (2) evaluating them through formal prospective experiments. A prospective experiment requires that all components of a model must be specified a priori to zero degrees of freedom. Additionally, the data used to evaluate the models must be agreed upon and be obtained from independent, authoritative sources, e.g., USGS ComCat. Additionally, the metrics used to evaluate the models must be agreed upon before the start of the experiment. A prospective experiment eliminates any potential conscious or unconscious biases introduced during model development and self-evaluation. The parameters for an experiment should be developed through a community effort to build consensus of the experiment guidelines (Schorlemmer et al., 2018).

Since its inception following the Regional Earthquake Likelihood Model (RELM) experiment, the Collaboratory for the Study of Earthquake Predictability (CSEP) has conducted formal earthquake forecasting experiments in various natural laboratories around the world – most notably in California, Italy, Japan, and New Zealand (Jordan 2006, Jordan et al., 2011). Supporting multi-region and global scale experiments is critical for engaging and supporting a global community of earthquake scientists as well as for collecting sufficient data that allow meaningful testing of hypotheses about the large earthquakes we care most about. CSEP provides the cyber-infrastructure to perform autonomous prospective earthquake forecasting experiments and is currently expanding its capabilities to accommodate more general types of experiments. The cyber-infrastructure provided by CSEP contains the hardware required to run the computations and the software developer time to build the software for the Testing Centers active in the above-mentioned countries. Recent software-development efforts at CSEP involve modularizing the current testing center codebase into an open-source Python package for greater community access. These efforts will assist model developers by providing communityendorsed tools to evaluate forecasts and predictions and thereby facilitate inter-model comparisons and improvements. The scientific achievements of the recent CSEP activities were documented in the July 2018 Focus Section of Seismological Research Letters (Michael and Werner 2018). Given its experience organizing and conducting formal prospective experiments, CSEP is perfectly positioned to continue leading this effort.

Evaluating earthquake forecasts remains an important task due to the societal relevance, such as their use in developing probabilistic hazard models. As new and interesting ideas are emerging about the predictability of earthquakes we must support experiments that can address these ideas. Here, we propose several Grand Challenge ideas that should be addressed within a future earthquake science center to create a step-change in earthquake predictability. We envision having prospective tests underway preceded by retrospective tests and annual progress reports. Progress requires building hypothesis testing into forecasting experiments and describing portable hypotheses in simple testable terms. Null hypotheses should also be developed. We propose the following Grand Challenge questions and ideas that should be readily addressed within the future earthquake centers:

1. Does elasto-static Coulomb stress from past earthquakes provide predictive information, beyond that from presently accepted clustering, for future earthquakes? It seems obvious to many, but past efforts at testing have suffered from complexities not fully integrated into the
   models. Can such complications, like the orientations of target faults and the initial stress state be included in models, or are there testable features less dependent on those complications? Can physics-based models consistently outperform statistical models such as the ETAS model to create more informative operational earthquake forecasts?
   
2. Are spatial and temporal b-value variations indicative of future large earthquake probabilities? Again, obvious to some, dubious to others. Past efforts have been confounded by the need to define spatial and temporal observation and test areas, and by other effects not included in the forecast models. 
   
3. Can large earthquakes be forecast well from observations of smaller ones? Are large and small events simply parts of a single process with common statistical properties like the GutenbergRichter magnitude distribution and Omori clustering, or do crust-mantle structure and fault properties cause separate behavior for large and small quakes?
   
4. Are there spatial and temporal behaviors of small to moderate earthquakes that signal the imminence of larger ones? Do small events gradually concentrate around future epicenters or faults on which big earthquakes will occur? Can such behaviors be defined well enough to assign probability distributions or to count false alarms as well as successful forecasts?
   
5. Do proposed precursory anomalies in electromagnetic, ionospheric, geochemical or other variables signal the imminence of large earthquakes? What is the physical link between such proposed precursors and the earthquake generation process? Can identified precursors be used individually or collectively to systematically improve earthquake forecasting?

These Grand Challenge ideas address our fundamental need for a better understanding of the earthquake process and their answers would significantly contribute to our development of useful earthquake predictability tools. These ideas have the same fundamental theme, that is: which observations or theory contain information about future seismicity, and of these observations, which can be applied toward prediction? Meaningful answers will require sustained support for multidisciplinary forecast model development and a community-driven, open-science approach for rigorous model comparison and evaluation. A future earthquake science center should support formal prospective experiments to address these Grand Challenge ideas: the results may fundamentally transform operational earthquake forecasting and seismic hazard prediction. 

References

Field, E. H., T. H. Jordan, M. T. Page, K. R. Milner, B. E. Shaw, T. E. Dawson, G. P. Biasi, T. Parsons, J. L. Hardebeck, A. J. Michael, R. J. Weldon, P. M. Powers, K. M. Johnson, Y. H. Zeng, K. R. Felzer, N. van der Elst, C. Madden, R. Arrowsmith, M. J. Werner, and W. R. Thatcher (2017). A Synoptic View of the Third Uniform California Earthquake Rupture Forecast (UCERF3), Seismological Research Letters 88 1259- 1267.

Jordan, T. H. (2006). Earthquake predictability, brick by brick, pubs.geoscienceworld.org 77 3-6.

Jordan, T. H., Y. T. Chen, P. Gasparini, R. Madariaga, I. Main, W. Marzocchi, and G. Papadopoulos (2011). Operational earthquake forecasting. State of knowledge and guidelines for utilization, Annals of Geophysics.

Michael, A. J., and M. J. Werner (2018). Preface to the Focus Section on the Collaboratory for the Study of Earthquake Predictability (CSEP): New Results and Future Directions, Seismological Research Letters 89 1226-1228.

Schorlemmer, D., M. J. Werner, W. Marzocchi, T. H. Jordan, Y. Ogata, D. D. Jackson, S. Mak, D. A. Rhoades, M. C. Gerstenberger, N. Hirata, M. Liukis, P. J. Maechling, A. Strader, M. Taroni, S. Wiemer, J. D. Zechar, and J. C. Zhuang (2018). The Collaboratory for the Study of Earthquake Predictability: Achievements and Priorities, Seismological Research Letters 89 1305-1313.