1. INTRODUCTION

California’s Fourth Climate Change Assessment includes nine regional reports, two of which, the San Diego and Sierra Nevada Region Reports, involved members of the California Nevada Climate Applications Program (CNAP) as coordinating lead authors. The regional reports are the first to have been included in a California Statewide Assessment and were published online in 2018 (Thorne et al. 2018). Their purpose is to provide local and regional governments with climate change and impact information to support adaptation planning and implementation specifically at those scales (Thorne et al. 2018). In 2020, a CNAP researcher who was not involved in the development of the regional reports conducted an evaluation of the San Diego Region Report (Kalansky et al. 2018) and the Sierra Nevada Region Report (Dettinger et al. 2018) to generate insight into their use and effectiveness in meeting that purpose. This report presents a summary of the evaluation findings.

The evaluation is based on semi-structured interviews conducted with 12 regional report stakeholders (seven in San Diego and five in the Sierra Nevada) and written responses collected from an additional stakeholder of the Sierra Nevada Region Report, all between July and October 2020. Each stakeholder had served as an advisor to or reviewer of the report corresponding to their region or was an internal colleague of those advisors/reviewers. These stakeholder advisors/reviewers were identified for participation in the evaluation by the coordinating lead authors of the reports; internal colleagues were then identified through chain referral. At the time of evaluation, stakeholders were employed by or affiliated with the following organizations: San Diego Gas & Electric, Port of San Diego, San Diego Association of Governments, San Diego Water Authority, City of San Diego Public Utilities Department, City of San Diego Planning Department, Sierra Nevada Aquatic Research Laboratory, California Tahoe Conservancy, Tahoe Regional Planning Agency, Sierra Business Council, and Sierra Forest Legacy.

The interviews were conducted virtually and were audio recorded. Written summaries of responses to individual questions were created from the interview recordings and detailed notes. The summaries were then analyzed to identify patterned themes (Bernard 2017). Those themes include: (1) credibility and trust, (2) participation, (3) audience, (4) actionability and use, and (5) additional information needs. Following is a discussion of those themes and a list of accompanying recommendations that stakeholders offered for the development of future regional reports. However, when interpreting those themes and recommendations, it is important to understand the context in which the regional reports were developed. The regional report teams worked voluntarily and were organized relatively late in the course of the Assessment workstream. With less than one full year for the development of each report, it was difficult for the teams to entrain broad stakeholder participation, including from city, county, and state agencies, tribes, and in the case of San Diego, stakeholder groups in Baja California. The especially large geographic scope of the Sierra Nevada Region Report further added to this difficulty. Nonetheless, both teams initiated report development with a “discussion session” involving community and sector representatives, some of whom then additionally served in the roles of stakeholder advisors/reviewers as described above.

There are two limitations to this evaluation that should be noted. The first is the lapse in time between development, publication (2018), and evaluation (2020) of the reports, resulting in reduced clarity of recall for stakeholders about their service as advisors or reviewers and about their use of the reports over time. The second is the small, purposive sample for which the feedback and recommendations presented here should not be considered representative of all stakeholder advisors, reviewers, and other potential end users.

2. SUMMARY OF FINDINGS

2.1 Credibility and Trust

Stakeholders overwhelming expressed appreciation for and satisfaction with both reports, emphasizing the value in having a single, “easy-to-read,” “definitive” document that summarizes the “best available science” on climate change and impacts to help inform their work. Stakeholders additionally noted the importance of local academic authorship in fostering confidence in decisionmakers that the science communicated is “nonpartisan” and “trustworthy.” Inasmuch as the regional reports are perceived to reflect those characteristics, they have been helpful to stakeholders in gaining decisionmaker support for and motivating progress toward adaptation planning and implementation. Similarly, stakeholders attributed their own expressed high levels of trust in the reports to the reputations of the coordinating lead authors and their intuitions. The stakeholders were not familiar with CNAP.

2.2 Participation

As indicated, the stakeholders were either advisors to or reviewers of the reports, or internal colleagues of those advisors/reviewers. For those stakeholders who had served as advisors/reviewers, their participation consisted primarily of periodic review of and comment on drafts of the reports, a process most described as satisfactory but also somewhat rushed and ad hoc. Were stakeholders to participate in the development of future regional reports, their desired level of involvement would vary according to available time and interest. Some would prefer only to perform a one-time review whereas others would prefer to inform the content and direction of the reports (in addition to providing review) to ensure a more stakeholder-driven output that aligns directly with local and regional adaptation planning and implementation. Regardless of preferred level of involvement, stakeholders expressed the desire for earliest possible engagement in the process of report development and for clear expectations and timeline regarding their participation.

2.3 Audience

There was consensus among stakeholders that the regional reports are appropriate for and relevant to a more technical, practitioner audience like the one they comprise, but that the intended audience(s) of the reports is not clear. Several stakeholders added that there may be opportunities missed in producing only limited outputs from the Assessment in the form of comprehensive (and therefore lengthy) reports. For example, they noted that decisionmakers are not likely to read them: “Decisionmakers, they don’t pick up these reports. They need a one- or two-page sheet that you can look at and say, ‘Here’s the biggest implications, here’s the biggest next steps, here’s what you need to be focusing on.’” Stakeholders therefore encouraged diversification of outputs related to future regional reports, however, not just for decisionmakers but also for community-based organizations, the public, and other potential stakeholders. As one stakeholder explained:

More and more we’re going to start to see climate data needed in more and more circles. […] Don’t be afraid to go beyond just the climate and environmental sphere for engagement. I think that there are a lot of groups that are going to pay more and more attention to climate data because it’s going to inform, hopefully, some more foresight and forethought and planning and action in what I think is going to be an increasingly disrupted world.

In addition to fostering decisionmaker, public, and other potential support for local and regional adaptation planning and implementation, stakeholders noted that efforts to diversify outputs will also help foster important and much needed information equity.

2.4 Actionability and Use

Several stakeholders commented that the intended (or potential) uses of the reports are not clear. Nonetheless, all stakeholders indicated that they personally had used the reports. They provided examples specifically of conceptual use (i.e., the use of information to enhance knowledge base or to inform a process or plan) and justification use (i.e., the use of information to justify implementation of a process or plan, or to substantiate a decision already made) (for more description of types of climate information use, see VanderMolen et al. 2020 and Wall et al. 2017). Examples of conceptual use included individual stakeholders utilizing the reports to learn about climate change and impacts within their region and to inform (in San Diego County) the San Diego Integrated Regional Water Management Plan, City of San Diego Sea Level Rise Vulnerability Assessment, and San Diego Regional Plan, and (in the Sierra Nevada Region) the Inyo-Mono Regional Water Management Plan, California Tahoe Conservancy Climate Adaptation Plan, Mariposa County Recreation Master Plan, and Sierra Business Council Regional Vulnerability Assessment. Specific examples of justification use were more difficult for stakeholders to cite as they frequently reference the reports in passing interactions with decisionmakers when communicating the importance of climate change and impacts, particularly in advocating for the adoption of adaptation policies.

Stakeholders agreed that the reports are sufficiently actionable for conceptual and justification uses within their organizations and that to attempt to foster instrumental use (i.e., the use of information to directly influence or prompt a new decision or action) would likely require an unrealistic level of granularity in the science presented. Accordingly, there was consensus among stakeholders that the regional reports should remain high-level summaries of the best available science on climate change and impacts that can serve to motivate more local-level adaptation planning and implementation. Several stakeholders nonetheless suggested that it would be appropriate and helpful to direct readers to publicly available datasets, tools, and other resources to assist with more local-level efforts, in addition to providing more explicit guidance for how the reports themselves might be used. For example, they strongly encouraged the inclusion of case studies demonstrating how different organizations have used the reports: “I think the combination of having a solid reference document that everyone can key to, together with lots of examples of how that information is actionable rather than having the report be the driver, is probably my summary.” They also specifically recommended the inclusion of guidance for and case studies demonstrating the navigation of uncertainty to help organizations overcome its “paralyzing” effects to adaptation planning and implementation.

Some stakeholders additionally encouraged a more holistic presentation of climate change and impacts in a way that “tells a story” and helps demonstrate how different impacts are interrelated. As one stakeholder explained:

For example, during extreme heat events people are going to use their air conditioning more which means that the system is going to have more stress. But meanwhile, if we have a Santa Ana or high wind advisory, we may be shutting power down in certain parts of the region and what does that do for people who have health risks that may be exacerbated by heat. Just tying all those pieces together. People are going to extract the pieces that are most of interest or relevant to them, but it’s one of the things we’ve had to really work on […], explaining that whole system approach and the holistic nature with which we need to be planning, and how it really all does tie together.

They added that should the outputs be diversified (see Section 2.3 Audience, on page 4), such “story telling” may become even more important for its perceived effectiveness in fostering decisionmaker and public awareness and education, in addition to compelling the former group to action.

In comparing the regional and statewide reports, several stakeholders indicated that they use both but emphasized that the relative specificity of the former is more immediately useful to their work and in some cases has been critical to gaining decisionmaker buy-in. As one stakeholder explained, “It’s also really helpful when we’re talking to stakeholders to have something that’s San Diego specific because people will pretty quickly say, ‘Oh, that’s not for San Diego, that’s for the entire coastline, does that apply here as well?’ So it’s really nice for the city to have a scientific research paper that can back up everything that we’re saying and can back up the policies that we’re drafting as we move forward with the adaptation plan.” Stakeholders therefore advocated strongly for the continued inclusion of regional reports in subsequent statewide climate assessments for their value in helping stakeholders keep up to date on the state of the science and in motivating policy action.

2.5 Additional Information Needs

Stakeholders made some requests for additional climate science information, both in general and with respect to the regional reports. Examples of those requests included, in San Diego, improved predictive capacity and lengthened lead times for precipitation and drought forecasting (e.g., to inform reservoir operations), and in the Sierra Nevada, better understanding of the impacts of changing precipitation on winter flooding and lake levels. Several stakeholders also requested more information on climate impacts to local and regional populations with attention specifically to public health and social justice, topics that were addressed in some detail in the San Diego Region Report. Stakeholders explained that these topics are moving to the forefront of adaptation planning and suggested that all future regional reports should include at least a high-level summary of them, for example: “It would be quite the heavy lift to look at every factor but maybe there’s discussion of precipitation or heat or sea level rise, then there’s also discussion of what would make a community more vulnerable to these or where are there considerations that should be made for more equitable planning.” Relatedly, several stakeholders expressed that should there be future regional reports, they would very much like the content to evolve in close step with changing stakeholder needs, hence the desire for a more stakeholder-driven approach to and earliest possible engagement in the process of report development (see Section 2.2 Participation, on page 3).

3. RECOMMENDATIONS

Following is a summary of the recommendations stakeholders offered with respect to each of the topics presented above, for consideration in the development of future regional reports.

3.1 Participation

• Engage stakeholders earlier in the development of the reports and consider offering them different opportunities for participation, including to help:

— Inform the questions asked and directions taken;
— Indicate the types of information and analyses that would best enable/support local and regional adaptation planning and implementation;
— Review and provide feedback on drafts of the reports.

• Recruit representative but more diverse subject matter experts to assist with review of the reports to ensure that:

— Each section is relevant to stakeholders;
— Overly broad-brush generalizations are avoided;
— The content is generally accessible to all target audiences (see “Audience,” page 8).

• Involve researchers and/or stakeholders with detailed knowledge of how climate information is used within organizations to help inform the content and layout of the reports.

• Include stakeholders from Baja California in the development of San Diego Region Reports to support cross-border adaptation planning and implementation.

• Allow a 4-5 week minimum for stakeholders to review and comment on drafts of the reports to accommodate internal organizational permissions and timelines.

3.2 Audience

• Determine a clear audience(s) for the report, and in doing so:

— Consider a broader and more diverse audience(s), including tribes, community-based organizations, chambers of commerce/other commercial groups, fire safe councils, among others.

• Diversify Assessment outputs according to the needs of the identified audiences(s), for example:

— For a technical audience: the current report format is considered appropriate, however, it would also be beneficial to have supplemental materials, e.g., a bulleted list of key talking points and a package of the figures/graphics with permissions for their reuse.
— For policymakers: a 1-2 pager communicating the “biggest implications” and “biggest next steps.”
— For all audiences, including the public: story maps and/or other online resources that “repackage the report into easily digestible bits instead of relying on one document.”

• Diversify outreach according to audience types.

3.3 Actionability and Use

• Provide examples throughout each report that demonstrate how different organizations have utilized climate change and impacts information from previous region reports.

• Discuss uncertainty where it exists and provide recommended steps for working through it and/or include case studies that demonstrate how some organizations have managed it in their decision making.

• Present climate change and impacts information more holistically (i.e., “tell a story”) to help stakeholders understand and communicate the interrelatedness of impacts.

• Make presentation of published reports to stakeholder groups a regular component of their release.

3.4 Additional Information Needs

• Include information on climate impacts to local and regional populations with emphases on public health and social justice in all regional reports, even if only in high-level summaries.

3.5 Other

• Move toward a more equitable and collaborative model for report development that makes compensation available to stakeholder organizations that cannot donate their time to such efforts.

• Explore the pros and cons of standardizing the regional reports for organizational clarity and ease of use.

• Explore whether the geographic scope of each regional report aligns with stakeholder needs and whether (and how) it influences actionability.

• Allow more time for stakeholder engagement so that the reports may be developed in closer collaboration with stakeholders and in accordance with the above and other potential recommendations.

4. CONCLUSION

The purpose of this evaluation has been to generate insight into the use and effectiveness of California’s Fourth Climate Change Assessment San Diego and Sierra Nevada Region Reports in supporting local and regional adaptation planning and implementation. All 13 stakeholders who participated in the evaluation expressed satisfaction with the reports and provided several concrete examples of use in adaptation planning and implementation. The use types were primarily conceptual but also included justification. No stakeholder reported instrumental use of the reports for lack of required granularity, noting that it is not the role or purpose of the reports to provide it. Stakeholders offered specific feedback about credibility and trust, participation, audience, actionability and use, and additional information needs. They also offered recommendations corresponding to those topics for consideration in the development of future regional reports. Many of those recommendations may fall outside the purview and decision making of individual authors (and/or CNAP), and to consider them adequately may require examination of larger questions about the scope, accessibility, and ultimately the purpose of regionally focused climate change assessments. Nonetheless, the feedback and recommendations that stakeholders have offered lend insight into their perspectives, preferences, and priorities with respect to the production of regional climate change and impacts information intended for their use.

5. REFERENCES

Bernard, H.R. (2017). Research Methods in Anthropology: Qualitative and Quantitative Approaches. Lanham, Maryland: AltaMira Press.

Dettinger, M., Alpert, H., Battles, J., Kusel, J., Safford, H., Fougeres, D., Knight, C., Miller, L., & Sawyer, S. (2018). Sierra Nevada Summary Report. California’s Fourth Climate Change Assessment. Publication number: SUM-CCCA4-2018-004.

Kalansky, J., Cayan, D., Barba, K., Walsh, L., Brouwer, K., & Boudreau, D. (University of California, San Diego) (2018). San Diego Summary Report. California’s Fourth Climate Change Assessment. Publication number: SUM-CCCA4-2018-009.

Thorne, J.H., Wraithwall, J., & Franco, G. (2018). California’s Changing Climate 2018. California’s Fourth Climate Change Assessment, California Natural Resources Agency.

VanderMolen, K., Meadow, A. M., Horangic, A., & Wall, T. U. (2020). Typologizing Stakeholder Information Use to Better Understand the Impacts of Collaborative Climate Science. Environmental Management, 65(2), 178-189.

Wall, T. U., Meadow, A. M., & Horganic, A. (2017). Developing evaluation indicators to improve the process of coproducing usable climate science. Weather, Climate, and Society, 9(1), 95-107.