1.1 Multidimensionality, problem orientations, and norms

In the field of CPR and SES research, synthesis studies play a unique role in attempting to bring together empirical data across existing studies to identify common variables associated with successful governance. They attempt to capture patterns in the relationships between social contexts (e.g., property regime, configuration of rules, or household indicators) and biophysical characteristics (e.g., species richness, forest condition, fish abundance), often using outcome variables to interpret the success of particular governance arrangements or institutions. However, individual studies considered within synthesis studies do not necessarily measure the same independent and dependent variables and concepts (Frey 2017). Thus, the conclusions they draw are likely to vary depending on the definition of success.

More generally, synthesis assessments of success must inevitably reduce complex systems to a few measurable or aggregate indicators of either ecological or social performance such as species richness, tree condition, catch per unit effort, income or rule compliance. Therefore, despite clear strengths, these forms of analysis might miss important dynamics, processes and conflicts along the way. For example, the majority of the conservation literature does not address feedbacks between social and environmental variables (Miller et al. 2012). Similarly, a recent systematic review found that the community forestry literature did not adequately address the links between population dynamics, market forces, biophysical characteristics and environmental and livelihood outcomes (Hajjar et al. 2016). This challenge reflects long-standing debates in the social sciences regarding the relative merits of quantitative, qualitative, inductive, and deductive methods for understanding social dynamics and outcomes (Poteete et al. 2010). Poteete et al. (2010) argue for a methodologically pluralist approach to examining CPR and SES challenges to promote diverse theoretical approaches and to alleviate methodological debates. This pluralism can be enhanced through insights from grounded theory, which emphasizes generating theory inductively, and systematically and iteratively refining theory and concepts in relation to empirical data (Glaser and Strauss 2006). We generally support this pluralist approach by demonstrating the limitations in many synthesis studies in defining the construct of success, and by reviewing recent qualitative approaches that can enhance our ability to examine success as multidimensional and open to diverse problems and power dynamics.

Successful CPR governance can be attributed to original exemplars, such as Hardin’s (1968) “tragedy of freedom in the commons,” or Ostrom’s (1990 p. 27) response, positing the potential for “individuals to use and govern CPRs”. Specifically, Ostrom (1990, 29) defined successful CPR institutions as those “that enable individuals to achieve productive outcomes in situations where temptations to free-ride and shirk are ever present.” Scholars have expanded on discussions of successful governance, interrogating what success might mean by exploring the intersections between CPR and SES governance and concepts such as justice, gender, livelihood security, knowledge, and power (Mansfield 2007; Brewer 2012; Barnett and Eakin 2015; Pinkerton and Davis 2015; Barnett et al. 2017). While these contributions have undoubtedly enriched our understanding of governing CPRs and SESs, they have also drawn greater attention to the many challenges in defining success.

The first challenge relates to multi-dimensionality. Since studies often reduce complex and multi-dimensional outcomes to singular or few measures of success, important information about social-ecological processes and context can be lost. For example, the Northwest Connemara oyster case study (Steins and Edwards 1999) was used in meta-analyses conducted by both Cox et al. (2010) and Baggio et al. (2016). In both meta-analyses (Cox et al. 2010; Baggio et al. 2016), coders were compelled by predefined criteria set out in their codebooks to code this case as “unsuccessful.” However, the original article argued that “[p]redefined distinctions between categories of ‘successful’ (e.g., cooperation) and ‘unsuccessful’ (e.g., free-riding) are of little help,” because the use of these categories obscure more in-depth analysis of complex contingent processes (Steins and Edwards 1999, p. 551). Additionally, as Steins and Edwards’s (1999) demonstrate, while the Connemara Shellfish Farming Cooperative discussed in this case study did not achieve its stated goal of rebuilding native oyster stocks, it successfully achieved the cooperative’s “hidden objective,” of maintaining boundaries to protect important fishing grounds from the expansion of aquaculture. The cooperative was thus a political success from the perspective of a fish harvester, but an ecological and collective-action failure from the perspective of case study analysts. This example suggests that unidimensional examinations of multidimensional outcomes can obscure the meaning and management implications of social-ecological phenomena. This is a salient difficultly in large synthesis studies. A trade-off exists between describing reality accurately in detail and drawing meaningful and generalizable lessons. This is not uniquely associated with synthesis methods per se, but is rather a problem of methodological consistency across individual case studies, with the need to link to general theoretical frameworks but also adapt data collection and analysis techniques to each case individually so to best understand them. (Cox 2008; Ostrom 2009). Methodological transparency and data sharing during publishing is an additional barrier for synthesis studies (Partelow 2018).

Multidimensionality is particularly problematic given that the social context can affect multiple dimensions of success, creating trade-offs between social and ecological goals (Klain et al. 2014; Oberlack et al. 2015; Nilsson et al. 2018). For example, Oberlack et al. (2015) demonstrated that the community-based forest governance system in Mawlyngbna, India, was robust in ecological and institutional terms, but it involved community inequality. Persha et al. (2011) found evidence for tradeoffs and synergies between improved subsistence livelihoods and tree species richness in forest systems. Where forest users participated in the rulemaking process, the likelihood of “joint wins” was enhanced. Understanding if, when and how trade-offs exist is an important dimension of defining success.

The second challenge relates to problem orientation or problem framing: What is the problem that needs to be ‘successfully’ overcome? The ways in which researchers frame and define the problems that need to be solved form the basis for delineating performance indicators and successful outcomes. These framings may be explicitly stated or inferred implicitly. For instance, if studies examine resource governance through the lens of appropriation or provisioning dilemmas, they might be likely to measure success through metrics of ecological and institutional robustness (rather than, e.g., justice) to explain why natural resource systems are overused or degrading (Ostrom et al. 1994). However, in-depth case studies demonstrate that problem orientations and framings are linked to normative and political foundations, where framings of the problems themselves can be highly contested (Leach et al. 2010; Eriksen et al. 2015; Barnett et al. 2017). The framing of a given policy problem or governance dilemma reflects particular understandings of the kinds of intervention that are possible or favorable, and the kinds of actors responsible for bringing about solutions (Fischer 2003). Similarly, in sociological studies of science by actor-network theory (ANT) scholars, the formulation of problems (i.e., problematization) has been identified as a powerful means through which scientists influence the range of possible solutions and how they can position their expertise as central for resolving them (Callon et al. 1986). Therefore, analyzing problem orientations in CPR/SES studies is critical for understanding how success is evaluated, the implications, and what is absent in these formulations of success.

The third challenge is that measuring success is inherently normative (Moser and Boykoff 2013; Conley and Moote 2003), which reflects the researcher’s perspective. What is interpreted as successful is likely to vary across case studies based on what local stakeholders prefer, and/or by the aims of researchers doing the analysis. Underlying choices about the definition of success in turn shape the selection of dependent variables and related indicators. What success is and how it is measured is often guided by disciplinary orientation, or increasingly, through transdisciplinary deliberations involving decision-makers, stakeholders and scientists (Schneider et al. 2019). Such deliberations can create ‘target knowledge’, i.e., knowledge about the values or goals to be achieved in a particular context (Hirsch Hadorn et al. 2008; Abson et al. 2014; Partelow and Winkler 2016). Further, to the extent that assessments of performance depend on input from resource users, stakeholders, managers, and experts, individual case studies can also reflect the values and norms elicited in individual case-study research.

2.1 Study design and data collection

We conducted a Scopus database search for peer-reviewed studies examining resource system types including fisheries, forestry, agriculture, irrigation, common-pool, natural resources, marine reserves, and SESs (See Appendix A1 for full search strings). These searches resulted in 1,246 results, which were then narrowed down based on the following criteria:

1. The study examined common-pool resource systems, natural resource management, or social-ecological systems
2. The analysis was based on a large-N, meta-analysis, systematic review, or comparisons of a medium to large number of case studies, here defined as more than 20 case studies. Though most syntheses included 40 or more case studies, the 20 case-study criteria allowed for the inclusion of studies using qualitative comparative analysis (e.g., Srinivasan et al. 2012).
3. The study either implicitly or explicitly addressed an outcome of interest related to success or similar evaluative statements, (e.g., bright spots, win-win)

The above criteria reduced the data set to 55 articles which was further refined to a final data set of 45 articles during the coding process as 10 articles were excluded upon further examination that did not fit the above criteria for our study (see Appendix A2 for list of articles).

2.2 Coding

A team of 11 coders developed two datasets. The first dataset consisted of descriptive questions about the studies, such as the temporal range of case studies included, geographic extent, whether the study addressed tradeoffs, resource systems and property systems included, and analysis methods. Coders filled in this dataset using a questionnaire and spreadsheet with a predefined set of responses, which was developed collaboratively and pilot tested on a sub-sample prior to implementation. Coder responses to questionnaires automatically updated a collaborative spreadsheet.

The second dataset consisted of qualitative themes coded from each study using collaborative spreadsheets. Coders entered direct quotations of problem orientations, definitions of success, and evaluative criteria for each article, which were later used for inductive coding.

The qualitative themes were defined as follows:

1. Problem orientations: Statements made in the first few paragraphs (i.e., introduction) of an article indicating the primary challenge(s) to be addressed in the study, or in natural resource management practices. Problem orientations are related to commons dilemmas as highlighted in seminal commons literature (e.g., Ostrom 1990; Ostrom, Gardner, and Walker 1994), though some additional problem orientations are highlighted in the analysis below.
2. Definitions of success: More commonly associated with methods and results, definitions of success provide details regarding what outcomes are to be achieved for a case to be deemed successful. In some cases, success is defined in the introduction, especially in cases where success is measured using a binary coder evaluation. In some studies, the definition of success emerged from an inductive analysis of case studies.
3. Evaluative criteria: Methods used to determine the conditions under which success criteria were met (correlations, regressions, qualitative analysis, etc.).

While problem orientations and definitions of success are related concepts, there are important distinctions to highlight for the analysis that follows. A problem orientation focuses the reader’s attention to specific interactions and dilemmas, and thus provides a frame within which conditions for success can be met. For example, while a fishing community may face various problems such as safety at sea, high unemployment, or high transaction costs in the buying and selling of fishing quotas and permits, a commons study may focus on the overexploitation of fisheries at sea, an appropriation dilemma, and potential demand-side provisioning dilemma (see Ostrom, Gardner and Walker 1994). It follows that success in the fishing community would be measured in terms of exploitation rates and measures of ecosystem health. Safety at sea, unemployment, and fishing quotas and permits would only be of interest to the analysis if they have hypothesized relationships with exploitation rates and measures of ecosystem health. By examining the relationship between problem orientations and definitions of success, we explore where attention is focused in the CPR and SES literature, and whether success measures adequately assess success in addressing the full suite of problems.

All coding steps were conducted by two coders for each paper. The independent coding was followed by documented discussions addressing any disagreements between coders. All remaining coding disagreements were resolved by the first author to ensure consistent interpretations.

After text segments relevant to problem orientations, definitions of success, and evaluative criteria were coded, the first author sub-coded each text segment. Text segments were sub-coded for the presence of different dimensions of problem orientations and definitions of success, and methodological approaches, the diversity of which resulted in a large number of codes. The number of codes was reduced by aggregating codes within categories, according to a modified typology developed by Ostrom (1990, 2005, 2007) and Ostrom et al. (1994). We first categorized codes according to their relevance to challenges at different levels of institutional analysis (Ostrom 1990, 2005). The distinctions between these levels have been simplified for this analysis to reflect the data available in our coding. Collective choice and constitutional levels were combined in our analysis since studies often referred to management or participatory decision-making without specifying the applicable institutional levels. These levels refer to problems and successes related to processes affecting collective choice decision-making (constitutional), and the generation of operational rules (collective choice). Operational level problems orientations and definitions of successes relate to the interaction among individuals and their environments in light of internal and external incentives, and the operational rules influencing these interactions (Ostrom 2005). We further categorized codes that captured the benefits derived from interactions at the operational level (socio-economic benefits) and temporal dimensions (functional stability). In contrast to an operational level demand-side provisioning problem, in which current appropriation levels impact future appropriation potential, this time dimension captures the overall functioning of the CPR system or SES over time, and relates more closely to concepts such as resilience and adaptive capacity. An additional category was used to indicate studies using an inductive method to define success.

Within these categories, we further categorized codes according to commons dilemmas detailed in Ostrom et al. (1994) and the evaluative criteria highlighted in Ostrom (2005). Ostrom et al. (1994) describe dilemmas relating to appropriation, including appropriation externalities, assignment problems, and technological externalities, and dilemmas relating to provisioning, including demand and supply-side provisioning problems. While these distinctions between dilemmas and applicable forms of success would be beneficial, we found that problem orientations and definitions of success in the literature reviewed often did not reflect these distinctions. Thus, our coding categorized problem orientations and definitions of success related to appropriation, and demand-side provisioning. Ostrom (2005) provide helpful evaluative criteria for additional categories of problem orientations and definitions of success: economic efficiency, equity, adaptability, resilience and robustness, accountability, and conformance to general morality. We used these categories for guidance in the development of the problem orientations and dimensions of success for our analysis, with some modifications due to data constraints or a lack of specificity in the literature reviewed.

2.3 Statistical analysis

First, we examined gaps in the literature based on a summary and qualitative assessment of descriptive questions used to code case studies. We assess the degree to which aspects of power and definitions of social groups are addressed in CPR and SES synthesis studies. This more qualitative element of our analysis provides a basis for a discussion of studies that demonstrate the potential of incorporating multidimensionality, power, and well-defined social groups in their research.

Our second research question examines the multidimensionality of success. Once dimensions of success were categorized and described, we quantified the multidimensionality of success definitions in a given study as the sum of dimensions considered. Inductive measures of success were not included in this measure since inductive approaches elicited other substantive dimensions. This resulted in a maximum of 6 dimensions of success. We conducted an ordered logistic regression to examine factors influencing the number of dimensions of success. We examine three sets of factors influencing the number of dimensions of success; resource system types, methods used in the study, and property systems.

Our third research question examines problem orientations, and the relationship between problem orientations and definitions of success. We first examined the differences between frequency histograms of problem orientations and definitions of success. Second, we used a pairwise Spearman’s rank correlation matrix to determine whether dimensions of success correlate with similar problem orientations. These correlations estimate the degree to which a study addresses the critical challenges set out in the initial framing of the article.

3.1 Background data and gaps in the literature

The case study contexts of all meta-analyses are shown in Figure 1a and 1b. The most common resource contexts studied were forestry and fisheries. Common property cases were the most frequently studied form of property rights followed by state and co-management arrangements. Of a total of 45 studies, 53.3% (n = 27) analyzed a globally distributed cohort of cases, which we defined as representation of three or more continents. A similar number of studies (n = 23) claimed that their results had global implications. The majority of studies (n = 28) included specific property rights arrangements, such as state (e.g., protected areas), private (e.g., catch shares), and common property, as an explicit inclusion criterion for selecting their cases. Most studies used data individually collected from a literature search (n = 24), while some utilized data from existing databases such as the International Forestry Resources and Institutions (IFRI) database (n = 13), a few collected empirical data (n = 7), and 1 study used a combination of these approaches.

We found potential gaps in the literature for addressing power and social dynamics, as well as tradeoffs. Out of the 45 synthesis studies included in our analysis, few (n = 8) explicitly considered power dynamics as part of their analysis or discussion. A majority had implicit references to power (n = 20), such as mention of group inclusion, empowerment or participation, while the remaining 17 studies did not address power implicitly or explicitly. The 8 studies that address power dynamics did so in various ways, including individual measures or independent variables (2), overall discussions and problem orientations (2), the discussion of outcomes and conclusions (1), or in multiple sections of the article (3). Less than half of studies (n = 19) mentioned the actors or resource user groups to whom success would apply. Studies framed system interactions in different ways. Single direction linkages (n = 19) between independent and success variables (e.g., regressions) were common, along with unidirectional analysis that incorporated moderating or intermediate variables (n = 18). All five studies that examined trade-offs considered two or more dimensions of success. Almost all studies examined case studies at a local level (n = 40), but only 40% made the geographic data or general locations (n = 18) for their cases available. Temporal data was available for 32 (71.1%) case studies, usually in the form of a list of primary literature articles, including the date of publication.

3.2 Problem orientations

As indicated in the definitions and examples in Table 3, six of eight problem orientations corresponded with definitions of success. For example, authors mentioned the need for participation and inclusive decision-making at the constitutional or collective choice level, which corresponds with definitions of success related to participation or community empowerment. Similarly, appropriation problems corresponded to sustainable appropriation. We found, however, some problem orientations that differed from dimensions of success. At the operational level, some studies pointed to lacking operational rules to incentivize sustainable resource use or behavioral change. This problem orientation, however, relates well to success in terms of behavioral and attitudinal changes, since operational rules influence behavioral outcomes. Additionally, some studies pointed to problems stemming from property rights, such as collective action challenges stemming from common property rights, or explicit statements problematizing a lack of private property rights. This result is not surprising, since the nature of property rights and boundaries are often proposed in the problematization of CPR and SES challenges, but property rights are rarely deemed successful outcomes in themselves.

Figure 3 shows the frequency of different problem orientations and definitions of success (not mutually exclusive) within the meta-analyses themselves. Some clear differences are evident. Appropriation problems were by far the most common, followed by socio-economic benefits, governance/management problems and demand-side provisioning problems. In contrast, demand-side provisioning was the most common dimension of success, followed by socio-economic benefits, effective management and sustainable use of resources. Figure 3 also demonstrates that studies often mention problems related to socio-economic benefits, governance, and participation/inclusion, but less often address these problems when measuring success. The difference between the frequencies of appropriation and demand-side provisioning themes likely stem from the close relationship between these problems and measures of success. Measures of ecosystem, habitat or resource conditions often serve as indicators that resources are being appropriated sustainably.

Problem orientations were rarely associated with dimensions of success, and half of associations were negative (Table 4). Studies that framed problems in terms of a lack of operational rules were less likely to examine socio-economic indicators of success, while studies focused on demand-side provisioning problems were less likely to examine dimensions of effective management and functional stability. Studies oriented towards socio-economic challenges were more likely to use inductive approaches to examine success, though it is important to note that only five studies used inductive approaches. Of the 45 studies, only 3 explicitly frame problems in terms of functional stability. Despite the small number of studies, we found a significant positive association between stability problems and dimensions of success that considered participation and inclusion and functional stability.

4.1 Gaps in the literature

Similar to other reviews of the literature (e.g., Miller et al. 2012; Hajjar et al. 2017) our analysis identified gaps and missing linkages in the CPR and SES literature. The majority of studies did not address tradeoffs between outcomes. Hajjar et al. (2017) found that the literature focused on environmental outcomes and relied on qualitative assessments of socioeconomic outcomes. We found a similar focus, and where authors addressed socio-economic problems, they were more likely to use inductive techniques to elicit success measures.

Very few studies explicitly considered power relationships. While studies often mention empowerment, participation, and inclusion, few examine the ways in which natural resource governance and management interventions can reconfigure the relationships between legitimized knowledge, claims to authority, and the subjectivities of resource users, managers, and other stakeholders (Agrawal 2005; Eriksen et al. 2015). Explicit examinations of power dynamics included indicators of public power exercised for private gain (Cinner et al. 2016), degree of devolution of power (Casse and Milhøj 2011), indicators of relative bargaining power and local influence over decision-making, monitoring, and operational rules (Maliao et al. 2009). Howe et al. (2014) analyzed case studies of the use of ecosystem services for human well-being with explicit attention to tradeoffs, synergies, and the dynamics that create winners and losers in management outcomes. Hirschnitz-Garbers and Stoll-Kleemann (2010) argue that protected areas outcomes should be assessed using indicators of biocultural diversity, participation and meaningful voice in decision-making, and indicators of class, education, income, ethnicity, and gender. Mwangi et al. (2012) analyze case studies with attention to the relative bargaining power of smallholders, and the conditions that contribute to elite capture of decentralized natural resource management regimes. These measures and approaches to power can further unpack the details of participatory process, and attributes of processes that help managers and resource users navigate tradeoffs and synergies in CPR and SES management.

Most often studies that considered social groups in their inclusion criteria used terms like resource users or communities, but few studies address the social and political construction of these groupings. This can be problematic given that some actors may be historically and systematically excluded from formal designations of who counts as a legitimate resource user (Smith and Basurto 2019). For example, few studies differentiate outcomes of success across gender, despite the recognition of their importance in CPR systems like small-scale fisheries (Smith and Basurto 2019; Meinzen-Dick et al. 2014). Gender also represents dimensions of power, which may be difficult to measure, but are nonetheless recognized as important outcomes associated with CPR and SES success (Epstein et al. 2014; Fabinyi et al. 2014; Boonstra 2016). These social group dynamics often intersect with collective or individual property rights, which can further influence the nature of exclusive or inclusive access to resources among social groups (Meinzen-Dick et al. 2014; Barnett et al. 2017; Brewer et al. 2012). Future research could benefit from integration with scholarship examining the relationships between the environment and race, ethnicity, gender, religion and other dimensions of human and social diversity (Ishiyama 2003; Hartberg et al. 2016; Baldwin 2017; Pulido 2017; De Lara 2018; Meinzen-Dick et al. 2014; Grove et al. 2020). Additionally, this research could explore the relationships between social group dynamics, private and collective property systems, and tradeoffs and inequities in outcomes for communities inside and outside clearly defined resource boundaries. This research presents a challenge to CPR and SES research: how do we determine the efficacy of conclusions drawn about successes and failures of management when boundaries drawn may systematically exclude some users and ignore dimensions of power altogether?

We also found a lack of truly long-term studies showing trends over time (but see De Moor et al. 2016). Thus, SES and CPR research cannot adequately assess intergenerational aspects of success such as justice or equity in access (Gibson 2006), long-term tradeoffs between use and conservation (Klein et al. 2013) or whether institutional path dependencies set limitations on potential social-ecological outcomes (Mahoney 2000; Tekwa et al. 2019). Further, it is important to consider the implications of examining case studies at one moment in time to draw generalizable conclusions about success (Barnett et al. 2016). As these cases are used in synthesis studies, they often lose their historical context and may be treated as temporally invariant, when in reality, outcomes deemed successful or unsuccessful are in flux and are context-specific. These challenges highlight the need to develop trans-disciplinarity, and to examine the role of variables and less-studied processes in determining successful outcomes.

4.2 Multidimensionality

Our second research theme addressed the dimensionality of success measures. We found an emphasis on single measures of success, most often measures of ecosystem conditions or sustainable resource use. While SESs are inherently complex, the simplification of outcomes is a common feature of the success discourse. Numerous studies looked at just a single metric of success, in order to address specific hypotheses. The challenges arise in understanding why that metric is being measured as a success indicator (i.e., the transparency issue), and how these data and conclusions are used to inform generalizations about success despite difficulties for comparison across cases due to methodological heterogeneity (i.e., theory building and external validity).

Logistic regressions revealed some characteristics that corresponded to the multidimensionality and uni-dimensionality of studies. The multidimensionality of irrigation success measures may be due to the often-stated challenge of maintaining the provisioning capacity of irrigation infrastructure and water abundance, while also addressing inequalities between head-enders, who often have the capacity to use more irrigation water than tail-enders (Ostrom and Gardner 1993; Baland and Platteau 1999). Correlative studies often examine the effects of independent variables on a target outcome, and thus often consider fewer dimensions of success. Gutierrez et al. (2011), however, aggregated social, economic, and ecological binary outcomes to develop a multidimensional success score with a minimum of zero and maximum of eight. Their resulting analysis did not examine if and how variables such as social cohesion and leadership influence social, economic, and ecological success variables differently. Grassland and rangeland studies often focus on the maintenance of ecosystem variables and biodiversity (Torralba et al. 2016; Robinson et al. 2014; Porter-Bolland et al. 2011), though Howe et al. (2011) explicitly examined tradeoffs and synergies between ecological and socio-economic outcomes. Synthesis studies examining state property systems, such as protected areas, were more likely to consider success as unidimensional. One exception is a meta-analysis conducted by Maliao et al. (2009). This study compared eight indicators encompassing five dimensions of success before and after the implementation of community-based coastal resource management programs, and found mixed success across indicators. These exceptions to unidimensional analyses of success in CPRs and SESs demonstrate that interventions may be effective in one dimension, but have negative or different impacts on others.

While tradeoffs did not contribute to our logistic regression model, all five tradeoff studies considered at least two dimensions of success. These studies provide examples of how to examine the tradeoffs and synergies between livelihoods, sustainability, and equity (Agrawal and Benson 2011), the relative strengths and weaknesses of various typologies of international seafood trade arrangements (Crona et al. 2016), and the potential for projects that incorporate capacity-building and participation in governance to generate synergies among social and ecological outcomes (Brooks et al. 2012; Persha et al. 2011). These findings suggest that although tradeoffs are rarely explicitly examined, they can contribute to more holistic analyses of multidimensional outcomes in synthesis studies.

4.3 Problem orientations

Although we found corresponding dimensions of success and problem orientations, we did not find a strong association between descriptions of the overall problem to address and dimensions of success. This suggests that the normative foundations of defining success are not explicitly defined in the framing of journal articles, but implicitly determined by choice of success indicators, or the limitations of available data. Success measures often cannot adequately address the full suite of problems recognized in synthesis research. Our analysis suggests that future research could expand on our current understanding of CPR and SES governance and address problems by developing and testing measures of successful process, such as participation, inclusion in decision-making, and indicators for good governance or management practices. These processes constitute intermediate steps within a causal chain influencing social, ecological, and economic outcomes, trade-offs, and synergies. As we note above, the relationship between private and collective property rights and social group dynamics also play a role in this process by determining who participates in natural resource exploitation and management. Further, socio-economic outcomes are frequently stated as a motivating challenge in synthesis studies, but socio-economic indicators should also be included when assessing outcomes.

Providing a thorough account of the normative foundations to SES and CPR synthesis studies is beyond the scope of this study. Other factors may influence the lack of association between problem orientations and definitions of success, and biases in defining success, such as data availability, challenges when operationalizing and testing dimensions of power, participation, and inclusion, and missing or unavailable data in synthesis studies that review primary literature studies. However, our data indicates that CPR and SES synthesis studies often place the most value on maintaining ecological conditions and reducing the use of natural resources, perhaps unsurprisingly given the theoretical roots of CPR and SES studies. However, our review demonstrates that by expanding our focus on the political processes of natural resource management, the socio-economic impacts on communities, and the conditions that generate tradeoffs and synergies, we may develop more holistic examinations of success to improve the conditions of natural resources, livelihoods and equity, and governance arrangements.

4.4 Critical discussion on collective action for evaluating success

Measuring success in the CPR and SES literature reflects a broad trend in many fields studying natural resource governance, where finding joint evaluative criteria and definitions as well as making data comparable through increased accessibility and methodological transparency poses significant challenges. If developing robust theories of commons and social-ecological systems governance research is a shared goal and high priority, making generalizable claims about success needs to be well grounded in comparable empirical data. However, Conley and Moote (2003 p. 376) remind us that “whatever form an evaluation takes, researchers must acknowledge that evaluation is normative, and inevitably political…”, and thus “…the criteria, weightings, and methods used must be made clear if evaluations are to be fairly compared to each other.” The problem is not only due to diverse ways to measure success and its normative nature, but rather trying to make generalized theoretical claims about what success entails with a wide diversity of definitions and a large body of data that is highly heterogeneous and not directly comparable. The vagueness of the term ‘success’ is both an asset and a limitation in advancing cumulative knowledge about successful SES and CPR governance – an asset because it facilitates dialogue across specific disciplinary lenses, but a limitation because the precise meanings of ‘success’ in specific disciplines and studies often becomes blurred, which limits the precise accumulation of knowledge about successful SES and CPR governance. Thus, we need to use more specific definitions of success and evaluative criteria (Gibson 2006; Partelow et al. 2019). Much of this difficulty stems from the need to develop methodologies to better structure data comparison, particularly with the use of qualitative data collection and analysis methods, and the combination of qualitative and quantitative methods.

One major challenge for assessing success is data comparability, which stems partly from a collective action problem. Research on knowledge commons suggests that the provision of knowledge is a public good, and thus producing knowledge within science itself represents the structure of a provisioning collective action problem, similar to those observed in natural resource systems. There are few incentives for researchers (with individual interests) to generate and use data collection and analysis techniques that are broadly comparable (the group interest) (Hess and Ostrom 2003; Poteete et al. 2010; Fisher and Fortmann 2010). Methodological novelty in case study research is often rewarded and incentivized in top journals. This leads to new combinations of methods, variables and indicators in a way that will give individual research the best contextual understanding of a case study (higher internal validity). It is also difficult to replicate published research because there are limited incentives to share data or improve methodological transparency in the social sciences (Moravcsik 2014; Rowhani-Farid et al. 2017). Even using common frameworks such as Ostrom’s design principles (Ostrom 1990) or the SES framework (Ostrom 2009) across cases does not preclude heterogeneous data collection and analysis methodologies, which makes findings from these studies difficult to compare, requiring secondary coding or analysis that further abstracts from their original meaning (Cox et al. 2010; Thiel et al. 2015; Oberlack et al. 2016; Ratajczyk et al. 2016; Partelow et al. 2018).

Poteete et al. (2010) outline a research program for advancing CPR and SES research through the integration of different methodologies (i.e., experimental with observational; qualitative with quantitative) to advance the field. The core aim of many frameworks, such as Ostrom’s SES framework, is to assist data comparability across cases during the research process by providing a tool for researchers to design research questions, data collection and analysis methods using common variables. CPR and SES researchers have developed databases to solve the problem of data comparability post-hoc, by providing structured coding and data entry procedures, retrofitting existing data into data structures that better facilitate meta-analysis. However, substantial progress is lacking, which can largely be attributed to a lack of joint individual incentives and current methodological barriers. For instance, while the SES framework provides some 50 second-tier variables to characterize resource systems and units, governance systems and actors, contexts and related ecosystems, it is vague in terms of operationalizing ‘social performance’ and ‘ecological performance’, i.e., the precise metrics of success. Further, there remains space for elaboration on these frameworks, as demonstrated by Clement’s (2010) recommendations for expanding on the SES framework by including analyses of discourses, political and cultural factors, and adopting a multi-level analysis of power dynamics. These challenges add to the complexity of developing generalizable theory, and bring into question whether such a goal is attainable. Despite these challenges, we contend that methodological pluralism, combining methodological approaches such as grounded theory, synthesis studies, experiments, and in-depth case studies can improve the process of generating, refining, testing, and revisiting theory, and ultimately better inform CPR and SES management practices.

4.5 Methodologies for examining multiple dimensions of success

As mentioned in section 1.1, the challenge of addressing success in all its complexity while building meaningful theory is not exclusive or necessarily linked to large-N analyses. Some of the reviewed studies indeed demonstrated interesting and innovative methods for explaining success and failure in CPR and SES governance. Most of these studies were ultimately excluded from the above analysis but are discussed qualitatively. Crona et al. (2015) and Srinivasan et al. (2012) approached the evaluation of interactions and outcomes using a syndromes approach. Since social-ecological change is multi-faceted, bringing together multiple processes, flows, and factors co-occurring and operating at different scales and affecting different sectors is valuable. Thus, instead of analyzing social-ecological change in terms of a limited number of predefined “good” or “bad” outcomes, an analysis of syndromes enables researchers to identify the processes, characteristics, trade-offs, and synergies that co-occur with institutional and infrastructural arrangements in different contexts (see also Schellnhuber et al. 1997; Lüdeke et al. 2004; Manuel-Navarrete et al. 2007). These studies make a compelling case for wider usage of analyses of syndromes for understanding complex systems such as CPRs and SESs. Similar approaches are associated with archetype analyses in sustainability research (Eisenack et al. 2006; Sietz et al. 2012; Vaclavik et al. 2013; Oberlack et al. 2016; Eisenack et al. 2019; Oberlack et al. 2019), which identify building blocks of causal interactions beyond the focus on problematic, ‘clinical’ symptoms typically associated with syndromes (UNEP 2007). Such analyses, however, are most suitable for analysis of an intermediate number of case studies, between in-depth case study research and large-N studies.

Leslie et al. (2015) operationalized Ostrom’s (2007) diagnostic SES framework using a combination of mapping to determine SES regions as a unit of analysis and an examination of the relationships between SES variables. They developed 13 indicators encompassing dimensions of resource units, resource system, governance system, and actors. Each variable was quantified and normalized based on their expected relationship to sustainability. Leslie et al. (2015) demonstrate an approach that can integrate qualitative and quantitative data to examine the sustainability of spatially differentiated regions, and to test hypotheses on the interrelationships between SES variables. Further, based on their analysis, they conclude that there are diverse pathways to sustainability facilitated by various combinations of social and ecological factors. Similar to a syndromes approach, this operationalization avoids overemphasizing a limited number of outcomes, and requires in-depth analysis, which may not be suitable for large-N studies. Lastly, Leslie et al. (2015) develop variable scores that abstract from and pre-define relationships between aspects of an SES and sustainability outcomes.

We found some studies that include resilience, adaptation, and the long-term endurance of a governance system as a dimension of success. These studies often rely on a researcher’s qualitative assessment of longevity. De Moor et al. (2016) provide a unique methodology for analyzing the success of case studies in terms of longevity. They examined archival records of regulations applied to Dutch, English, and Spanish CPRs spanning as many as 751 years, the link between longevity and commoner participation, sanctioning mechanisms, and operational rules. They suggest that their approach is limited, since archives will only account for formalized rules and not all rules-in-use, and since longevity may not co-occur with other dimensions of success, such as socio-economic benefits, or evenness of power relations. However, they argue that these challenges can be addressed by first reading formal institutional changes as indicators of the evolution of perceived optimal CPR use, and then by combining attention to participation, environmental sustainability, and the reproduction of communities involved. De Moor et al. (2016) demonstrate a novel and innovative strategy to improve our understanding of institutional adaptation and longevity that warrants greater attention among CPR and SES scholars.

4.6 Four recommendations for future success research

As a conclusion to the findings and discussion of this research above, we provide four recommendations for analyzing dimensions of success in CPR or SES governance:

1. Conceptual and Methodological Rigor and Transparency. Our analysis demonstrates that success definitions correspond with constitutional, collective choice, and operational levels of analysis, and behavioral, appropriation, and demand-side provisioning dilemmas. However, few studies explicitly stated these conditions and the social groups to whom success would apply. In order to generate robust theory, the scope and validity of research findings needs to be assessed [by others] accurately (Magliocca et al. 2018). Further, the degree to which this secondary data interpretation and coding is accurate depends on the ability to understand what, how and when data was collected and for what analytical purpose. Methodological rigor should be enhanced by investing more in common, well-validated research instruments to operationalize outcome variables in CPR and SES research. To further promote methodological transparency and internal and external validity of study results, studies should include metadata for synthesiss studies.
2. Iterative and Complementary Analysis. We agree with recommendations for methodological pluralism (Poteete et al. 2012). Further, we recommend increased innovative and constructive integration of qualitative and quantitative methods, as well as inductive and iterative analyses that seek out and examine the gaps, inconsistencies, outliers, and counterfactuals inevitably left by synthesis analyses. For example, Baggio et al. (2016) found configurations of design principles that influenced outcomes in different infrastructural contexts. These results were further supported by and expanded on by a qualitative examination of changing contextual conditions (Barnett et al. 2016). Replication studies and iterations that examine previously defined successes from a multi-dimensional trade-off perspective would expand on and add complexity to CPR and SES governance findings.
3. Assess Success as Multidimensional The current reliance on singular dependent variables and simple unidirectional models limits our ability to understand the dynamic and contingent nature of success. Studies most often measured success in terms of demand-side provisioning. This leads to many questions, such as, did management practices improve conditions? Did improved conditions negatively or positively impact livelihoods? Future research can place greater emphasis on multiple tradeoffs and the interactions between multiple dimensions of success to contribute to our understanding of SES and CPR as complex, multi-dimensional, and interactive systems.
4. Integrate Critical Approaches. Social heterogeneity has important implications for SES and CPR management, and some studies have begun to examine the intersections between, for example, sustainability, resilience, justice, equity and socio-cultural diversity measures such as gender and power. Elements of these critical approaches are already present in the literature reviewed. We recommend efforts to increase the frequency of these approaches, and to further integrate with additional critical social science perspectives. Above, we highlight various studies that integrate critical theory concepts to examine CPR and SES management and success, whether adopting inductive or deductive analyses, or large-N or alternative methods. Additional research can further explore how and where various critical approaches can apply to a wide array of CPR and SES studies, particularly the role of interdisciplinarity and transdisciplinarity.