Increasing demand for local and regional-scale ecosystem service (ES) mapping and assessment to support biodiversity management (Nagendra et al. 2013, Posner et al. 2016), land-use planning (Darvill and Lindo 2015, Kopperoinen et al. 2014) and environmental impact assessment (Geneletti 2013) has driven a need for robust and scientifically sound methods for assessing ES capacities, demands and/or preferences (Harrison et al. 2017).

Various methods for assessing ES have been used in various studies (see Burkard and Maes 2017, Harrison et al. 2017), requiring various types and degrees of expertise from people implementing them and mobilising various amounts of data (Harrison et al. 2017). The right choice of method should articulate the goals of assessment and mapping (Jacobs et al. 2015) but also the applicability and appropriation of the method and the results expected by stakeholders and land managers. Beyond quantifications by expert-scientists, the ES concept can only be put into action if the assessments are understood and supported by the end-users (Harrison et al. 2017). The actual complexity and perceived difficulty in interpreting results is a limiting factor to be considered in any ES assessment and mapping exercises (Jacobs and Burkhard 2017). Furthermore, acquiring, compiling and processing multiple data sources can prove challenging if not intractable for many ES, particularly at finer scales.

Amongst the different methods available, spatial proxy models — which link land cover classes or ecosystem types to ES — are very flexible and readily adaptable to different data sources and modelling techniques. The matrix approach — particularly with participatory expert-based scoring as developed here — has proven to provide rapid and readily-appropriable ES assessments (Burkhard et al. 2011, Burkhard et al. 2012, Jacobs et al. 2015). End-users, principally land managers and decision-makers, can appropriate them quickly and apply them to implement land-use planning or resource management policies that need an understanding of the importance of specific land cover or ecosystem types rather than precise estimates of ES quantities (Grêt-Regamey et al. 2008).

The matrix approach is increasingly being applied (Fig. 1) in various contexts, which offers good feedback on the pros and cons of the method. It has been applied in many case studies (e.g. Hermann et al. 2013, Kroll et al. 2012, Stoll et al. 2015, Vihervaara et al. 2010), in countries from Austria and Hungary (Hermann et al. 2013) to China (Cai et al. 2017), the USA (Cottillon 2013), Thailand (Kaiser et al. 2013), France (Campagne et al. 2017) and Nepal (Paudyal et al. 2015) and at scales from local (Nedkov and Burkhard 2012) to national (Depellegrin et al. 2016) and continent-wide (Stoll et al. 2015). The diversity of application confirms that "the approach has the potential to integrate all kinds of ES-related data based on diverse scientific disciplines or ES quantification methods and of varying quality and quantity in illustrative matrix tables and maps" (Burkhard 2017).

Grêt-Regamey et al. (2015) proposed a tiered approach to ES mapping that classifies ES mapping and assessment methods into 3 levels (tiers) of increasing complexity from tier 1 to tier 3. Under this tiered framework of ES assessment methods, the matrix approach and expert knowledge is classified as tier 1 or tier 2 depending on the complexity of the methods used to build the matrix (Grêt-Regamey et al. 2017). Several aspects of capacity and demand can be addressed, including biophysical capacity, economic demand and societal preferences.

Burkhard (2017) propose a methodology of the matrix application for ES mapping based on ES indicators and a collection of suitable spatial data. Nevertheless, if source data is scarce, fragmented or not publicly available, expert knowledge can be the best available source or proxy for ES estimates (Jacobs and Burkhard 2017, Kienast et al. 2009). The expert knowledge is commonly used as a surrogate for empirical data in many fields of ecological research (Drescher et al. 2013). Experts use a combination of field observations, formal knowledge and mental models to generate quantitative information (Fazey et al. 2006. The use of expert elicitation methods is shown to be an efficient option to address the simplicity/complexity and precision/uncertainties trade-offs (Jacobs et al. 2015). Hou et al. (2013) and Jacobs et al. (2015) addressed critiques of the matrix approach and some of the issues regarding consideration of experts' uncertainties have been address in Campagne et al. (2017). Expert-based assessment has many well-known biases (subjectivity, interpretations etc.) that some methods help to reduce (Martin et al. 2012, Mukherjee et al. 2018).

The methodology we propose here aims to reduce biases linked to interpretation and influence biases when using the expert-based matrix approach for ES scores elicited best-knowledge estimates, not individual preferences. Indeed, it is possible to estimate the biophysical capacity, economic or social value of ecosystem services of different ecosystem types by knowledge or preference elicitation methods based on individual interviews and/or collective deliberative workshops. This is the case for the widely-used multi-criteria approaches, ranking methods and life satisfaction approaches and also for other less frequently used methods, such as the Q method (Rodríguez-Vargas and Marburg 2011) or the Delphi prospective approaches (Martin et al. 2012).

The method presented here is a knowledge elicitation protocol aiming to evaluate the capacity of different ecosystem types to provide their best evaluation of the potential (or the use, the demand etc.) of each ecosystem type to provide ES. So the ES scores elicited are not individual preferences (i.e. Kopperoinen et al. 2017), but best-knowledge estimates (i.e. Stoll et al. 2015). We propose to adopt an explicitly structured and robust procedure, involving several steps linked with the study design, capacity building, scoring and expert accuracy and uncertainty assessment.

Guidance is needed on how expert knowledge should be collected and integrated to construct scientifically sound ES evaluations (Martin et al. 2012). Here we propose general guidelines on how to implement expert-based ES matrices based on published papers using the matrix approach and a series of local and regional assessments conducted in France with groups of experts. We propose a 7-steps methodology for producing semi-quantitative estimates of ES quantities based on expert knowledge that starts by defining the goal that will characterise the precision in the definitions and lists of ES and ecosystem types and ends by the different outputs possible (Fig. 2).

Step 1 - Goal and preparation phase

Step 2 - Workshop

Step 3 - Initial matrix

Step 4 - Filling-in the matrix

Step 5 - Compiling the Values

Step 6 - Reliability and validation

Step 7 - Outputs

As an illustrative example, we present an application of the matrix approach in the Alpilles RNP in the south of France. This ES assessment was done in August 2017 (Surun 2017). The methodology steps were applied described below.

(Step 1) The goal was to assess the potential impact of a Life+ Nature project on the ES capacity of the ET managed through actions led under the Life+ project. The LIFE is the EU’s financial instrument supporting environmental, nature conservation and climate action projects throughout the EU (http://ec.europa.eu/environment/life/). The LIFE12 NAT/FR/000107 is a LIFE + Nature project on the Alpilles RNP which aims to optimise the link between human activities and the maintenance of ornithological biodiversity, to promote the appropriation of ecological issues by local stakeholders and to strengthen ornithological recognition of the territory by enhancing certain practices. Its actions, analysed here, are restoration of a grass layer in olive groves (A1), plantation of multispecies hedgerows (A2), creation of open areas in dense garrigue scrub by shrub cutting (A3) and implementation of good forestry practices with opened forest by shrub cutting and culling (A4).

Linked to these actions, the ET selected for the matrix are not linked to landcover/use typology but to those addressed by Life actions with before-and-after action states, i.e. Bare-soil olive groves vs. Olive groves with grass cover, Multispecies hedgerows vs. Monospecific cypress hedgerows, Opened garrigue vs. Closed garrigue, Opened coniferous forest vs. Dense coniferous forest. Another 6 ecosystems were also considered in order to factor-in other important ET within the study area, i.e.: Deciduous forest, Rocky habitat, Vineyard with bare soil, Orchards with inter-row grass strips, Grassland, Annual crops. The list of ET was completed by detailed descriptions and local pictures to harmonise participants’ mental representations.

Based on CICES (V4.3, Haines-Young and Potschin 2013), we proposed an ES start-list that we then adapted by working with the park managers to select relevant ES and to adapt the definitions and examples to fit the local context. At the end, 10 regulating services, 9 provisioning services and 5 cultural services were chosen (presented in Fig. 3). Moreover, in the context of the local conditions, the ecosystem disservice (EDS) “Contribution to fire risks” was added to the matrix.

Following Burkhard et al. (2009), ES and EDS scores were ranked from 0 to 5 to express capacity and we asked for a confidence index from 1 to 3 following Campagne et al. (2017).

In total, the matrix included 14 ET and 24ES+1EDS giving a total of 350 scores. As we assessed the capacity, the experts invited to participate had theoretical and practical knowledge of the local environment and/or ES. The panel of experts counted 5 generalist profiles, 3 forest profiles and 2 naturalist profiles, along with specialist profiles in livestock, agriculture and hunting.

(Step 2) The workshop was held in August 2017 with all the in-panel experts and the course of the workshop followed the previous recommended description.

(Step 3) As the evaluation had a very specific context (assessment of Life action impacts), there was no existing matrix so we started with an empty matrix.

(Step 4) We chose a full individual fill-in to get 12 full matrices and run comparative analysis.

(Step 5) Compilation was by meansof the 12 matrices given equal weight between experts. Adjustment by consensus was not done, as the matrix was too long to be filled in during the workshop and technical reasons made it difficult to organise two workshops. The final matrix is the central scores and mean confidence scores for each ES/EDS and ET (Fig. 3). The impacts of Life actions were computed with the difference between the two linked ET states (i.e. before and after the Life actions; Fig. 4). The potential impact of the Life+ project was analysed with a Student’s t-test for paired data using the R software statistics package (R Development Core Team 2017) to test whether the two scores were significantly different.

(Step 6) Comparison with quantitative data or models would have been a challenge, as the application was too specific for reliable comparison against existing data. We computed two indices of inter-rater reliability, i.e. Krippendorff’s alpha and ICC, using the ‘irr’ package in R software. For Krippendorff’s alpha, we considered the scores as interval data and, for the ICC, we considered the consistency of the scoring. The index values ranged from 0.062 to 0.90 for Krippendorff’s alpha and from 0.09 to 0.90 for ICC. All the ICC values were significantly different from 0, indicating some degree of agreement between the experts, but there was nevertheless strong differences in concordance values for the different ES. This indicates that some ES have a very low level of agreement of rating amongst the experts and this has to be further analysed. Eight ES had high ICC values (>0.6), i.e. PS1 "Cultivated crops (including seaweed farming)" (0.9), PS6 "Materials and fibres" (0.63), PS9 "Biomass-based energy sources" (0.61), RS1 "Global climate regulation by greenhouse gas reduction" (0.72), RS2 "Local climate regulation and local atmospheric composition" (0.6), RS7 "Mass stabilisation and control of erosion rates" (0.63), RS8 "Protection against winds and storms" (0.81) and RS10 "Limitations of noise pollutions and odour and visual nuisances " (0.68). DS1 "Contribution to fire risks" had an ICC value of 0.72. This indicates that, for these ES and DS1, the different experts consistently scored the different ET scores in the same rank-order, which means they should be robust. Inter-rater reliability was very low on some ES, such as PS5 "Drinking water" (0.09), PS8 "Genetic and medicinal materials " (0.3) and CS3 "Existence and bequest" (0.27), indicating very low agreement between experts on the scores of the different ET. These results can mean that ES scores for these services lack reliability or that there are diverse interpretations of the ES between experts or these services encompass different types of knowledge that require different types of expertise. Those services should be interpreted with caution and further analyses are needed to understand the cause of the disagreement amongst experts as helicopter interviews of experts and literature reviews.

(Step 7) To illustrate the impact of Life actions, graphs bundles of ES can be used (Fig. 5). The results show that action A1 (restoration of a grass layer in olive groves) generates benefits in 21 ES including 14 ES with significant differences in the ES provided before and after the actions (t-test, p<0.05). There were no estimated changes for 3 ES. Provision of DS1 "Contribution to fire risks" increases significantly, associated with the increase in grass cover (Fig. 5). Similar analyses can be made for the impacts of actions A2, A3 and A4.

These results need to be read in relation to the confidence scores expressed by each expert (averages shown in the margins -in grey- of Fig. 3) and to inter-rater reliability. Some services such as RS8 "Protection against winds and storms" and PS1 "Cultivated crops (including seaweed farming)" have both a high confidence value and high ICC, indicating the experts are self-confident and provided concordant scoring. For some other ES such as PS5 "Drinking water", PS8 "Genetic and medicinal materials", the confidence indices are low but the ICCs are low indicating. These ES indices should be taken with cautious and certainly need complementary analyses or a new expert consultation. Finally some other ES such as the Cultural services have a high confidence index and low concordance. They will also need some further analysis since the experts seem to consider they have knowledge of them but they score very differently.

We proposed a 7-step methodology for producting an ES matrix based on expert elicitation protocols. The example we presented here was for the sake of demonstrating the implementation and the interest in the proposed methodology. Our methodology aims to promote cogency in the method and coherency in the matrices produced and to reduce biases linked to experts' judgements. In a recently published book, Burkhard (2017) proposed a 10-steps methodology for the matrix application for ES mapping. His protocol can be complementary to ours, since he focused mainly on the mapping issues and the use of spatial data, while our methodology aims to promote good practices to produce a reliable expert-based ES matrix and the analysis of different sources of variabilities. As stated by Choy et al. 2009, expert elicitation possesses several key issues of relevance for ecological applications that are relevant in our context: 1. Experts' opinions are an important source of information, 2. Ecological problems should be broken down to facilitate the expert knowledge elicitation, 3. communication style is important in targeting different groups, 4. technology could be used for providing more interactive environment, 5. Expert panels provide a useful mechanism for facilitating elicitation. The Choy et al. 2009 discussion resonates with our 7-steps approach for an ES matrix. ES are mainly complex and based on several interacting ecosystem functions and generally human contribution. The expert opinion is generally based on integrating several pieces of information rather than a very specific indicator, such as necessarily used by quantitative modelling. Our recommendations for the organisation of steps 1 to 5 and interactions with key stakeholders and experts are important in order to define the objectives and to exchange with the expert panels. We state that these steps are very important and should be overseen as mere preparatory elements. Good planning and good communication with experts are needed to ensure a reduction of many biaises that can be linked with ES scoring by experts. We preconise to use individual scoring instead of consensus methods in order to keep simpler options for statistical analysis of uncertainties (Campagne et al. 2017). More generally, interesting discussions on the use of the matrix approach can be found in Burkhard et al. (2012), Campagne et al. (2017), Hou et al. (2013), Jacobs et al. (2015). Burkard and Maes (2017) discuss ES mapping in general. In the following paragraphs, we summarise and regroup the pros and cons of the matrix approach in general, the use of expert elicitation in particular and precise the pros and cons linked to the methodology we propose here.

Pros

Flexibility

We agree with Schröter et al. (2012) by considering that Burkhard et al. (2012) made “an important contribution to the development of mapping supply and demand of ES”.

The matrix approach enables quick ES assessments (Hermann et al. 2013, Stoll et al. 2015) and provides readily understandable and "mappable" ES data (Jacobs et al. 2015). The flexibility of the matrix approach in terms of detail and levels of abstraction from rather simple to highly complex further adds to its attraction (see Burkhard et al. 2014). As stated earlier, it can be applied based on either very simple or fairly complex methods (i.e. simple or advanced matrix mapping in Harrison et al. 2017 and Tiers 1 to 3 in Grêt-Regamey et al. 2015). Moreover, other concepts can be added to the ES assessed as ecological integrity indicators (e.g. Schröter et al. 2012, Islam Sohel et al. 2015) including the notion of biodiversity and the concept of ecosystem disservices, as recommended by Stoll et al. (2015) and employed in our example application and analysis in Campagne et al. 2018.

Appropriability

The underlying ES concept, simplicity, flexibility and participatory approach with co-production of the results converge to make the final output readily understandable and appropriable by stakeholders. As in the case applied by Kopperoinen et al. (2014), experts and stakeholders appreciated the approach as very fruitful and interesting as it combined input from all experts working on a territory and provided output data integrating all their expertise-sets.

Cost efficiency

The matrix approach and expert elicitation are cost-efficient, as they provide a quick assessment for a large number of ES and a large number of ET (Jacobs and Burkhard 2017). Providing that rules are used and carefully applied (this paper), the results also have a quantified reliability and credibility that is not necessarily lower than more complex modelling approaches that also incorporate expert knowledge at different stages of parameterisation (Jacobs and Burkhard 2017, Campagne et al. 2017).

As shown in Table 1, depending and the initial matrix, the filling in and the precision of ES and ET, the time required to apply the approach varies.

Integrative

There have been very few attempts to quantitatively assess regulating services and cultural services due to the fact that available data are scarce and relevant indicator quite complex (European Commission 2014 in Schröter et al. 2012). However, the matrix approach can quickly provide estimates for all types of ES. As quantitative data based on different units poses the problem of comparability of ES/EDS potentials using different metrics, the matrix-based scoring approach is more integrative and provides same-scale estimates (i.e. 0 to 5). The intrinsic complexities of estimating values linked to ES or EDS, particularly those that cannot be easily quantified (some regulation services and most cultural services), challenges the reliability of many biophysical or economic valuation approaches (Aldred 2006). In the case of economic valuation, the question of putting value on services raises many problems (for a recent synthesis, see Rey-Valette et al. 2017), such as the perception and identification of services by social actors or the measure of option and non-use values

Cons

Spatiotemporal invariance

The matrix gives an average score of ES provided by ET/ LC types. Two distant areas with the same ET will thus have the same scores without accounting for their specificity (Jacobs et al. 2015). Protection status, ecosystem health status, topographical, topological and other features, such as socio-economic data (human population density), are not directly taken into account. Additional spatial analysis can be factored (e.g. Hermann et al. 2013) or spatial invariance (e.g. regional variation) can be considered in the ET lists in the matrix but this considerably increases the matrix size and time required to fill-in.

ES provision is temporarily variable (Kandziora et al. 2013). As stated by Turkelboom et al. (2017), the matrix with a “list of ES can give the impression that provisioning, regulating and cultural ES can be met at the same time, while in most situations it is impossible to manage ecosystems in such a way that these ES are simultaneously utilized at desired levels”.

Lack of consideration of spatial processes

The matrix does not take into account trade-offs and synergies between ES as they form an interaction network (Schröter et al. 2012), but correlation can be analysed. Also, the matrix alone does not take into account the impact in the ES provision of the interaction between ET and landscape structure with its mosaic of patches (Syrbe et al. 2017)

Expert-based estimates

The results, based on expert judgements, are limited by the experts’ own understanding and interpretation and by a number of cognitive and social biases. However, carefully-designed methods can reduce some of the impact of specific biases (see Drescher et al. 2013, Mukherjee et al. 2018).

Some ES and ET are more difficult to define, which may lead to different interpretations. Moreover, the distinction between the different components of ES notions (supply, capacity, uses, demand etc.) is also debatable and may again lead to different interpretations. The workshop plays an important role in addressing this limitation, as it serves to set aside the time needed to explain all the matrix-related definitions and to go back over them if needed.

An intrinsic limit of expert-based methods is the subjectivity of their estimates. By definition, they are not measurements but a plausible score based on the best knowledge of the experts mobilised (Jacobs et al. 2015). Expert knowledge and empirical data exist on a continuum of subjectivity and both require validation steps (Martin et al. 2012). Spangenberg and Settele (2010) stressed that, despite an illusion of precision given by the rigorous methods used, the economic evaluation of services is based on conventional premises that preclude any claim to “objectively” calculate their value. Although the expert elicitation of ES scores contains many sources of uncertainties (Hou et al. 2013), it is not obvious nor easy to determine if that uncertainty is higher or lower than other empirical methods (Drescher et al. 2013)

Relative quantification

As the ES scoring used in the matrix approach is only semi-quantitative and expert-based, we cannot equate the scores directly to actual biophysical quantities, there is an obvious need to confront them to actual or model-based quantitative estimates in order to better define their domain of validity. Some preliminary results seems to indicate a monotone relationship between expert-based ES scores and quantitative estimates but more research is needed.

Finally, we are convinced of the usefulness of the expert-based matrix approach and recommend a complete individual filling with an empty matrix for its fair compromise between the time requested, taking into account all expertise, analytical and statistical advantages and reasonable participatory time to avoid over-solicitation. As presented in the example, we quickly obtained relevant and appropriable results for stakeholders. The flexibility of the approach allows an unlimited adaptation to contextual objectives. However, for the matrix to be with you, this flexibility should be framed in order to achieve results with high scientific standards. We recommend the adoption of our explicitly structured and robust procedure, involving several steps linked with the study design, capacity building, scoring and expert accuracy and uncertainty assessment. However, we strongly recommend that the ES matrix should not only focus on ES central scores, but also address the variabilities and uncertainties as part of the ES assessment.

This study was partly supported by the H2020 ESMERALDA project (grant agreement #642007) and the IMAGINE project (ERANET BIODIVERSA 3).

The authors thank the two reviewers for the feedbacks and suggestions; Candice Christin for her support on reviewing the applied matrix approach literature and Juliet Abadie for her general feedbacks.

The applied matrix approach example presented here was carried out by Clément Surun of the Alpilles Regional Natural Park under local supervision from Jean-Michel Pirastru and Lisbeth Zechner and further supervision from Aurélien Guingard of CDC (Caisse des Dépôts copilotée) Biodiversité et la Mission Economie de la Biodiversité. The application was funded by the Alpilles NAT/FR/000107 LIFE programme and the Alpilles Regional Natural Park.