The RRI Citizen Review Panel: a public engagement method for supporting responsible territorial policymaking

Abstract

Responsible Territorial Policymaking incorporates the main principles of Responsible Research and Innovation (RRI) into the policymaking process, making it well-suited for guiding the development of sustainable and resilient territorial policies that prioritise societal needs. As a cornerstone in RRI, public engagement plays a central role in this process, underscoring the importance of involving all societal actors to align outcomes with the needs, expectations, and values of society. In the absence of existing methods to gather sufficiently and effectively the citizens’ review of multiple policies at a territorial level, the RRI Citizen Review Panel is a new public engagement method developed to facilitate citizens’ review and validation of territorial policies. By using RRI as an analytical framework, this paper examines whether the RRI Citizen Review Panel can support Responsible Territorial Policymaking, not only by incorporating citizens’ perspectives into territorial policymaking, but also by making policies more responsible. The paper demonstrates that in the review of territorial policies, citizens are adding elements of RRI to a wide range of policies within different policy areas, contributing to making policies more responsible. Consequently, the RRI Citizen Review Panel emerges as a valuable tool for policymakers, enabling them to gather citizen perspectives and imbue policies with a heightened sense of responsibility.

Introduction

In a constantly changing landscape of Responsible Research and Innovation (RRI), there is a strong need to explore more deeply the critical role of involving the citizens (Sánchez-Nielsen et al., 2014). This article emphasises the essential connection between public participation, innovation, regional policies, and the overarching idea of Responsible Territorial Policymaking (Angelidou et al., 2022). At the centre of this article exploration lies the RRI-LEADERS project, that helps us to navigate the complexity of ethics, progress, and adaptability (IIASA, 2019; Woermann & Cilliers, 2012).

In our time, it is evident that many regions and cities are confronted with significant economic and social challenges in terms of urban development (Toukola & Ahola, 2022). These challenges encompass issues such as air pollution, climate change, sustainable mobility, an aging population, and a dynamic process of in- and outmigration (Environmental Protection Agency, 2022; EPA, 2018). Addressing each of these challenges necessitates the allocation of institutional, human, and financial resources to effectively tackle them (Taylor et al., 2008). In the context of the RRI-LEADERS project, in the Region of Western Macedonia (Greece) and Sofia Municipality (Bulgaria) specific and locally oriented policy areas were elaborated for analysis and evaluation to explore the potential incorporation of the RRI-framework in policy formulation and implementation (Mamman et al., 2019). The challenge of selecting these policies was to reassess the entire policymaking process, considering the growing necessity for consensus-building around contentious societal matters and to facilitate more efficient citizen engagement structures and practices at a systemic level across various policy domains.

Responsible Territorial Policymaking, as envisioned by this project, is a guiding principle, emphasising the importance of considering ethics, societal impact, and sustainability in every decision made by policymakers for communities (Mikic & Grassc, 2002). So, why is involving the public into the policymaking process so important, and what drives us to include the public and the RRI Citizen Review Panel? Applying RRI principles into territorial policies is apparent and well documented as shown below in this paper and the citizens inclusiveness is one of the very basic and important principles (Fraaije & Flipse, 2020).

It all boils down to our commitment to inclusivity—the idea that everyone’s perspective matters when shaping policies that affect our lives (Béland & Katapally, 2018). The RRI Citizen Review Panel plays a pivotal role in this effort, acting as a bridge between decision-makers and the public (Miller et al., 2017). It offers a platform for regular people to share their thoughts, concerns, and fresh ideas in the policymaking process (Moodie et al., 2021), fostering a sense of teamwork (Driskell et al., 2018), shared responsibility, and collective ownership of the policies that shape territorial development (Alaoui & Mamoun, 2021). But the power of involving citizens goes beyond merely having them participate (Alamoudi et al., 2022). Citizens often bring forth new viewpoints that challenge the status quo and add depth to policy discussions, making public engagement not just desirable but essential (Sánchez-Nielsen et al., 2014). Overall, this article analyses the citizen’s perspectives in the Region of Western Macedonia and Sofia Municipality to see whether they contribute to making territorial policies more responsible. In doing so, it invites readers to navigate the complex terrain surrounding public engagement and explore the political landscape that shapes this critical process (Khazraee, 2019). In this article we are dealing with the following research questions: how can the RRI Citizen Review Panel support Responsible Territorial Policymaking?

Literature review

The topic of participation has gained prominence in global discussions since the 1990s, specifically in relation to the crucial matter of the interplay between representative democracy and deliberative democracy, as well as the potential for citizens to gain empowerment by engaging in policymaking processes (Dalton, 2008; Moini, 2012; Rondinelli, 2006). There are numerous terms that encompass the concept of citizen engagement (Adler & Goggin, 2005; Nabatchi & Amsler, 2014; Rowe & Frewer, 2000). Public engagement, public participation, stakeholder involvement, co-creation, co-production, political participation, civic engagement, deliberative democracy, and participatory democracy are some examples (Arnstein, 1969; Carpini et al., 2004; Creighton, 2005; Nabatchi & Amsler, 2014; Nabatchi & Leighninger, 2015; Rowe & Frewer, 2000).

One can establish a differentiation in terminology pertaining to the conduct of individuals by placing emphasis on the identity of the actors involved. For instance, the term “public” in the context of public participation refers to the broader populace residing within a specific geographical region. Similarly, the term “stakeholders” in relation to stakeholder involvement denotes individuals or groups who possess a vested interest in governmental matters. In the case of co-creation, the term “citizens” pertains to problem solvers or expert stakeholders who contribute to resolving intricate issues. Lastly, within the realm of political participation, the term “citizens” refers to eligible voters.

The concept of citizen engagement we follow in this article, as defined in the literature, refers to a participatory process wherein the public actively participates in policy development and is involved in various activities and services provided by the local government (Ekman & Amnå, 2012; Gawlik et al., 2018; Nabatchi & Amsler, 2014; Siebers, 2020). Numerous studies have demonstrated the manifold advantages associated with citizen engagement. These include the utilisation of local knowledge and innovation, the mitigation or prevention of conflicts, the enhancement of social inclusion or cohesion, the mobilisation of fresh resources such as voluntary labour, the reduction of transaction costs, and the cultivation of trust and social capital (Smith, 2009).

This study presents a comprehensive examination of the utilisation of citizens’ panels, as a deliberative approach (Elstub & Escobar, 2019), advocating for the meaningful engagement of citizens in deliberations about policy processes. It expands upon the findings made through empirical research on RRI Citizen Review Panels in two different territories, the Region of Western Macedonia (Greece), and the Municipality of Sofia (Bulgaria). The two territories being examined encompass a diverse array of policies implemented at either the regional or the municipal levels, which have a significant impact on the day-to-day experiences of the citizens. In order to gather a comprehensive understanding of the effects of the proposed policy transformation, this paper is based on a study on the involvement of citizens in territorial policymaking. The study seeks to analyse the citizen input to determine its impact on the development of responsible territorial policies, particularly its effectiveness in shaping or improving policies that embrace an RRI approach. By adopting this approach, it can be guaranteed that the planned change involves not just policymakers, stakeholders, and specialists, but also includes the representation of citizens in the policy discourse, allowing them to contribute their values, opinions, implicit knowledge, and personal experiences. In addition, engaging citizens in the consultation process offers the potential to gather a perspective rooted in common sense regarding the specific policy domain under consideration, devoid of any inherent biases stemming from individual interests.

Responsible research and innovation

RRI has gained significant importance in EU policy during the past decade. It has been introduced as a framework to link research and innovation with wider social values and to support both concepts on a regional level (European Commission, 2018). This framework encourages a wide range of societal participants, such as researchers, policymakers, and business organisations, to collaborate in order to enhance the congruence of their actions, procedures, and results with the values, requirements, and anticipations of society. This collaboration aims to foster responsible, intelligent, sustainable, and inclusive regional development. Several definitions of RRI have been suggested since then (Burget et al., 2017), however the most frequently used one remains the definition put forth by von Schomberg (2011):

[RRI is] ‘a transparent, interactive process by which societal actors and innovators become mutually responsive to each other with a view to the (ethical) acceptability, sustainability and societal desirability of the innovation process and its marketable products (in order to allow a proper embedding of scientific and technological advances in our society)’.

The notion of RRI was initially associated with six important elements: (1) Public Engagement (PE), (2) Gender Equality (GE), (3) Science Literacy and Science Education (SE), (4) Ethics (ETH), (5) Open Access (OA), and (6) Governance (GOV) (Rome Declaration, 2014), but then reduced to five due to the excessive stringency in implementing the element of "governance" (Owen et al., 2021; Rip, 2016). Nevertheless, the introduction of the RRI keys was met with much scepticism among researchers, as noted by Saille (2015), who emphasised that:

“….without greater citizen involvement in both political and scientific governance, RRI will remain a vague set of hopeful ‘Keys’ which must be incorporated into funding proposals, but do not significantly influence the norms, discourses and functions of other institutions in the EU”.

Recent studies have emphasised the territorial potential of RRI and its incorporation into development policies, offering a framework to support territories, as discussed by Panciroli et al. (2020). The notion of RRI in territorial policymaking is derived from the idea of anticipatory governance, as proposed by Guston and Sarewitz in 2002. It involves the development of future scenarios and effective alternatives to mitigate potential dangers in decision-making processes, as discussed by Quay in 2010. The involvement of diverse stakeholders, including the public, aims to enhance the probability of societal benefits and mitigate any adverse outcomes resulting from governmental actions or inactions (Burget et al., 2017). In this context, the so-called AIRR dimensions of governance (Stilgoe et al., 2013), those of Anticipation, Inclusiveness, Reflexivity and Responsiveness, have gained particular importance as a base for enhancement of the RRI framework at territorial level. To build on both notions of RRI, the present paper applies both the five RRI keys and the four AIRR dimensions as a framework in the analysis of the citizens review of territorial policies.

Deliberative techniques and the need for the RRI Citizen Review Panel

In recent years, there has been a notable evolution in the conventional methods employed to establish a connection between citizens and decision-makers, as well as in the practices associated with policymaking. There has been a transition from citizens as passive consumers to active participants in the policymaking process. The shift in perspective regarding citizen interaction as a crucial aspect of knowledge acquisition enhances the legitimacy, transparency, and equity of policy development (Sánchez-Nielsen et al., 2014). Citizen engagement may enrich policymaking by adding a variety of perspectives and opinions to policy decisions. It can also lead to better informed policies and programmes linked with people’s needs by putting the experience of those affected by the outcome at the centre of deliberations.

Deliberative approaches to engagement are characterised by a process of reasoning, where participants are provided with the chance to contemplate, question, discuss, and think (Parkinson, 2004). Several deliberative approaches to citizen engagement have been discussed in the theoretical literature. These include Consensus Conferences (Dryzek & Tucker, 2008; Grundahl, 1995; Hendriks, 2005), Citizens’ Juries (Crosby, 1995; Parkinson, 2004; Pickard, 1998; Smith & Wales, 2000), Citizens Panels (Crosby et al., 1986), Planning Cells (Dienel, 1999; Hendriks, 2005), and Deliberative Polling (Fishkin, 1991). Some others, like World Café (Brown & Duguid, 2001), are predominantly excluded from academic peer-review and evaluation. Certain scholars in the field of deliberative methodology propose the integration of multiple methods with the intention of mitigating the limitations of one method by leveraging the advantages of another (Carson & Hartz-Karp, 2005, p.121). Conversely, there are those who contend that the practical implementation of these methods, as initially outlined, is not feasible in real-life contexts (Pickard, 1998). The successful implementation of citizen engagement techniques necessitates a capable and knowledgeable team equipped with the necessary abilities (Irvin & Stansbury, 2004; Lawrence & Deagen, 2001).

Consensus conferences

A consensus conference is a public meeting where stakeholders, including experts and citizens, collaborate to assess an issue, find common grounds, and provide consensus-based input. This process typically spans from two to four days and involves panellists asking questions, audience participation, and the creation of a joint position statement that represents the collective decision of all participants. The conference aims to empower community members, bridge the gap between experts and laypeople, and facilitate diverse perspectives in public debate. However, it comes with challenges like high setup costs, the need for strict implementation rules, and the requirement for an effective facilitator. Successful planning includes forming an advisory committee, advertising the meeting widely, selecting representative citizen panels, hiring professional facilitators, and choosing experts who can present opposing views effectively. The consensus conference process requires significant resources, including staffing, materials, and planning time, and is most suitable for higher levels of participation and collaboration among stakeholders. While it offers advantages such as giving communities a voice and facilitating inclusive discussions, its formal nature may limit impartiality, and strict adherence to rules is crucial for success (Grundahl, 1995).

While Consensus Conferences may enhance individuals’ perceived capacity to engage in technoscientific matters, their impact on policy or policymakers may be minimal or non-existent (Powell & Kleinman, 2008). While Consensus Conferences can satisfy inclusivity and evidential complexity, they might struggle with achieving true consensus or constraint (Stegenga, 2016). The consensus conference is an effective approach for offering policymakers policy proposals. However, it is highly time-consuming and resource-intensive.

Citizen juries

Citizen juries are a participatory decision-making tool where a representative sample of citizens, usually selected randomly, is briefed on a specific issue or project affecting the community. They are presented with various alternatives and, after deliberation, produce a decision, often in report form, which may include recommendations. This method aims to involve the broader community in a democratic and representative manner, ensuring that decisions are transparent, non-aligned, and incorporate diverse perspectives (Crosby, 1995). The process bears resemblance to Peter Dienel’s Planning Cells, which were devised two years prior to the Citizens Jury process. The jurors are remunerated for their presence in hearings, during which they acquire knowledge about the subject matter. Hearings normally span a duration of four to five consecutive days. If a two-tiered system is implemented, wherein many regional juries convene and thereafter choose one to three members to participate in subsequent joint meetings, the participants in the second conference may allocate a maximum of 12 days for the hearings. While longer meetings may enhance comprehension, there is a possibility that the increased knowledge could be counterbalanced by a decrease in the rate of agreement among participants.

While Citizen Juries offer a transparent and credible process that complements other consultation forms, they come with challenges. Ensuring a fair juror selection, clarity on how results will be used, and the time and cost-intensive preparation are some hurdles. The sponsoring agency is typically expected to implement the jury’s decision or provide clear reasons for not doing so. Effective planning, including engaging expert witnesses and ensuring broad community engagement, is crucial for the success of this method.

Planning cells

Developed by Professor Dr. Peter Dienel of the University of Wuppertal, Germany, Planning Cells convene selected public groups to address specific planning or policy challenges. They are optimal for urgent issues with multiple solutions and minimal pre-existing public polarisation. Organisations responsible for relevant planning or policy decisions are the primary users, provided they are receptive to the resulting recommendations (Dienel, 1999).

Typically, 25 randomly selected individuals collaborate for a short period, guided by two process escorts, to devise solutions. They engage with experts and stakeholders, culminating their findings in a citizen report for the commissioning entity. Success factors include viable solution options, participants’ awareness of potential downsides, and the commissioning body’s commitment to the outcomes.

Planning Cells have drawbacks including bureaucracy, inefficiency, and limited transparency. They may not fully represent marginalised groups, resist change, and face budget constraints and political influence. Conflicts of interest and weak enforcement can arise, and developers may resist recommendations. Flexibility in adapting to changing circumstances may also be lacking. Solutions include increasing transparency, community engagement, and enhancing professionalism and independence. Additionally, it is necessary to provide remuneration to individuals involved in a Planning Cell, which should cover both their active involvement and the compensation for any salaries lost during the process. On the one hand, this guarantees that members of a planning cell approach their responsibilities with a high level of dedication and concentrate on achieving the cell’s aim. However, on the other hand, it also results in a significant expenditure of resources (The Community Foundation: Northern Ireland, 2019).

Deliberative polling

Deliberative polling is a method where a representative sample of the population is provided with information about a specific issue, allowing them to deliberate and form a considered opinion. The process begins by selecting participants based on various demographic factors like gender, race, and socio-economic background. Initially, participants express their views through a questionnaire. They then convene for several days, accessing unbiased materials, discussing in facilitated small groups, and interacting with experts. Post-deliberation, the same questionnaire is administered, and the changes in opinions are analysed. These shifts are believed to reflect the broader public’s potential conclusions if similarly informed. While Deliberative Polling often results in significant opinion changes, some shifts might revert over time. The process is sometimes televised, broadening its reach and impact (Fishkin, 1991).

Typically, a Deliberative Poll involves 100 to 600 participants, ensuring a diverse representation. Specialist recruitment organisations often manage participant selection. The process is resource-intensive, both in terms of cost and time. Major expenses include participant recruitment and potential media involvement. The entire process, from setup to conclusion, can span over six months. Strengths of this method include its ability to merge statistical representation with deliberation, enhance public understanding, and showcase the difference between uninformed and informed views (Han, 2023). However, it might not always yield qualitative insights and might rely on media partnerships for broader impact.

Deliberative Polling, though a valuable tool for gauging informed public opinion, has its drawbacks. It can be costly and time-intensive, and there is no guarantee of perfect representation from the selected sample. Participants might drop out between stages, and the opinion shifts observed post-deliberation may not be enduring. The process can be influenced by the quality of briefing materials, dominant voices leading to groupthink, logistical challenges, and potential media biases when sessions are broadcasted. Additionally, its effectiveness might be limited to specific, well-defined issues rather than broader topics (Han, 2023).

Citizen panels

Citizen panels in particular have experienced a process of adaptation and evolution. Developed in the early 1980s by the Centre for New Democratic Processes, they were designed based on the jury system and bear resemblance to a process that was autonomously devised by a group of social scientists from West Germany. Citizens Panels offer a useful tool for engaging citizens in policy decision-making. The modest size of these deliberative exercises allows for efficient deliberation, making them more cost-effective compared to bigger planning cells and consensus conferences. Additionally, their diversity ensures that participating citizens are exposed to a wide spectrum of public experience and opinions (Street et al., 2014).

A Citizen Panel is a representative group of residents, primarily used by statutory agencies to assess public opinions and preferences. These panels, which can range from a few hundred to several thousand members, are typically recruited through random sampling from electoral rolls or postcode files. The goal is to ensure a broad representation, including hard-to-reach groups. Once enrolled, participants engage in various activities, such as surveys, focus groups, and workshops, to provide feedback on local priorities and services. The concept has expanded beyond local settings, with endeavours to establish panels at international levels, like the European Citizens’ Review Panel (Crosby et al., 1986).

The operational costs and time for these panels are moderate, influenced by factors like panel size, consultation frequency, and membership renewal. While they offer several strengths, such as the ability to target specific groups and assess local needs, they also have inherent challenges. These include potential exclusions of certain demographics, the need for significant staff support, and declining survey responses over time. Originating from market research, Citizens Panels have evolved from traditional opinion polls.

Presented below is a table that outlines the described deliberative techniques (Table 1).

Table 1 Deliberative techniques

In summary, the methods described can all facilitate valuable input to various policymaking process, showcasing variations in terms of their distinct objectives, durations, group sizes, and scopes. Consensus Conferences address intricate scientific or technological topics through extended deliberations spanning multiple days, with the aim of achieving consensus following expert presentations. While effective in providing detailed recommendations to policymakers, Consensus Conferences come with high implementation costs. Citizen Juries engage in focused and brief discussions on specific issues, after being presented with various scenarios by experts. Planning Cells are dedicated to urban and regional development supervision, incorporating the perspectives of both experts and citizens. Deliberative Polling is a method that assess the transformative power of informed deliberation on public opinion, involving a large number of people. Citizen Panels, which may have broader scopes and engage thousands of people, can function continuously to gather public feedback on a wide range of themes, including activities such as surveys, focus groups, and workshops. For most of the methods, participants deliberate during the course of multiple days to reach a solution to the policy issue. These unique features of the methods make them applicable in different policy context and suitable to solve specific objectives. In the next section, the method for the RRI Citizen Review Panel will be explained, as well as the specific needs surrounding the development of this particular method.

Research context and method

This research was carried out as part of the RRI-LEADERS project, which explores the application and sustainability of the RRI-AIRR framework within territorial policymaking (RRI-LEADERS, 2023a). Responsible Territorial Policymaking is a vision developed within this project. The vision asserts that for sustainable development and enhanced resilience at the territorial level, policymaking must not only encompass relevant scientific, technological, innovative, economic, and environmental factors but also extend to the consideration of societal and ethical issues, aligning with societal values and leveraging the collective knowledge of all territorial stakeholders (RRI-LEADERS, 2023b). The RRI-AIRR framework is a core element of this vision, being the guiding principles of a responsible policymaking process.

To align with the needs and values of society in the development of territorial action plans, the RRI-LEADERS project has engaged a wide range of societal actors, including policymakers, research and education communities, business and industry, as well as civil society. The RRI Citizen Review Panel is a method developed to facilitate the review and validation of the territorial action plans by citizens. The present paper analyses the citizens’ review of two territorial action plans, developed within the Region of Western Macedonia (Greece) and Sofia Municipality (Bulgaria). The two territories face different territorial challenges as well as a diverse range of opportunities and implications for RRI:

Similar to many other major cities in the present day, Sofia is confronted with significant economic and social challenges in terms of urban development. These challenges encompass issues such as air pollution, climate change, sustainable urban mobility, an aging population, and a dynamic process of in- and outmigration. Addressing each of these challenges necessitates the allocation of institutional, human, and financial resources to effectively tackle them. In the context of the RRI-LEADERS project, Sofia Municipality has chosen four policy areas for analysis and evaluation in order to explore the potential incorporation of the RRI-framework in policy formulation and implementation. These policies encompass (1) support for innovation, (2) digital transition and new skills, (3) youth employment and entrepreneurship, and (4) sustainable urban development.

Western Macedonia’s mega-task of energy transition comes at a time when the region is economically vulnerable, suffering from significant levels of unemployment, heavy reliance on the lignite sector, and structural weaknesses in its governance model. The regional development model should be based on a clean energy strategy, exploiting the technical skill base of the region’s workforce and developing a flexible, inclusive and participative governance framework. In the context of the RRI-LEADERS project, the region’s main policy objective was to address the aforementioned challenges in an inclusive manner. That is, the development of a clean energy transition strategy based on stakeholder engagement, efficient territorial governance and the development of a methodology aiming to a smooth and innovative transition towards an alternative development ‘paradigm’.

The RRI Citizen Review Panel method

In the absence of an applicable method that could answer to the specific needs of the project in facilitating citizens’ review of territorial policies, the RRI Citizen Review Panel was developed. The method aims at accommodating a number of practical concerns. First, the method had to be implementable in a territorial context, creating the need for reducing implementation costs and efforts. As a result, the method had to be a one-day event, involving a manageable amount of approximately 25 citizens.

Secondly, the actions presented to citizens were a result of a structured methodology foreseen in the RRI-LEADERS projects’ previous activities, involving policymakers, experts, and stakeholders in the development of the actions. The method therefore had to be a consultation process where citizens were asked to review and validate existing policies.

Thirdly, the method had to accommodate several policy foci and the need to present the reflection of the RRI-AIRR framework in the whole process. To guide this process, the RRI Citizen Review Panel builds on the piloted experience of three multi-criteria assessment tools. These tools were developed and piloted to support decision-making processes in different contexts, allowing a better understanding of strengths and weaknesses as well as risks and opportunities of the policy under scrutiny. In 2013, the DESSI method was launched to improve the decision-making process on security investments (Wächter & Peissl, 2013). By use of a multi-criteria assessment process, the DESSI tool aimed at introducing the many and complex societal dimensions of security investments, including individual rights, and other significant social, political, and economic implications. To assess the investments, participants were asked to discuss a total of 41 criteria, scoring each criterion on a 4-point scale from disagreement to strong agreement. Building on the experiences of DESSI, the AI 360-tool was developed in 2019, adapting the DESSI method to the specificities of AI as a technological and societal phenomenon (Bitsch et al., 2019). In this case, participants rated a set of 30 criteria on a 5-point scale. A colour code was used to visualise the ratings, ranging from green (very good impact) over yellow (neutral) to red (very negative impact). Finally, in 2020, the SAFIRE tool was developed to provide decision-makers in the European Commission with a practical foresight tool to support policymaking on R&I, guiding the reflections towards overarching EU values such as social equity, ecological sustainability, and economic growth (Focken et al., 2020). Using again a 5-point scale that translated into a similar colour coding as in the AI 360-tool, participants were asked to individually rate the performance of the policy proposal based on 41 predefined criteria. As with the two other methods, the participants deliberated about the strengths and weaknesses of the policy based in the ratings and ended by discussing and identifying possible improvements to the policy.

The RRI Citizen Review Panel applies several of the features from these multi-criteria assessment tools, including an individual colour-coded assessment of different criteria, group deliberation based on the assessments, and making suggestions for improvements. However, since the three decision-support tools were mainly targeted at professionals, including stakeholders, experts, and politicians, simplifications of the criteria and rating system were needed to make the process accessible for citizens with no professional expertise in RRI.

Based on the practical concerns and experiences with the multi-criteria assessment tools, the RRI Citizen Review Panel was developed, consisting of the following four sessions (RRI-LEADERS, 2022):

• 1) Acceptability of actions: In the first session, citizens were asked to assess the acceptability of all actions in the territorial action plan based on three predefined criteria. The aim was to identify which actions and aspects were most important for the citizens to work on improving. The citizens were organised in smaller groups where they deliberated on the actions and stated their individual opinion and reasons for not accepting the proposed actions. When the group had assessed all actions, the citizens looked at the assessments and decided which actions they would like to improve.

• 2) Improvement of actions: The second session was the main session of the panel, seeking to understand how the proposed actions could be adjusted to better incorporate the concerns, values, opinions, and lived experiences of the citizens. The groups went through the chosen actions one by one to make suggestions on how to improve the actions as well as suggest additional actions. The citizens finished the work on each action by individually assessing the action anew, based on the suggested changes.

• 3) Writing of statement: In the third session, each group wrote a statement considering whether the actions proposed in the action plan adequately addressed the overall challenges in the policy area, giving the citizens an opportunity to review the action plan as a whole.

• 4) Prioritisation: In the final session, the citizens prioritised between all the original actions and suggestions for new actions based on the following question: which actions do you find to be the most important? The purpose was to provide policymakers with information on the importance ascribed to the actions by the citizens.

During May 2023, Sofia and Western Macedonia each conducted an RRI Citizen Review Panel with the participation of 27 and 23 citizens, respectively. The experiences in the two territories demonstrate how structured input from the citizens can support Responsible Territorial Policymaking aimed at enabling deep societal transitions. In what follows, the analysis focuses on the input generated by the citizens in the first two sessions, more specifically, the comments and suggestions noted down by the citizens while assessing the acceptability and improving the actions.

Analysis

The analysis examines how RRI-AIRR is reflected in the citizens’ review of territorial policies. The analysis first looks at how RRI-AIRR is reflected in the actions proposed by the territorial stakeholders, ranged according to the assessment made by the citizens. Focusing on some of the least acceptable actions, the analysis then sets out to explore whether the citizens’ perspectives add some of the missing elements of RRI-AIRR to the territorial policies.

Region of Western Macedonia

In Table 2 depicted below, the level of acceptability of the actions assessed in the RRI Citizen Review Panel of the Region of Western Macedonia, as well as the reflection of the RRI framework through its keys/dimensions, are shown. The table depicts how citizens prioritised the actions in terms of the level of acceptance of each action and also what is the reflection level of RRI for each distinct action according to the authors’ analysis. A full list of actions can be seen in Appendix A. The general picture is that the most reflected RRI element is science education, with anticipation and public engagement to follow closely. Indeed, the actual concept of RRI contains research and innovation as its core ingredients and this is how it was primarily perceived by the stakeholders who initiated the actions and the citizens who assessed them. As a result, most of the proposed actions were perceived as science-related by the citizens, reflecting science education. Anticipation in policymaking is a crucial issue for all societal actors including citizens, since energy transition has already brought harsh conditions for the local economy and decreased social well-being. Policy actions have been formatted, promoting the dimension of anticipation, specifically in terms of preventing negative future consequences on the local economy and society. Citizens seem to confirm the anticipatory aspect in many of the actions. Public engagement in policymaking is the stake for the effective implementation of the energy transition, as the rest of the quadruple helix actors (businesses, citizens, academia) often feel neglected from the decision-making segment, which is solely practised by the policymakers’ helix. The experts/stakeholder’s duo proposed policy actions that aim to engage the rest of the helices in the core of the policymaking, and mostly the citizens’ helix. That is so, in terms of both the final beneficiaries of the actions as well as the implementors.

Table 2 Correlation between citizens’ assessment and reflected RRI-AIRR framework in actions, Western Macedonia

Considering the most acceptable actions, Action 2.2. The establishment of an innovation zone in the Region of Western Macedonia, has been rated by the citizens as the most acceptable one. In the framework of Just Development Transition Programme (JDTP), plans for the establishment of an innovation zone in the Region of Western Macedonia are already under development (Zervas et al., 2021). The citizens of the region underline the necessity for the effective implementation of projects related to responsibility in governance for the future development prospects of the region. A poly-thematic innovation zone could significantly increase the scientific performance in the region and also supply the regional entrepreneurial system with skilled workers and innovative applications. In terms of RRI reflection into the action, an obvious selection is the science education key as a crucial pillar of the content of an innovation zone. The analysis from Table 2 entails that anticipation is a dimension reflected in the action, since an innovation zone may absorb in the most effective way the scientific capital of the region and direct innovation solutions towards the facilitation of energy transition. Moreover, the establishment of an innovation zone is perceived by the citizens as a major determinant of the regional governance’s ability to respond to emerging needs, specifically regarding the most effective exploitation of the local scientific personnel and the technology needs of the investments planned under the JTDP framework.

Action 3.3, which foresees the creation of an MA-level course in environmental sustainability and circular economy has received the second highest number of positive votes. Citizens consider a specialised university course as a catalyst in the creation of a knowledge capital that is specialised in the distinctive needs of the energy transition framework and all the dimensions that this contains. These skills cannot be exclusively economy-oriented, industry-oriented, or environment-oriented, but should encompass wider areas of knowledge and fields of circular economy. This action again is primarily focused on science education, as it mainly includes scientific characteristics. Indeed, the proposed course should be based on the latest innovative scientific developments of the industry/economy/environment triptych, fulfilling the needs of effective energy transition. Furthermore, open access is also reflected in the action. That is so mainly due to the specific provisions that were set by the citizens during the RRI Citizen Review Panel. For instance, the citizens proposed that the university course would be offered for free to social groups that have been negatively affected by the energy transition (former miners and PPC workers) and vulnerable groups such as the unemployed. This provision makes the course literately accessible to everyone and is an important indicator of inclusiveness, allowing vulnerable and most affected-by-transition groups to get primarily benefitted. Moreover, the introduction of digital means such as online platforms and accessible scientific portals and databases further increases the inclusion of the RRI key of open access.

On the contrary, the most rejected actions in terms of acceptability and RRI/AIRR reflection level, are the ones that the citizens considered to have the most negative effect for the regional economy for the medium and long term. As depicted in Table 2, action 3.4—the initiation of an environmental tariff—has received the smallest number of positive votes. In fact, citizens seem to be very hesitant to accept more financial burden for potential and current investments, implying that this is a highly anti-investment measure, despite the fact that it appears to be environmentally friendly. This case constitutes a clear indication of how “responsibility” is accrued from the citizen’s responses, often outside the strict RRI framework. For the people, apart from the reflection of the keys and the dimensions, any action should lead to a clear social benefit in order to be considered ‘responsible’, and the type of ‘responsibility’ has to be primarily attributed into financial terms. For this particular action, no RRI key or AIRR dimension is reflected, even neglecting the clear “green” dimension of the action.

Improvement of actions

The objective of this section is to uncover the “sides of responsibility” according to the citizens’ perceptions by presenting the actions that need improvement and the specific improvement proposals by the citizens. Table 3 depicts the citizens’ perspectives on one of the least acceptable actions of the RRI Citizen Review Panel.

Table 3 Reflected RRI-AIRR framework in citizens’ perspectives, Western Macedonia action 3.1

Action 3.1—the implementation of a regional efficiency programme aiming at the improvement of the current energy balance in the region, foresees the conduction of a regional foresight study that will include current and future energy needs. The citizens initially objected the action as too generic. They also considered that the time schedule should be limited, indicating an indirect expression of anticipation. Citizens in the region have long experienced promises by the policymaking bodies for improvement of their way of living, which often are never actually fulfilled. By asking for a strict time schedule, they express their intention to demand works rather than studies and plans, anticipating a quick implementation of any study that must shortly be transformed to an actual project. Furthermore, some citizens argued that there has to be an increased level of responsiveness, by transforming the programme in the proposed action to a different programme that should aim at supporting the existing energy communities in the region or contributing to the development of new ones. Citizens at this point respond to the imperative needs of the energy communities in Western Macedonia, stating that energy balance could be achieved by the development of the existing energy communities rather than the conduction of new studies regarding the current and future energy needs. In simple words, they state that if the energy communities are allowed and supported to provide the RES-based energy they can produce, the local communities will be greatly benefited and at the same time, the energy market itself will set the limits of the needs. Moreover, the citizens underline the importance of grounding any action at solid scientific principles, by pointing out the need from the regional university to take over any programme or study that contains scientific information. In fact, citizens in this occasion express their concern in the possibility of politicising science, since energy needs and demands are often involved in private and public interest competitions. In simple words, the citizens state the current and future energy needs of the region should be assessed according to purely scientific principles rather than being decided by financial or political power centres that often promote stances according to their own interests. The last perspective by citizens in action 3.1 assesses the ethics issue in policymaking. Citizens state in this particular case that the policymaking authorities often demand actions and measures by the citizens that have a clearly negative impact on their financial conditions. As an example, the citizens agree with the transition to a RES-based energy mix, regardless of their short-term financial interests. However, they demand that the same measures have to be taken also by the public authorities, as a sign of ethics in policy and commitment to the achievement of the policy focus of the region.

Table 4 presents the citizens’ perspectives on another low-rated action, according to their own perceptions. In this case, citizens appear hesitant on the implementation of massive costly infrastructures that have a doubtful effect on the real economy and the wider societal needs. Firstly, it is stated that rather than being general, the infrastructure development should be exclusively directed to specific types of infrastructures that the region is currently lacking, but they also have the potential to operate as investment-boosters, in favour of the local economy. So, they indicate a high level of reflexivity, by directing the infrastructure construction to the build-up of railway network that is absent in the region, the development of large logistic facilities that may empower the industrial capability in the region and also the initiation of hydroplanes. That last proposal is fully innovative, since the region has several lakes dispersed at a large area, which may serve as transportation hubs for citizen and commercial purposes with a minimised level of investment and a maximised social benefit. Furthermore, the citizens state that it should be a matter of ethical integrity of the policymaking side to take over the development cost of all the infrastructures that should have been made in the past. Moreover, specifically in the time of energy transition in which the local economy suffers from major turmoil and the citizens face unemployment at high rates, it would be ethically unacceptable for the central government to pass that cost to the regional and local authorities or the citizens. Finally, the citizens appear to propose a totally different direction for the action, which may explain the low rating it received in the first place. The citizens openly question the real developmental effect of large and costly infrastructures. Indeed, during the previous years, massive infrastructures have been built in the region (extensive road networks). However, the region still suffers from extremely high unemployment rates and brain drain. As a result, the citizens doubt the effect on the economy and the society, and they rather propose that this money should be given directly to the local communities and businesses in order to relief the harsh current economic condition. Actually, the citizens state that a truly inclusive development plan should include the citizens and other regional actors as direct beneficiaries of transition measures rather than indirect beneficiaries through projects implemented by non-regional organisations in the forms of infrastructure works.

Table 4 Reflected RRI-AIRR framework in citizens’ perspectives, Western Macedonia action 1.5

Summarising the citizens’ interventions in the two actions, it is evident that they highlight those specific RRI/AIRR elements that according to their perceptions promote most effectively the social aspect of responsibility. That aspect is perceived by the citizens in primarily financial terms and secondary in social, environmental, ethical, or other perspectives. For the negatively affected citizens of Western Macedonia, every action must eventually lead to the improvement of their daily lives, and the choice of every RRI key or AIRR dimension depends on the level of its potential contribution towards that specific goal. In that direction, we see that keys or dimensions that are not directly related to any financial or developmental effect—e.g. gender equality—have not received any significant voting from the citizens in all 15 actions depicted in Table 2. And this case remained similar in Tables 3 and 4, where citizens do not add perspectives that correspond to keys or dimensions that may have a clear financial or social benefits.

In particular, citizens’ contribution to make an action more “responsible” primarily in sustainability terms by implementing the RRI/AIRR framework, is indicated by the contrast between Tables 2 and 3, 4. While action 1.5 contains only the AIRR dimension of anticipation and the RRI key of open access in its initial form (depicted in Table 2), we see that the citizens add the AIRR dimensions of responsiveness/inclusiveness and the RRI key of ethics in the citizens’ perspectives shown in Table 4. It is apparent that all added RRI elements are connected to their potential financial impact on society, since responsiveness in the citizens’ perspectives (Table 4) may be reflected through the build-up of infrastructures that will strengthen financial development in the region and ethics may be reflected through the funding of the infrastructures exclusively by national authorities, taking this financial burden off the shoulders of the citizens. Similarly, anticipation is reflected in action 3.1 in its initial form (Table 2), but the citizens add perspectives that may transform the action to being responsive, scientifically creative, and even more anticipative, through their four perspectives depicted in Table 3. More specifically, the citizens consider that the programme of energy efficiency has to be developed in a strict timetable (anticipation), it has to support primarily the local energy communities (responsiveness) and the local university should take the leading role in action’s implementation, securing action’s scientific validity (science education) and the local authorities should be the first ones to implement the energy efficiency programme (anticipation).

Overall, it is safe to consider that the citizens added in the RRI framework a level of “social responsibility”, transforming policy actions that initially had a narrow scope of implementation into multi-factorial RRI-based action frameworks that aim at a single objective: to make governance more socially friendly and financially sustainable, maintaining the RRI keys and AIRR dimensions which constitute the basic pillars of the RRI framework.

Sofia municipality

Sofia municipality has selected four policy areas to be examined and assessed through the lens of RRI-AIRR in the project. These areas include support for innovation, digital transition and new skills, youth employment and entrepreneurship, and sustainable urban development. In the RRI Citizen Review Panel, participants were asked to evaluate 15 actions (around four actions per policy area). Approximately one-third of the original actions received citizen support without significant modifications. The RRI Citizen Review Panel primarily focused on sustainable urban development, with new or improved actions proposed in areas such as traffic safety, age-friendly and disability-friendly city infrastructure, communication with citizens, public consultations, and citizen contact points within the administration. These actions aimed to enhance citizen engagement in the policy cycle.

Key issues crucial to territorial policymaking, such as control and monitoring of implementation, were identified as critical across policy fields. In the policy area of support for innovation, the actions related to enhancing the expertise and skills of the administration through regular personnel evaluations, attracting and retaining young professionals for administrative roles, and utilising urban living labs to pilot policies and measures before full implementation, gained the highest level of consensus.

Participants in the RRI Citizen Review Panel also strongly supported the digitalisation of administrative services to improve transparency. The utilisation of digital platforms to enhance public transportation, waste management, and other communal services also emerged as a priority. Citizens emphasised the importance of implementing robust data protection policies and enforcing them in accordance with the law.

Lastly, sustainable urban development was the most extensively discussed policy area during the CRP. Citizens considered the political and institutional commitment of local government officials crucial for achieving sustainable development. This commitment resonated strongly with the specific requirements of city neighbourhoods and concerns related to adequate public services. Additionally, it addressed the imperative of equitable resource allocation to tackle social and economic inequalities within the city. All participants emphasised the necessity for closer collaboration between the administration and local communities, facilitated through the appointment of city representatives within district administrations or even at the level of residential neighbourhoods. Moreover, citizens advocated for more opportunities to actively participate in infrastructure decision-making and regular engagements with the government to discuss policy matters.

Table 5 demonstrates that the most acceptable actions embody all the keys and dimensions of RRI-AIRR. A full list of actions can be seen in Appendix B. An even distribution of the keys is observed, with emphasis on ethics, anticipation, and reflexivity. Ethics contributes a broader perspective to proposed policies and actions by considering their potential social and environmental impact. Ethical aspects in the actions, particularly those on the right-hand side of the table, relate to the outcomes of research and innovation. This ensures informed and balanced decision-making by government and decision-makers, guaranteeing that innovations and policy interventions are driven by concerns for social justice, social well-being, and considerations of ethnic and cultural diversity and intergenerational issues.

Table 5 Correlation between citizens’ assessment and reflected RRI-AIRR framework in actions, Sofia

Open access is another key factor that promotes good governance and policymaking. It has the potential to enhance the governance of research and innovation practices, connecting them closely to transparency and integrity requirements. Additionally, open access policies facilitate result sharing, enabling free access to research and innovation outcomes, knowledge-building, and improving citizens’ critical thinking and assessment capacity when integrating higher-level technology products into everyday life.

Anticipation and responsiveness are strongly emphasised across all policy fields and actions. An anticipatory approach is deemed necessary to enhance policymakers’ ability to diligently carry out risk assessments, identify potential multi-level impacts in advance, coordinate complex transition processes, and respond to them with care for all involved parties.

The most accepted actions, namely the digitalisation of public services and administrative processes (Action 2.3) and the systematic provision of municipal infrastructure as living labs and testbeds for innovation and scaling up solutions to urban issues (Action 4.1), embody all the RRI-AIRR keys and dimensions. This indicates a high level of consensus among policymakers and citizens regarding policy goals and responsibilities for achieving them. Action 2.3 calls for transparent and reflexive governance and administrative practices through digital means, open to the public and responsive through virtual services. Action 4.1 emphasises public engagement and bottom-up initiatives guided by shared scientific and expert capacity in developing solutions to urban challenges. In this case, top-down and bottom-up mechanisms for policy implementation reinforce each other, leading to an effective transformation of existing policy practices. Learning from actors ‘on the ground’ (living labs, citizen co-creation formats) can prompt policy change processes, which are complemented by more efficient governance arrangements (virtual governance modes) to scale up bottom-up solutions and facilitate policy implementation.

Improvement of actions

The objections raised by citizens against certain actions and their considerations of them as less acceptable reflect multiple RRI-AIRR keys. For instance, regarding action 3.3, citizens agreed on the importance of equitable access to resources, education, and employment opportunities for individuals from diverse social groups and genders. However, they believed that additional measures for gender equality were not crucial for implementing the specific policy on youth employment and entrepreneurship. Instead, their objections focused on the need for greater responsiveness and inclusiveness in the policy agenda setting and governance practices, which occasionally displayed institutional bias and discriminatory attitudes.

Other critical remarks highlighted the lack of measures to encourage participation in research and innovation, regardless of gender, ethnicity, or nationality. Citizens suggested incorporating carefully designed mechanisms for public engagement and science education to address this issue. In terms of stakeholder involvement, citizens emphasised the importance of explicitly including vulnerable groups in the formulation of research methodologies that are gender-neutral or take into account gender-specific issues.

Suggestions for improvement included implementing diversity management policies for specific positions within the administration to effectively achieve diversity goals. This would involve anticipation and reflexive governance. Another proposal was to seek guidance from youth organisations to determine effective means of ensuring equality and non-discrimination in accessing funding and employment opportunities. By doing so, citizens aimed to legitimise policy measures and invite expert input to ensure the relevance of strategies and policies for vulnerable groups, promoting inclusiveness and responsiveness. Additionally, the proposed course of action aimed to increase accountability among those involved in designing public funding schemes for research and innovation, ensuring their inclusivity of different social groups (Table 6).

Table 6 Reflected RRI-AIRR framework in citizens’ perspectives, Sofia action 3.3

Regarding action 3.1, citizens expressed concerns and objections related to reflexivity and responsiveness, as well as the need for greater transparency in the establishment and management of municipal funding programmes. Citizens indicated their support for more active dialogue between policymakers and local communities to identify critical needs and develop appropriate solutions collaboratively. Participants in the panel discussions agreed that public institutions should define and share their responsibilities for providing high-quality education to the future workforce. The lack of responsiveness and anticipation in designing municipal strategies for youth employment was also highlighted. There was a clear emphasis on formulating and implementing actions that have a high social return on investment, demonstrating a concern for the future of society and highlighting the role of anticipatory governance (Table 7).

Table 7 Reflected RRI-AIRR framework in citizens’ perspectives, Sofia action 3.1

The discussions further emphasised the importance of all stakeholders, including local government, education institutions, and businesses, sharing their resources and capabilities to create education and employment opportunities that yield tangible results for the younger generation. This highlights the significance of accountability and sustainability in citizens’ perspectives, as they care about the future. Intergenerational issues, such as the equitable distribution of resources between younger and older citizens, were also addressed, indicating that citizens consider responsibility and social justice in education governance as crucial for maintaining public trust and social cohesion.

In conclusion, the RRI Citizen Review Panel revealed that citizens’ concerns may not always align with policymakers’ understanding of responsive and accountable governance. Differences in perceptions and knowledge of the policymaking process can make it challenging to create shared agendas to address societal challenges. The deliberations in the RRI Citizen Review Panel in Sofia showed that non-expert citizens find it difficult to fully grasp deep transformative changes due to their scale, multi-level impacts, diversity of policy instruments, and complexity of interventions. Urban development issues are more easily comprehensible and deliberated upon when broken down into concrete and visible results at the local and sub-local levels, which can be endorsed by non-institutionalised citizens. Therefore, elements of the RRI-AIRR framework, particularly the notion of responsibility, are broadly understood as the need for transparency, accountability, sound financial management, and consideration of the needs of vulnerable groups. Digital means of communication and services are seen as instrumental in enhancing transparency, enabling efficient citizen control and monitoring of policy implementation, and ultimately promoting future-oriented, anticipatory, and responsible governance.

The fact that 12 out of 15 actions were considered acceptable indicates a relatively high level of consensus and agreement between policymakers, stakeholders, and citizens. The opportunity for citizens to fine-tune the broader policy picture and provide details for corrective actions proved to be a successful approach in promoting changes in the policymaking process at the system level.

Discussion

The interplay between responsibility and acceptability

The analysis examined the correlation between citizens’ assessments of actions and the presence of RRI keys and AIRR dimensions in those actions (see Tables 2 and 5). The primary goal was to determine whether, despite being unaware of the RRI-AIRR framework, citizens would perceive the most acceptable actions as the most responsible. In both territories, a subtle trend emerges, indicating that the most acceptable actions indeed encompass more RRI keys and AIRR dimensions compared to the least acceptable ones. In Western Macedonia, the right half of Table 1 contains 80% more RRI keys and AIRR dimensions compared to the left half. Similarly, in Sofia, the most acceptable half of actions in Table 4 contain 32% more RRI keys and AIRR dimensions. Comparing the two most acceptable actions in Sofia to the two least acceptable actions, the tendency is even stronger, with the most acceptable actions containing more than twice as many RRI keys and AIRR dimensions as the least acceptable actions.

There is however a vast inconsistency in the correlation between the acceptability of actions and the RRI-AIRR elements reflected in the actions. Some of the less acceptable actions contain more RRI-AIRR elements than certain more acceptable actions, complicating the conclusion that citizens unequivocally consider the most responsible actions as the most acceptable. These incongruities open avenues for further investigation. If it is the case that citizens intuitively assess actions based on a concept of responsibility, this emphasises the relevance of public engagement activities such as the RRI Citizen Review Panel in Responsible Territorial Policymaking. It also indicates that the RRI-AIRR framework might be a good tool for framing responsibility in territorial policymaking, emphasising values that resonate with citizens.

Examining the specific RRI-AIRR elements in Tables 2 and 5, another noteworthy trend appears in both territories. In Western Macedonia, the five most acceptable actions all contain science education which is in stark contrast to the least acceptable actions that do not contain this RRI key. The great representation of science education in the most acceptable actions is connected to the policy area of energy transition, involving actions of re- and upskilling of citizens, establishment of an innovation zone for the regional science and businesses environment, and new university courses specialised to build knowledge capital to meet the needs of the energy transition. The citizens support the need for science education in the region as the most effective mean to ensure sustainable development according to a responsible and societal-friendly manner. This is reflected, not only in the assessments made by the citizens, but also in the citizens’ discussions, underlining the need for such actions.

In Sofia, there is a similar trend for public engagement. The five most acceptable actions all contain this key, whereas it is absent in the least acceptable actions. The most acceptable actions span across all four policy areas and include actions such as providing training and access to digital services for vulnerable groups, providing mentoring programmes and internships for young people at Sofia Municipality, and engaging citizens in selecting public sector initiatives as well as in the city’s’ social and technological innovation. That public engagement is important for the citizens in Sofia is evident in their discussions of some of the least acceptable actions, where they stress the need for citizen engagement mechanisms to encourage people to engage in research and innovation and for more active dialogue between policymakers and local communities.

The representation of specific RRI keys in the most acceptable actions in the two territories indicate that some aspects of responsibility might be more important to citizens than others, differing across territorial context and policy areas. On the contrary, some RRI keys and AIRR dimensions appear in both the least acceptable actions and in the most acceptable. For example, open access and anticipation is reflected in some of the most acceptable actions in Western Macedonia but is also reflected in the least acceptable ones. Similarly for Sofia, there are several examples of RRI keys and AIRR dimensions that appear in both the acceptable and unacceptable actions (such as anticipation, inclusiveness, and responsiveness). Those results indicate that all RRI-AIRR elements do not have the same influence on the acceptability of actions. It might be the case, though, that some RRI keys or AIRR dimensions are more decisive than others. In the case of Western Macedonia and Sofia, it appears as if science education and public engagement, respectively, are particularly important to the citizens. This aspect could be examined further. If policymakers are able to distinguish which elements of responsibility (or combinations of elements in relation to specific challenges) is most important to the citizens, this might help to develop more equitable policies that respond better to the needs and values of the public, helping the process of Responsible Territorial Policymaking. Moreover, based on the RRI Citizen Review Panel results we could argue that the citizens’ choices help to understand better how to improve policy formulation at system level.

The variability of the findings in the two territories also indicates that the RRI Citizen Review Panel contains a flexibility that allows citizens to bring into the discussions their most pressing concerns across different policy areas and territorial contexts. In Western Macedonia, the energy transition has a huge impact in the social and financial situation of the citizens, which is reflected in the review of the citizens. In Sofia, the policy objectives did not directly address issues of social justice and cohesion. In the next section, the benefits and disadvantages of the flexibility of the RRI Citizen Review Panel method will be discussed, more specifically with regard to the assessment criteria.

Efficacy of the assessment criteria in the citizens’ review of territorial policies

The predefined assessment criteria used by the citizens were developed to prompt the citizens to discuss aspects of RRI-AIRR. Looking back at the process of RRI Citizen Review Panel, it is interesting to reflect on whether these principles sufficiently prompted the citizens to make policies more responsible.

Two foundational principles guided the development of the assessment criteria:

• 1) The assessment criteria should be broad enough to allow for the citizens to incorporate their own perceptions of responsible policymaking into the discussions.

• 2) The assessment criteria should not mention RRI-AIRR specifically, to allow for citizens to participate in the discussions without being knowledgeable about the RRI-AIRR framework.

The first assessment criterion asks citizens: “Do you think this action will help reach the objective?”. This was considered a good starting point, since citizens can state whether they think the action fulfils the overall goal by use of their own principles, including their needs, expectations, and personal values. The second assessment criterion asks citizens: “Do you think the possible negative effects of the action are acceptable (e.g. on people, environment, or health)?”. Here, citizens are prompted to think more specifically on the possible consequences of the action. Some examples are given to help citizens think about potential consequences, but the criterion is still open for the citizens to use their own principles. The third criterion asks citizens: “Do you think the action addresses the needs of the citizens and/or intended target group(s)?”. With this question, citizens are encouraged to think about their peers and how the action helps address their needs, still allowing for the citizens to define what is understood by their needs.

One question that comes to mind is whether these broad assessment criteria sufficiently prompted citizens to consider aspects of RRI-AIRR, or if the assessment criteria should instead have targeted certain aspects of the framework. One risk of having the broad assessment criteria is that citizens are missing central aspects of RRI-AIRR in their discussions. In the case of Western Macedonia, gender equality and diversity is an example of an area that was neither included in the actions, nor considered by the citizens. Translating all RRI-AIRR aspects into assessment criteria could create a systematic approach to discussing certain RRI-AIRR aspects that was considered central to the given policy area. A criterion might have asked: “Do you think that the action will affect societal groups differently (in terms of gender, age, race, ethnicity, disability, and socio-economic status)?”. If the citizens had been prompted in this way to discuss gender equality more specifically, the policymakers would have had more knowledge about the citizens’ dispositions on these aspects, helping to make policies respond better to the views of the citizens. On the other hand, the advantage of the broad assessment criteria is that the citizens can use their own principles to assess the actions, which do not necessarily match those of the policymakers. In the analysis, there are several examples of citizens making actions more responsible, pointing to the needs of local communities, the need for better transportation networks, and more exchanges between policymakers and citizens. All of which are suggestions that can be understood within the RRI-AIRR framework, demonstrating that the citizens are thinking in terms of responsibility when reviewing policies. If the assessment criteria had been more specific, they would have pointed the discussions towards certain aspects of RRI-AIRR. This could potentially limit the discussions to these aspects and risk missing out on the most pertinent concerns of the citizens.

Another reason for developing the broad assessment criteria is of a practical concern. The original multi-criteria assessment tools DESSI, AI-360, and SAFIRE all used 30 or more criteria to assess a single policy. In the case of the RRI Citizen Review Panels, the citizens had to assess the acceptability of an entire action plan consisting of 10 to 20 actions, as well as complete three other steps of improving actions, writing a statement, and prioritising actions. This all had to be done in a one-day event to make it applicable in a territorial context. To get enough input on the different actions, all groups were asked to assess all actions. Therefore, to make the process accessible and manageable for the citizens, the number of assessment criteria was reduced. Having assessment criteria that should prompt citizens to discuss all the RRI keys and AIRR dimensions more specifically would have demanded some changes to the process. Some of these changes could have been to select a few actions for assessment, inviting more citizens for the panel to be able to divide actions between groups, or to identify a few RRI keys and AIRR dimensions that were important for the territories to get the citizens’ input on.

In conclusion, the balance between broad and specific assessment criteria in citizen-driven discussions on RRI-AIRR remains a nuanced challenge. While broad criteria encourage diverse perspectives and maintain practical feasibility, a more specific approach could ensure a comprehensive exploration of RRI keys and AIRR dimensions. Future endeavours should consider this balance, adapting criteria to the context and objectives of citizen engagement in responsible policymaking.

Conclusion

In times of rapid changes and high uncertainties, Responsible Territorial Policymaking emerges as a concept to make sustainable and robust decisions. The RRI Citizen Review Panel is a method designed to support this process, facilitating the review and validation by citizens of territorial action plans.

The study showed that the RRI Citizen Review Panel can support territorial policymaking, making policies responsible across a wide range of policy areas. In Western Macedonia, some of the least acceptable actions to the citizens were concerned with the implementation of a regional efficiency programme aiming at the improvement of the current energy balance and creation of infrastructures in the region. In the improvement of these actions, citizens were adding elements of anticipation, inclusiveness, responsiveness, science education, and ethics, expanding the RRI-AIRR framework originally reflected in the action. In particular, the improvements made were centred around making the energy transition more socially and financially sustainable, reflecting the most pressing concerns of the citizens.

In Sofia, some of the least acceptable actions included the gender equality and diversity measures in the Youth Strategy in Sofia and the collaboration with research institutes, businesses and municipal enterprises to support more funding for innovative entrepreneurship education and youth employment. Similar to the case in Western Macedonia, the citizens were expanding the RRI-AIRR framework by adding elements of anticipation, inclusiveness, responsiveness, reflexivity, science education, and ethics. In particular, the citizens emphasised more active dialogue between policymakers, citizens, and experts, to align with societal needs and legitimise policy measures. They also pointed to the need for transparency, accountability, sound financial management, and consideration of the needs of vulnerable groups.

While a general trend suggests that more responsible actions are perceived as more acceptable by the citizens, there are inconsistencies, with some less acceptable actions featuring more RRI-AIRR elements than certain more acceptable ones. This opens avenues for further examination. If citizens inherently evaluate actions through a lens of responsibility, it underscores the importance of public engagement initiatives like the RRI Citizen Review Panel in responsible policymaking. It also underscores the capacity of the RRI-AIRR framework as a valuable tool for framing responsibility in policymaking, emphasising values resonating with citizens.

Specific trends also emerged in each territory, with science education being crucial in Western Macedonia’s most acceptable actions, and public engagement playing a significant role in Sofia’s most acceptable actions. The study suggests that not all RRI-AIRR elements have the same influence on the acceptability of actions, and some elements may be more decisive than others. Policymakers could benefit from identifying the elements most important to citizens to develop policies that better align with public needs and values.

The balance between broad and specific criteria raised questions about whether the former sufficiently prompted citizens to consider RRI-AIRR aspects. While broad criteria allowed citizens to use their own principles, there is a risk of missing central aspects, such as gender equality and diversity, as observed in the case of Western Macedonia. Translating all RRI-AIRR aspects into specific criteria could provide a systematic approach, but it might limit discussions to certain aspects, potentially overlooking citizens’ most pertinent concerns. Moreover, there is a practical concern of managing a large amount of assessment criteria while collecting the citizens’ input on multiple policies. Future endeavours should carefully consider this balance, adapting criteria to the context and objectives of citizen engagement in responsible policymaking.

Appendices

Appendix A: Objectives and actions, Region of Western Macedonia

Policy area: energy transition

OBJECTIVE 1: Putting forward an extensive upskilling of the local workforce through the creation of efficient, decentralised and innovative procedures, where the local research institutions will play a primary role	Implementation time frame
Action 1.1: Conduction of a Regional Foresight Study for re-and-upskilling that will set the exact educational and training needs of citizens that have been mostly affected by energy transition	2024–2025
Action 1.2: Establishment of a broad and inclusive upskilling programme that will be mainly focused to the citizens that have been mostly affected by energy transition (miners, electricity production workers, etc.). University of Western Macedonia will be the primary pillar of the action	2024–2025
Action 1.3: Integration of digital transformation infrastructure for easing and supporting effective upskilling that will be directed to all social backgrounds and offer high tech training solutions. This will include high-speed networking, user-friendly distance-learning platforms, scientific database for energy and other subjects, etc.	2024–2027
Action 1.4: Creation of innovative energy-related upskilling entities such as a Hydrogen University Course	2025–2027
Action 1.5: Extensive creation of infrastructures in the region	2024–2027
Action 1.6: Installation of digital infrastructure in the Regional Authority	2024–2027
Action 1.7: Creation of an Industrial and alternative tourism plan	2026–2027

OBJECTIVE 2: Creation of an entrepreneurial ecosystem in the Region of Western Macedonia	Implementation time Frame
Action 2.1: Initiation of an Action Plan for entrepreneurial growth in the region by the regional authority, the municipalities, the university and the local professional chambers. The plan will foresee specific incentives and funding will accrue from the regional authority / municipalities	2023–2025
Action 2.2: Establishment of an innovation zone	2023–2025
Action 2.3: Creation of a spatial plan that will clearly foresee land usage and distinguish areas for major investments, RES infrastructures, agricultural activities and other usages	2024–2025

OBJECTIVE 3: Development of a regional energy efficiency model with the inclusion of actions towards the achievement of environmental sustainability	Implementation time Frame
Action 3.1: Programme for achieving regional energy efficiency on an industrial and consumption level in connection to the Just Development Transition Programme. The programme will include a foresight study for current and future energy needs per municipality in the region, taking into consideration major investment plans	2024–2026
Action 3.2: Setting up a holistic action plan for environmental restoration of ex-mine lands	2024–2027
Action 3.3: Create an “Environmental sustainability and circular economy” university master-degree course	2025–2026
Action 3.4: Initiate an “environmental tariff” to non-green investments and environmental activities that will be exclusively used for environmental restoration purposes at a regional level	2024–2027
Action 3.5: Creation of a Waste Management Plan and spatial planning of waste storage	2025–2027

Appendix B: Objectives and actions, Sofia Municipality

Policy area: Support for innovation

Objective 1: Build and brand Sofia as an attractive location for top international researchers and research & innovation units of international corporations		Implementation time frame
Action 1.1	Establish a unit in Sofia Municipality to coordinate the collaboration with academia, business associations and NGOs in defining the objectives and impact pathways of innovation-driven city policies	2025
Action 1.2	Build strategic partnerships with European research and innovation and technology centres to support the recruitment, retention, and development of key research staff of local innovation hubs/centres	2027
Action 1.3	Run annual open competitions in which citizens select public sector initiatives to be implemented [based on the model of the annual initiative Sofia Chooses]	2025

Policy area: Digital transition and new skills

Objective 2: Sofia adopts regulatory framework with standards for digital rights, data security, transparency, and trustworthy technological solutions, applicable to research, innovation and business stakeholders and IT entrepreneurs		Implementation time frame
Action 2.1	Increase the capacity and digital competencies of the administration; train and retain local talent (within the administration)	2025
Action 2.2	Provide information, training, and access to digital devices and digital services to vulnerable target groups	2027
Action 2.3	Digitalisation of administrative processes and services. Establishment of a virtual office and information desk of Sofia Municipality	2024
Action 2.4	Adopt a new open data policy to optimise the dissemination of information to the public as well as the internal communication within the administration by 2025	2025

Policy area: Youth employment and entrepreneurship

Objective 3: The higher education system can impart strong basic skills as the foundation for teaching work-relevant capabilities at all qualification levels to benefit city areas of economic leadership by 2030		Implementation time frame
Action 3.1	Strengthen the collaboration with research institutes, businesses and municipal enterprises to fund innovative entrepreneurship education and youth employment	2025
Action 3.2	Establish a digital and physical space for start-ups in Sofia—development of new or customisation of an existing e-platform for start-ups and scale-ups	2027
Action 3.3	Include gender equality and diversity measures in the Youth Strategy 2027	2024
Action 3.4	Enable young people to quickly assume their first responsibilities through mentoring programmes and internships at Sofia Municipality	2025

Policy area: Sustainable urban development

Objective 4: Smart and sustainable urban solutions in Sofia Municipality comprise technological, organisational and social innovations, which are developed in an integral and interdisciplinary manner across different sectors and industries and always involve the population		Implementation time frame
Action 4.1	Apply the living lab approach to experiment and scale out/up innovations (social, technological) throughout the city	2025
Action 4.2	Develop continued training programmes for the municipal administration by 2025	2027
Action 4.3	Adopt & follow a unified model for data collection, sharing, storage, and open data	2024
Action 4.4	Make ex ante social and economic impact assessment of policies mandatory (in addition to ecological impact assessment) by 2025	2025