Phase 2: Deepen understanding - tasks

The tasks in this phase are mainly focused on gathering and critically evaluating data and evidence to deepen our understanding of the issue.

Task 4 of Phase 1 involved a preliminary assessment of available data. This was to help understand the origins of the policy issue and to refine the policy question.

During this stage of the process, we seek to “dive deeper” into the issue. To do this, we must first identify if there is additional data that we need to fully inform our understanding of the issue and potential options.

We need to identify and source data to enable us to understand:

• the scope and scale of the issue
• how other States addressed the same or a similar issue
• the impact of previous initiatives as well as the impact of policies pursued in other countries
• the reasons why previous policy initiatives or those in other countries either succeeded or did not succeed
• the preferred or recommended approaches of bodies such as the EU Commission and the OECD, and
• the costs of various policy options, as well as the potential impact and benefits of each option

In many cases, you may already have information available within your department or agency. However, it is always important to look outside your organisation to examine what other data and evidence might be useful. Useful sources of information, across a range of topics, include the Central Statistics Office (CSO), the Economic and Social Research Institute (ESRI), OECD reports and data, and EU databases and resources, such as Eurostat.

In addition to these general sources of information, there may be topic-specific sources, such as industry or sectoral reports. For example, you might review reports from the Construction Industry Federation (CIF) in the area of construction, or by Teagasc in the area of agriculture.

The role of the policy practitioner is to identify and locate this data and evidence. This leads to the next step – reviewing the data and evidence.

A desktop review involves an in-depth examination of existing data files and reports available from various sources as set out above.

Correlation and causation

It also involves identifying what the data is telling us about the issue in question. This requires a critical examination of the data and evidence, in particular distinguishing between correlation relationships (where two factors appear to be related) and causation relationships (where one factor leads to a change in the other). For example, there could be correlation between incidents of sunburn and ice cream sales, as both may increase on a given day. However, critically, the change in sunburn doesn’t directly cause the increase in ice cream sales. In contrast, there could be a causation relationship between hot weather and ice cream sales, as the hot weather is a factor which could cause an increase in ice-cream sales.

Limitations of data and evidence

Always ask whether there are any limitations to the data and evidence that is available. For example, for statistical reasons, household surveys tend to group people into bands or categories and so any analysis should always look at distributions within these bands.

For example, many analyses of poverty use the EU Survey on Income and Living Conditions (EU-SILC), which is a representative survey carried out annually by the CSO. The EU-SILC reports the proportion of older people whose income is below the poverty threshold. However, further examination of the distribution of income among older people reveals that a significant number of older people have an income very close to the poverty threshold. This means that even a small variation in income can lead to a large number of this group crossing the poverty threshold resulting in a sudden and sharp increase or decrease in poverty levels. Therefore, any policy analysis based on this data should proceed with this in mind.

We also need to contextualise data, particularly where it is used for comparison purposes between regions or countries or over time. This means considering wider social and economic conditions to determine if, and how, other factors may be influencing the data.

Similarly, we often see data being quoted claiming that a certain issue has increased by a large percentage either year-on-year or quarter-on-quarter as a justification for government intervention. However, point in time comparisons can be misleading if there has been a significant change in economic circumstances or there is evidence in seasonality. For example, the impact of COVID-19 between 2020 and 2022 means that comparing current trends with trends in that period can be misleading. Equally, we know employment in the hospitality sector increases during spring and autumn but reduces during winter and spring. It is always important, therefore, to use time periods that are comparable and to adjust for seasonality where relevant. Where possible, it is better to examine medium to long term trends in data.

Is the data valid?

We also need to critically examine the source of the data to ensure the data is objective and reliable. If we are using a report by another organisation, we should check whether it was peer reviewed by a reputable body or academic. If survey results are reported, we should check the sample size and methodology to establish whether the data is statistically valid. It’s also important to know who funded the report to assess whether there could be any bias or vested interests involved.

In short, we need to use best evidence, not any evidence. We must always ask ourselves whether the data we are using is accurate, relevant, and timely.

Where possible, it is better to use existing data, as it can save time and resources. As a first step, the CSO could be approached to assist with new insights from their administrative and survey data holdings.

However, the existing data may be obsolete, incomplete, or unavailable. In addition, it may be missing important contextual or ‘soft’ intelligence.

In this case, we may need to generate new data. We can do this, for example, by commissioning someone to gather new data, similar to the school meals example in Phase 1, by combining and cross-refencing datasets to extract additional information, or by field study visits, including to other countries.

These techniques can help us to:

• check and confirm the accuracy of existing information
• fill gaps in knowledge, and
• support policies with solid evidence

In particular, whenever possible in this phase, it is important to meet with other people who are involved in the policy area. Whether they’re service delivery professionals within government departments or policy experts in Ireland or abroad, meeting with them can provide valuable insights. These interactions can help illuminate key points from potentially dry data and even reveal any pre-existing biases we might carry.

Conducting bespoke surveys or research is a common approach to gather additional data. This is generally achieved by commissioning a specialist research agency to undertake the work. This type of research can generate quantitative data through questionnaires and qualitative data through methods such as interviews and focus groups.

When commissioning this type of data collection, we may need to conduct a competitive procurement process. This includes preparing and issuing a request for tender followed by an objective assessment of tenders received. All procurement processes undertaken must be in line with the Public Procurement Guidelines for Goods and Services to make sure the process is fair, robust, and transparent. You can find different examples of requests for tender on gov.ie, such as the Department of Justice and Equality request for tender for research services to estimate the scale of illicit markets in Ireland. (8)

(8) Department of Justice Requests for Tender for Research Services (www.gov.ie)

Preparing, commissioning, and conducting data collection takes time. We must balance the urgency of policy needs with the need for accurate data that we can reliably use and interpret. While timely decisions are essential, rushing the data collection process may compromise the quality of findings.

Data and evidence can give us useful insights into an issue and possible solutions. However, we also need to consult with relevant individuals, organisations, and groups who have an interest in, or are affected by, the specific policy issue. This helps us to be sure we have a full understanding of the issue and its impact.

To gather different viewpoints and insights, we, as officials, need to:

• identify key stakeholders
• hold interviews
• host focus groups, and
• consider other consultation methods including surveys and thematic forum

It is important to listen actively and to be transparent to make sure stakeholders are heard and feel valued. The aim is to build legitimacy, manage conflicts, and bring in stakeholder feedback. This will help create policies that are evidence-based, inclusive, and widely accepted. These lead to more effective and socially relevant policy solutions.

Consultations can vary widely in scope and scale, from Citizen’s Assemblies held to discuss topics such as gender equality and biodiversity, to more niche consultations focused on a specific policy change. Consultations may request written submissions, in-person discussions, or a combination of both. The specific approach adopted for a public consultation will depend on a number of factors, including:

• the stakeholders involved
• the subject matter, and
• the timeframe available

A department may have scheduled consultations as part of its annual calendar. For example, the Department of Finance and Department of Public Expenditure, NDP Delivery, and Reform host the National Economic Dialogue every year in advance of the Budget. The Dialogue provides a deliberative forum for stakeholders to participate in an open and inclusive exchange on the competing economic and social priorities facing the government. It is not intended to produce specific budget proposals or recommendations but to assist participants in preparing their own pre-budget submissions.

The ‘Public Consultation Principles & Guidance’ is a useful resource when designing a consultative process. It can also be useful to learn from other consultations. A link to the guidance, as well as other public consultation resources, is listed in Appendix A.

Once we have gathered all our data and evidence and consulted with our stakeholders, we need to evaluate it carefully. This will help to assess what might work to tackle the policy issue. Some evaluators advise ranking evidence according to the precision and strength of the methods used to gather the data. Bias can be difficult to spot when examining all kinds of evidence, but here are some general questions we can use to help identify bias:

• what is the source of the evidence?
• how has the evidence been funded?
• were appropriate research methods used for the subject of the research?
• does the content of the evidence seem to capture all related facts, or does it seem to support a particular outcome?
• does the evidence rely on strong or hyperbolic language?
• is the evidence supported by links and trustworthy citations?

We must be guided by the evidence when forming a judgement about what intervention would be most effective. To do this, we need to critically evaluate the data. During earlier tasks when we assessed and reviewed available data (Task 1 & 2) and sourced additional data (Task 3), we asked some important questions regarding the accuracy and validity of the data. At this point, the key question facing us is what does this data tell us about the issue, and is there an alternative interpretation of the data?

For example, the issue of the working poor has been increasingly raised as one that demands attention. While we might think a solution may lie in raising pay in general through an increase in the minimum wage, the data shows that most people earning the minimum wage are second or third earners in relatively high-income households. By contrast, half of all people in single-earner working-poor households depend on someone working less than 25 hours per week. Given this high incidence of low-hours employment, an intervention such as an increase in minimum wage is likely to have a limited effect. Other solutions, such as increasing the availability of child-care, might be more helpful. (9)

(9) Roantree et al, op. cit.

If we evaluate the evidence well, it should produce strong guidance as to the kinds of policy options that could effectively address an issue.