top of page

Nine step policy development process. 

policy development

Developing good policy is a complex and multifaceted process that demands careful consideration, collaboration, and substantial time and effort. It is an indispensable aspect of governance, whether in the public sector, private organizations, or even within communities. There are many compelling reasons why investing the time and effort into crafting effective policies is essential.

 

Well-crafted policies provide a structured framework for decision-making. They outline clear objectives, define roles and responsibilities, and establish guidelines for actions. This clarity helps organizations and governments navigate various situations with consistency and fairness, reducing the risk of arbitrary or biased decision-making. In turn, this fosters trust among stakeholders, whether they are citizens, employees, or investors, as they can rely on a transparent and accountable system.

 

Good policy development involves extensive research, data analysis, and consultation with experts and affected parties. This process ensures that policies are grounded in evidence and are informed by diverse perspectives. By integrating a wide range of inputs, policies become more robust, adaptable, and capable of addressing the complexities of real-world challenges. In essence, policy development serves as a mechanism for harnessing collective wisdom and knowledge, leading to more informed and effective solutions.

 

Investing in the development of good policies contributes to long-term stability and sustainability. Policies help mitigate risks, anticipate future issues, and promote strategic planning. By considering the potential consequences of various courses of action, organizations and governments can avoid costly mistakes and crises. This proactive approach not only safeguards the interests of stakeholders but also saves resources that might otherwise be wasted on reactive measures.

 

Developing good policy is hence a process that is worth the time and effort because it provides a foundation for fair and consistent decision-making, harnesses collective knowledge and expertise, and contributes to long-term stability and sustainability. Whether in the public or private sector, investing in policy development is an essential element of effective governance and responsible leadership, ultimately leading to better outcomes. 

 

While every organisation, government, or institution is likely to develop policy differently – there are some basic steps that should be included as part of the development process. Following these steps will increase the likelihood of high quality policy advice. The extent to which each stage is followed will depend on a range of conditions such as the project’s size, the resources available, the timeframes, and the risks associated with the project.

Contents

Step 1. Define the Problem

Step 2. Identify Objectives

Step 3. Develop Options

Step 4. Determine Criteria

Step 5. Access Options

Step 6. Formulate Reccommendations.

Step 7. Implement

Step 8. Monitor

Step 9. Evaluate

Step 1. Define the Problem

 

The aim of the first stage of the policy development process is to clearly define the problem you want to address.

 

This guidance covers the following topics:

  • Why defining the problem is important

  • Key tasks when defining the problem

  • Things to avoid when defining the problem

  • Key deliverables and achievements

  • Quality standards

 

1. Why defining the problem is important

 

During this stage in the policy development process you need to develop an:

  • in-depth understanding of the problem’s underlying causes

  • awareness of the competing values, ideologies and beliefs surrounding the problem.

 

By identifying and clearly stating the real underlying problem (as opposed to a perception of the problem), your policy response is more likely to address the problem adequately. If you define the problem incorrectly, your subsequent analysis is likely to be of limited value.

 

Revisit your problem definition as you gain additional information and carry out analysis throughout the policy development process. With each iteration of the definition you can be more confident you are focusing on the right problem.

 

Note: It is important to make the time to understand and define the problem clearly, even if you have only a short timeframe to develop a policy response.

​

 

2. Key tasks when defining the problem

 

Define the problem after gathering and analysing the information most relevant to the problem. You will use the information you gather during this stage throughout the policy development process, as it frames and generates the subsequent stages in the process.

 

Key steps when defining the problem

 

Follow these steps to define the problem.

​

1. Plan what information to gather and how

 

Good quality policy making depends on high quality information derived from a variety of sources. This information includes:

  • expert knowledge and opinion

  • domestic and international research

  • statistics and empirical data

  • stakeholder consultation information

  • evaluations of previous policies and interventions

  • new research

  • secondary information, including open source information from the internet.

 

Each policy project requires different amounts and types of information depending on its size and nature.

 

Your policy must be informed by the available information. If minimal or no published information is available, acknowledge this limitation explicitly.

 

2. Analyse the information

 

3. Draft the problem definition

 

4. Consult on the problem definition

Revisit the problem definition and revise it as needed.

 

5. Identify the level of approval

Identify the level of approval required for the project to proceed beyond this stage (e.g. approval from a policy manager, senior manager, Chief Executive, or a Minister(s).

 

Key questions when defining the problem

 

It is helpful to ask the following key questions when defining the problem.

 

What is the perceived problem?

 

What assumptions underlie the problem?

It is your role to identify, make explicit and explain the different assumptions and values behind the problem. The idea that there is a problem that needs to be resolved exists usually arises when someone thinks something is not working as well as it could (or at all) or has an idea about an improvement. 

 

Stakeholders have differing values and will not always agree what the problem is or even if there is a problem. People usually define a ‘problem’ as something that is ‘wrong’ or ‘undesirable’. However, different people define these terms differently, depending on their evaluative framework. You need to be aware that not everyone will think the facts you (or others) have defined as a problem are necessarily even a problem. 

 

Consider whether a stakeholder is presenting you with a solution rather than a problem.

 

Do any theoretical frameworks and values underlie the problem?

 

Different theoretical frameworks:

  • define policy problems differently

  • emphasise different outcomes

  • suggest different criteria for assessing options

  • lead to different options for addressing the problem.

 

Understand the key theoretical frameworks within which policy is developed, so you can work within the relevant framework and understand the language used by, and debate issues with, analysts using other frameworks.

 

Analysts working within different frameworks may take different approaches to the same stage of the policy cycle. This is particularly relevant when developing policy in a cross-agency/organisational context.

 

Understand the influence of common belief systems is important, so you can:

  • understand the assumptions behind policy proposals

  • use the intellectual frameworks associated with belief systems as starting points for analysing the causes of, and solutions to, a problem

  • identify their frames of reference and make allowances for these when considering policy matters.

 

What is the Government’s underlying framework?

 

Governments and organisations with different political and ideological beliefs have different views and make different assumptions about the Government’s role and the nature of people in society. For example, governments may have different views about equality:

  • Some may believe that equality is not possible, but governments should look after less fortunate people

  • Some may think that Governments should ensure equality of opportunity, then leave the responsibility to utilise the opportunity to the individual

  • Others may believe Governments should strive for equality of outcome.

 

Because of these different assumptions, different policies would be developed to provide for the same situation. Recognise and challenge these assumptions in your initial analysis of whether a problem needs more attention.

 

What are the problem’s perceived symptoms?

 

What is the context in which this problem has emerged? What conditions or events lie behind this problem?

 

Describe (rather than assess) the political, cultural and socio-economic aspects of the context surrounding the problem. Understanding the context is critical for policy design.

 

Determine how these and other factors (such as institutional capacity, the history of resource use and the policy, economic conditions (local, national or international), and pressure from society) might affect the problem.

  • What natural and built environmental conditions surround the problem?

  • Who is affected by the problem and how?

  • What are the characteristics of the population that is or will be affected by the problem?

  • How does the population vary by age, gender, ethnicity, employment status or locality?

  • Understand the needs and characteristics of the population affected by the issue, so you know what behavioural aspects to consider. Think carefully about what is most likely to influence the group to respond positively to a policy intervention.  

  • How is the effect of the problem distributed across different groups and regions?

  • How does the problem affect specific geographic, age, socio-economic and cultural groups?

  • What are the problem’s effects in terms of magnitude, duration and frequency, regional limits, reversibility, cumulative impact, local sensitivity and probability?

 

Multiple groups may be affected by the problem, so multiple views will be represented in the problem definition. A problem for one group may represent a benefit for another, or a range of problems might exist, but only for a certain group.

 

Consider also problems that may emerge in the future linked to symptoms or context factors present today.

  • Can the problem be quantified?

  • What is the problem’s trend?

  • Is the trend increasing or decreasing?

  • Is the nature of the problem changing, if not increasing in occurrence?

  • Is the problem linear, discontinuous or cyclical?

 

Think in terms of deficit and excess. Indicate the problem’s size. If something is developing ‘too slowly’, is ‘too big’ is or ‘too small’, define ‘too slow’, ‘too big’ or ‘too small’. 

 

Gather information to help you estimate the relevant magnitudes of the problem. Consider identifying a range of estimates as well as a point estimate. 

 

Is this problem similar to or different from other problems? What were the solutions to those problems?

  • What are the interdependencies between these issues and other issues?

  • Is there a larger problem of which this problem is only a part?

  • Are there smaller problems into which this problem can be divided?

  • Does this problem exist in other countries? If so, how is it defined and handled there?

  • How has this problem been defined and handled in the past?

  • Is this a universal problem for any society or is it peculiar to a society with a particular institutional or political structure?

 

Research and appreciate the history behind the problem (such as previous policy responses). Many situations requiring a policy response are not new.

 

Government agencies frequently have considerable experience and institutional knowledge in dealing with a wide range of situations. Draw on this experience to learn lessons and avoid repeating an aspect of a policy that has not worked well before. 

 

What are the wider community’s views about this problem?

 

Understand the interests and expectations of the people who will, or may be affected by, or have an interest in, the problem. Not everyone will think the facts you (or others) have defined as a ‘problem‘ really are a problem, as each person may apply a different evaluative framework to the same facts. 

 

  • Are people avoiding sensitive issues? If so, how are you managing this?

  • What other agencies are involved?

  • What non-governmental agencies are involved with or affected by this problem?

  • What agencies are working in the community or with the people affected by this problem?

 

Are you distinguishing between causes and symptoms?

 

Symptoms are often how problems present. Distinguish causes from symptoms by asking, ‘If the problem, as identified, was addressed would it be resolved?’ If your answer is ‘no’ you are probably identifying symptoms rather than causes.

 

Ask of each possible cause, ‘If this is the true cause of the problem, how does it explain each symptom?’

 

Different groups and interests will make different assumptions about the underlying causes. If there is no agreement on the cause, it may be difficult to formulate agreed policies.

 

You can summarise your causal analysis using a tool called a 'problem tree’. This is a practical way to represent the different problems and show how they relate to each other.    

  • Are there competing hypotheses about this problem's underlying causes?

  • What are the underlying assumptions of each hypothesis?

  • What theoretical frameworks underpin each hypothesis?

  • What evidence supports a causal link?

 

How does the problem affect your Government?

 

Consider the following.

  • What are the micro- and macro-economic effects of the problem? How does this problem affect the Government’s objectives?

  • What are current Ministers and government administrations declared interests and preferences? 

  • Can the Government influence the causes or symptoms? What are your agency’s role and mandate with regard to the problem?

  • Consult at both interdepartmental and public and stakeholder levels.

  • What are other departments’ perspectives?

 

Engage with other government agencies to clarify the policy problem and related issues across government. These discussions can highlight information, issues and potential problem areas, draw attention to important relationships that you will need to manage throughout the process, and highlight cross-government outcome considerations.

 

Other agencies may know nothing about the particular situation, but they may be able to share their experiences from dealing with similar problems and situations.

 

3. Things to avoid when defining the problem

 

When defining the problem, avoid the following pitfalls that can decrease your definition’s usefulness.

 

  • Defining the solution to the problem. Do not construct the definition of the problem being considered so that it includes an implicit solution. The definition must be open to a range of possible solutions. For example, saying ‘Too little housing exists for homeless people’ implies the solution is ‘more shelter’. This will inhibit you from thinking about ways to prevent people from ‘becoming homeless’. Start with ‘There are too many homeless people’. 

  • Forgetting to question apparent causal chains. Verify that the alleged causes of a problem are real; not assumed. Ask, ‘Do the problems you have identified lead to the outcomes? If so, to what degree do they lead to the outcomes?’

  • Cause and effect needs: a high degree of association between the causal factor and the effect; a logical time sequence where the programme precedes the effect; to eliminate other possible causes; an association that remains consistent when studied in different groups and at different times; agreement with known facts or theory; to show that with greater exposure there is a greater effect. Most single studies cannot show cause and effect on their own. By demonstrating the same results by researchers over several studies, you can feel more confident in the findings.

  • Not recognising the philosophical grounds or frameworks. The raw material for your initial problem definition may come from your Minister or other stakeholders. It may be narrowly confined to a technical problem or broadly located in a controversy of wide social interest. It may point to a condition people do not like or consider ‘bad’, like school children use of social media, media violence or global warming. Explore the philosophical grounds and frameworks in which you, your client or your eventual audience should or should not consider the alleged condition ‘bad’.

  • Attributing greater representativeness to the information that is valid. You are unlikely to be able to apply the results of a small-scale study to a wider group or geographic area. Your problem definition should reflect only the data you analysed. Take into account the credibility and validity of the information you have interpreted. Every policy process has limitations. Acknowledging these limitations only strengthens the policy’s credibility and likelihood of success.

 

4. Key deliverables and achievements

 

Your key deliverable at the end of this stage is a clear, concise, accurate and agreed problem definition. You will also have achieved the following:

 

An understanding of the significance of the problem being considered.

You will have produced, and got agreement on, a succinct summary of the problem (preferably quantitatively described). For example, your problem definition could state, ‘Groups x and y are suffering from the adverse outcomes a and b as a result of factors p, q and r’. This provides a clear reference point for later stages of policy development.

 

Clarity about whether your agency needs to pursue the problem.

You will be able to explain:

  • why the Government should be involved in addressing the problem

  • who else needs to be involved in developing a solution (e.g. other departments).

 

Authorisation for further policy development.

 

Without authorisation and support from senior levels the policy development process will not be workable and the resulting policy will not be viewed as official. However, not every policy issue needs ministerial approval. At the outset of this stage you will have identified with your manager the level of approval required for the project to proceed beyond this problem definition stage.

 

If approval includes applying project management methodology, scope the policy development and draft a project plan.

 

5. Quality standards

 

Your problem definition must meet the following standards and identify the context and parties, including:

  • an analysis of the current situation

  • current Government involvement (e.g. legislation)

  • the decision-making context

  • stakeholder assumptions

  • sector concerns, assumptions and underlying values

  • legal considerations (e.g. fit with law)

  • ethical considerations

  • risks of current situation

  • impact on outcomes.

 

The problem definition identifies:

  • the problem’s variables, symptoms and root cause

  • other factors occurring within and around the problem

  • the risks of change or no change

  • the benefits of change or no change.

 

A logical explanation exists for why your agency is advising on the problem. The problem definition exploits your agency’s collective institutional knowledge and experience beyond the agency.

 

Project management requirements for scoping and project planning have been met, if appropriate.

step 1
step 2 identify objectives

Step 2. Identify Objectives

 

The aim of the second stage of the policy development process is to identify what the policy needs to achieve (i.e. its objectives).This guide covers the following topics:

 

  • Why identifying objectives is important

  • Key tasks when identifying objectives

  • Things to avoid when identifying objectives

  • Key deliverables and achievements

  • Quality standards​

    • Planning with post it notes
    ​1. Why identifying objectives is important

     

    Objectives describe what the situation would look like after the problem was resolved. They focus on what has to be achieved, not how it is to be achieved.

     

    Well-founded, clear objectives:

    • show you understand what the policy needs to achieve

    • show you understand what the situation will look like once a policy has been decided and actions taken

    • identify what the Government wants to achieve through action or involvement with the problem

    • provide a sound basis for you to design and select options

    • provide a benchmark against which to evaluate achievements.

     

    2. Key tasks when identifying objectives

     

    This subsection covers the following topics.

    • Content of objectives

    • Steps when identifying objectives

     

    Content of objectives

    Identify what you hope to achieve through any action or intervention (i.e. the objectives). Objectives must contain the:

    • specific results being sought

    • contribution the results will make towards government or departmental outcomes or other objectives

    • measures or standards to be used to assess whether the expected results have been achieved

    • timeframe within which the objectives are to be accomplished.

     

    Steps when identifying objectives

     

    Follow these steps to clarify your objectives.

     

    1. Determine what has to change to resolve the problem

    Using your problem definition to determine exactly what it is that has to be corrected, maintained, improved or created to resolve the problem.

    Initially, it may be appropriate for you to focus on needs that are amenable to change or are the most prevalent, serious or pressing.
     

    2. Determine the policy’s intended outcome

    Establish the outcomes the objective is intended to achieve or contribute to.

    Determine the policy’s intended outcome after referring to:

     

    • goals the Government has specified for the government sector generally or your agency in particular

    • intermediate outcomes specified in your agency’s statement of intent (if your agency has one)  

    • outcomes specified in relevant legislation

    • specific Government or ministerial directives relating to the particular policy area in which you are working.

     

    3. Draft your objectives

    It is important to make your objectives SMART:

    • Specific

    • Measurable (quantifiable)

    • Achievable (feasible, realistic)

    • Results focused (explicit)

    • Achievable within a realistic timeframe.

     

    Specify the objectives broadly enough so all alternative solutions can be considered, but not so broadly the range of alternatives becomes too large to assess, or the extent to which the objectives have been met becomes too hard to establish.

     

    4. Review your objectives

    Are your objectives specific about what has to be achieved and internally consistent? If not, adjust them accordingly.

     

    5. Identify indicators of success

     

    Each objective needs clearly defined indicators. When measured, the indicators will tell you whether the policy’s objective has been achieved (i.e. whether the policy has been successful).

     

    An indicator must:

    • be a logical link between the available evidence (what is measurable) and the objective (what is to be achieved)

    • be sensitive enough to measure changes within an initiative's timeframe

    • be affordable (i.e. in terms of ‘return for effort’ and the balance of costs and benefits)

    • be stated clearly and concisely

    • make sense to stakeholders.

     

    6. Seek managerial approval for your objectives

    Arrange for the manager who will ultimately approve the new policy to review and agree on your objectives. This review is critical for avoiding misunderstandings about scope, timing, responsibilities and ownership.

     

    3. Things to avoid when identifying objectives

     

    When identifying the objectives, avoid the following pitfalls that can decrease their usefulness.

    • Identifying policy decisions in the objectives. Your objectives must not contain decisions about how something will be achieved; only what will be achieved. For example, the objective would not be ‘to provide literacy programmes to adults’ but ‘to increase adult literacy’.

    • Writing vague or over-optimistic objectives. Vague or over-optimistic objectives limit your ability to assess the options. Without clear objectives your policy lacks focus and a basis on which you can decide about options and implementation.

    • Be realistic. Vision statements are an inspiring picture of the future, but objectives should reflect realistic priorities.

     

    If you have arbitrarily selected objectives you may address needs that do not exist, do not fit with the Government’s goals, or are unimportant.

     

    4. Key deliverables and achievements

    At the end of this stage you will have:

    • identified clear and concise objectives that are consistent with the Government’s goals

    • authorisation and support for continuing the policy development process.
       

    5. Quality standards

    Your objectives must meet the following quality standards.

    • The objectives are outcomes or changes of state.

    • The objectives are SMART.

    • The objectives relate to the problem and the outcomes sought (not processes).

    • The objectives are consistent with existing policies, strategic statements and statutory constraints.

    • The objectives' relationship to your agency’s mission, priorities and role is clear.

    • The objectives’ relationship to the Government’s goals is clear.

    step 3 develop options

    Step 3. Develop Options

     

    The aim of the third stage of the policy development process is to develop options that are likely to resolve the problem.

     

    This guide covers the following topics:

    • Why developing options is important

    • Key tasks when developing options

    • Things to avoid when developing options

    • Key deliverables and achievements

    • Quality standards

     

    1. Why developing options is important

     

    More than one way to achieve a policy objective will exist. If you identify and explore all the different options, you can advise decision-makers of the merits and disadvantages of each.

     

    2. Key tasks when developing options

     

    This section covers the following topics:

    • Types of policy instrument

    • Steps for developing options

     

    Types of policy instrument

     

    Policy instruments are the means by which a policy is implemented.

    Your options will include one or more policy instruments, for example:

    • legislative instruments (e.g. new or amended Acts or regulations)

    • programmes and services administered through public or other agencies

    • informative materials (e.g. brochures)

    • infrastructure (e.g. hospitals, schools, roads, dams or buildings)

    • education (e.g. a public education campaign)

    • funding (e.g. using government spending and taxing powers (such as taxes, concessions, grants, contracts, loans, vouchers or transfer payments) to shape activity beyond government control).

     

    Considering a range of policy instruments is important, because the Government will want to use the most appropriate tool to do its job.

     

    The policy instrument you choose can have a major impact on a policy proposal’s cost, timing and acceptability, so choose carefully.

     

    Policies are often implemented using a combination of policy instruments (e.g. an amendment to an Act, a public education programme and an allocation of new resources).

     

    Steps to develop your options


    1. Develop a comprehensive list of alternative options

    List all the alternative options that might contribute to achieving the policy’s objectives, for example:

    • amending legislation

    • providing services

    • providing incentives

    • entering into agreements

    • purchasing land

    • building capacity in the community

    • doing nothing (i.e. taking no further action).

     

    Include the ‘do nothing’ option because it may be the best option. It may be unnecessary for the Government to address the problem now or naturally occurring change might mitigate or resolve the problem.

     

    In the last stages of your analysis, you will want to be assessing no more than three or four options. In the beginning, err on the side of comprehensiveness, but when assessing the options discard obvious losers, combine others, or reorganise others into a single ‘basic’ alternative with one or more subsidiary ‘variants’. 

     

    2. Simplify the list of options

    Simplify your list of options by considering the practicality, necessity and feasibility of each. This will make sure time and effort are not wasted further evaluating the benefits and cost of ineffective options.

    • Reject options that have little or no positive effect.

    • Reject options that fail to meet the minimum objectives.

    • Reject options that are impracticable.

    • Retain only those options that are realistic, feasible and practicable, will meet the objectives and are aligned with Government policy.

     

    The key to simplifying your options is to distinguish between a ‘basic’ option and its variants. The ‘basic’ option is a specific course of action; ‘variants’ describe different methods of implementation, collaboration and partnerships, and financing. 

     

    At the fifth stage of the policy development process you will assess the remaining options’ effectiveness, efficiency and comparative worth.

     

    3. Explain how each option will contribute to the objectives

    Identify the ‘chain of events’ linking inputs to results (i.e. what are the processes, techniques, tools, events, technology and actions that are involved in the option?). Identify the relationships between the inputs, planned activities, and intended outcomes or results. Determine the implementation timeframe. Identify what and how many resources each option will use and where they will come from.

     

    Resources include:

    • fiscal resources (e.g. appropriated funds, special grants, donations and user fees)

    • facilities and equipment

    • the knowledge and skill bases required to implement the option.

    • important partners or collaborators (e.g. local and national organisations involved in planning, delivery or evaluation).

     

    Explore the legal, administrative and compliance implications and costs.

     

    3. Things to avoid when developing options

    When developing the options, avoid the following pitfalls that can decrease the options’ usefulness:

    • Including variations on a single option. Even if you feel one option is a front-runner, keep an open mind about the other options; otherwise you are likely to develop only options that are variations of your favoured option. Alternative options are not just different ways of carrying out the same thing.

    • Assuming alternative options are mutually exclusive. That options are ‘alternatives’ does not mean they are necessarily mutually exclusive. Analysts may use the term ‘alternative’ to mean that choosing one option implies foregoing another or they may use it to mean that each option is just another action that might resolve the problem, perhaps in conjunction with other options (i.e. ‘alternatives’). Avoid this ambiguity in your list of options and be aware of it in others’ lists of options.

     

    4. Key deliverables and achievements

    At the end of this stage you will have:

    • a comprehensive list of all the options to resolve the problem being considered

    • a smaller list of practicable options to resolve the problem and achieve objectives

    • agreement from the appropriate level of management or Ministers about the range of options to be assessed.

     

    5. Quality standards

    Your options must meet the following quality standards:

    • A clear relationship exists between the options, problem definition and objectives.

    • Each option’s effect has been identified clearly, described on a basis of best evidence and justified.​

    Step 4. Determine Criteria

     

    The aim of the fourth stage of the policy development process is to identify an appropriate range of criteria to help you (and others) to make the necessary policy decisions.

     

    This guide covers the following topics:

  • Why determining criteria is important

  • Key tasks when determining criteria

  • Things to avoid when determining criteria

  • Key deliverables and achievements

  • Quality standards

    • ​

    step 4 determine criteria
    People writing on white board

    ​1. Why determining criteria is important

     

    Criteria are the mechanisms that enable you to assess the different facets of options and their projected outcomes objectively and, ultimately, to recommend a preferred course of action. Criteria allow different values to be brought into the policy process, because they are evaluative standards for judging the ‘appropriateness’ of each outcome’s projected outcomes. 

     

    Key tasks when determining criteria

    Link your criteria to the identified problem and objectives. The most important criterion is that the option resolves the policy problem to an acceptable degree.

     

    However, you need to identify other criteria so you can highlight how options differ from each other.

     

    This section covers the following topics.

  • Key questions when developing criteria

  • Common criteria

  • Key steps when developing criteria

    •  

    Key questions when developing criteria

    Answer the following questions when developing your criteria:

  • What agency policy frameworks are relevant to the problem being considered?

  • What policy frameworks will other government agencies use when considering this problem?

  • How can other government agencies' underlying values be reflected in the criteria?

  • What international implications need to be reflected in the criteria?

  • What ethnic and gender perspectives and human rights and privacy issues should be reflected in the criteria?

  • Do the options give rise to social, economic, environmental or cultural considerations that should be reflected in the criteria?

  • What goals or objectives of the Government should be reflected in the criteria?

  • Are there potentially competing Government goals or objectives that should be reflected in the criteria?

    •  

    Common criteria

    The most common criteria used in policy analysis are as follows:

  • Effectiveness: the extent to which the option's outcomes address the problem or enhance the achievement of the objectives.

  • Efficiency: the extent to which the results will be achieved for the lowest possible costs in terms of financial and other resources.

  • Cost: the expenses the option will incur, including who will pay for them and the extent to which the government will pay for them.

  • Administrative efficacy: the extent to which the option is administratively simple, efficient and effective.

  • Complexity: the extent to which clients and administrative and enforcement officers will understand the option.

  • Practicality: the extent to which the option can be practically implemented.

  • Distribution of benefits: the extent to which a net benefit (economic, social or legal) exists and who receives it.

  • Fairness: the extent to which differences exist between client groups, including whether target groups are properly addressed and whether the option distributes benefits or burdens differently among groups.

  • Policy consistency: the extent to which the option is consistent with other policies and meets legislative and the Government’s commitments.

  • Legality: the extent to which the option can be implemented under existing law or requires legislative amendment.

  • Community acceptability: the extent to which the option is acceptable to the communities affected (particularly the stakeholders identified in its development), whether client needs will be met, and whether other stakeholders will be satisfied.

  • Government policy and outcomes: the extent to which the option is consistent with Government policy and outcomes.

  • Political acceptability: the extent to which the option is acceptable to the Government (i.e. Ministers)

  • Gender implications: the extent to which and how the option’s outcomes affect the different genders.

  • Implications for other ethnic groups: the extent to which and how the option’s outcomes affect ethnic communities, having regard to the growing and diverse ethnic communities.

  • Human rights implications: the extent to which and how the option’s outcomes affect individual human rights.

  • Privacy implications: the extent to which and how the option’s outcomes affect privacy.

  • International compatibility: the extent to which other jurisdictions have followed the same approach or whether our different approaches are compatible.

  • Level of Government intervention: the level of government intervention each option requires (e.g. creating a new law is a high level of intervention, while introducing a new programme that provides incentives represents a lower level of intervention). Options that achieve the objectives with the lowest level of government intervention are often favoured.

    •  

    Key steps when developing criteria

     

    The key steps when developing your criteria are as follows:

     

    1. Develop a comprehensive list of possible criteria

    List all the possible criteria that may be useful in assessing the options’ projected outcomes. In formulating the criteria, consider the range of perspectives and values to be taken into account. Think about what each group of stakeholders considers to be important.

     

    Consider the policy frameworks within which the stakeholders operate, their underlying values, and the criteria that reflect those values.

     

    Consider the criteria from the point of view of different disciplines (e.g. legal, economic, social, environmental and constitutional):

  • A legal perspective may raise questions about the extent to which an option affects constitutional arrangements or what processes might be appropriate when changes are proposed to our constitutional arrangements.

  • An economic perspective may raise issues about the type of good that is to be provided (i.e. a public or private good) and the most efficient means of providing that good, whether Government intervention to control or regulate the supply of the good is warranted, and, if it is, the lowest level of intervention that would achieve the objective.

    •  

    2. Use ‘ends’ rather than ‘means’ as criteria

    When developing criteria focus on ends rather than means. For example, if you are looking to improve people’s health, do not assume increasing access to doctors (means) is the only or most effective way to improve health (end).

     

    3. Facilitate trade-offs

    Criteria that are expressed in simple either/or terms can be used to oppose or support any particular proposal. Such criteria discourage debate about the extent of advantages or disadvantages.

     

    4. Develop clear and appropriate measures for criteria

    When establishing the criteria, move from more general descriptors (such as ‘community wellbeing’) to particular, specifiable measures of such concepts. These specifiable measures will enable you to make and support your policy analysis and preferred options.

     

    Measures can include the following:

  • Specific measurable criteria: the individual criteria that enable an option’s projected outcome to be assessed (e.g. the fiscal cost to the government or a reduction in the number of problem gamblers).

  • Experts’ judgments: informed people or official bodies’ judgements about the expected results.

  • Multiple measures: if each measure is partial or imperfect, multiple measures supplement each other so an outcome can be assessed.

  • Subjective feelings: when human perception is involved, objective conditions may not correspond with affected parties’ subjective valuations (e.g. physiological health measures may not reflect people’s subjective health experiences).

    •  

    5. Reduce and simplify the list of criteria

    Consider the broad list of criteria you have developed. It is likely some will appear more, and some less, useful. You might be able to eliminate some criteria as being less significant or combine two criteria. Avoid selecting criteria that might bias the later states of the assessment.

     

    Finish with no more than four or five criteria. Larger numbers of criteria are likely to make the job of resolving conflicts between criteria increasingly complex, so the determination of the preferred option becomes more difficult.

     

    6. Things to avoid when determining criteria

    When determining the criteria, avoid the following pitfalls that can decrease the criteria’s usefulness.

  • Overlapping criteria.

  • Ensure your set of criteria does not conceal duplication, with the same concept expressed within different criteria. Overlapping criteria increase the risk that decision makers will be misled about the merits or otherwise of an option.

  • Identifying solutions rather than criteria.

    •  

    7. Key deliverables and achievements

    At the end of this stage you will have:

  • generated a comprehensive list of criteria that can be applied to each option’s projected outcomes

  • obtained agreement from Ministers, government agencies, your manager or stakeholders to the refined set of criteria you will apply when assessing your options in the next stage.

    •  

    8. Quality standards

    Your assessment criteria must enable measurement of all the policy objectives’ important dimensions.

    Person writing in notebook

    Step 5. Assess Options

     

    The aim of the fifth stage of the policy development process is to assess the options objectively to determine the preferred option.

     

    This guide covers the following topics:

    • Why assessing options is important

    • Key tasks when assessing options

    • Things to avoid when assessing options

    • Key deliverables and achievements

    • Quality standards

     

    1. Why assessing options is important

     

    Assess the options using the criteria to determine which will best resolve the problem being considered.

     

    Your objective assessment about each option will include:

    • how each option will be implemented

    • the relative merits of each option in terms of benefits, implementation costs and efficacy.

     

    Assess each option against your criteria.

     

    2. Key tasks when assessing options

     

    The key tasks when assessing the options are as follows:

    • Assess how options are likely to work in practice to:

      • identify practical constraints to be overcome if the options are to be successful

      • develop more accurate estimates of the likely cost and impacts of options

      • determine whether options need to be modified, so any population group that is intended to benefit as part of the outcome, does so

      • determine whether options are likely to represent value for money

      • determine whether options' benefits are likely to be sustainable in the longer term.

     

    • Apply the criteria to the projected outcomes for each option. Apply criteria to each option’s projected outcomes so you can assess each option objectively; but not judge each option. This enables you to determine: ‘Option X is expected to deliver outcome Y, which is the best of the possible outcomes’.

    • Identify the risks associated with accuracy of projected outcomes. You will probably need to reach agreement with officials from other departments (who may have a different perspective on what criteria are most important or what the projected outcomes will be). If you are not certain about your projections, qualify any statement you make about their accuracy.

    • Develop an outcomes matrix for all the options. An outcomes matrix:

    • brings together your analysis of the criteria and projected outcomes

    • enables you to form an overall view of the preferred option

    • helps you to identify and resolve conflicts among options.

    • Make the trade-offs between each option’s outcomes. The most common trade-off is between costs and the provision of a service (e.g. extending pool hours from 6 pm to 10 pm against an annual cost of $1M). Another common trade-off, especially in regulatory policies, is between privately borne costs (e.g. the cost to a company of installing pollution removal equipment) and social benefits (e.g. the improved health of the communities surrounding the company and better protected forests).Trade-offs occur at the margin, ‘If we spend an extra X dollars for an additional unit of service Y, we can get an extra Z unit of good outcome’. By considering trade-offs Ministers answer the question, ‘Does society value Z more than X? If the answer is ‘yes’, Ministers will purchase another Y; if the answer is ‘no’, they will not.

    • Consider a cost-benefit analysis. A cost-benefit analysis seeks to identify and measure the implications (including externalities) associated with each option, so you can assess objectively the option with the greatest level of net benefit to society. A formal cost-benefit analysis requires robust data and input from a range of people with specialist skills, so use it to resolve only the most significant of policy issues. It is not always appropriate or necessary to undertake a cost-benefit analysis. 

    • Consider other analyses. The nature of your analysis depends on the matter you are considering and the practices of the discipline within which you are working.

    • Consider using any of the following eight analytical tools, as appropriate.

      • Models predict likely outcomes in different circumstances.

      • Organisational maps show diagrammatically the formal and informal relationships between and within agencies. They are particularly important when establishing responsibilities in options involving several agencies.

      • Logic models map out the various interventions and their interrelationships to deliver a policy option.  

      • Cost-effectiveness analyses estimate the costs (direct and indirect) from options, so you can determine whether the option is likely to deliver value for money.

      • Cost-utility analyses compare different programmes by using a common outcome measure so each programme has a cost per outcome measure. This enables you to determine which competing uses of resources gives the best value.

      • Comparative analyses draw on experience and good practice in other countries tackling similar policy issues.

     

    3. Things to avoid when assessing options

     

    When assessing the options, avoid the following downsides that can decrease the assessment’s usefulness.

    • Being too optimistic or pessimistic about an option’s outcomes.

    • Assessing options on an advantage/disadvantage basis.

     

    If you assess the options solely on an advantage/disadvantage basis you risk introducing analyst bias into your assessment.

    The two major problems with this are:

    • your definitions (or values) of what is an advantage (‘good’) and disadvantage (‘bad’) are not explicit, so you tend to describe options as opposites of each other (i.e. if something is an advantage of one option you then describe it as a disadvantage of another option)

    • you will tend to apply different considerations to each option, so providing an incomplete picture of the options.

    • Taking additional factors into account when deciding the preferred option. If you take factors other than the criteria already agreed into account when you are deciding the preferred option, you tend to apply different considerations to each option. This provides an incomplete picture of the options and leaves the analysis open to analyst bias.

    • Thinking of trade-offs as being between options rather than projected outcomes. You cannot assess the options if you think the trade-offs are between alternative options rather than projected outcomes. Convert the alternative options into outcomes, so the potential trade-off is clear. 

     
    4. Key deliverables and achievements

     

    At the end of this stage you will have:

    • assessed each practicable option on the basis of agreed criteria

    • assessed each option’s implications for the Government and your agency

    • authorisation for further policy development.

     

    5. Quality standards

     

    Your assessment of the options must meet the following quality standards, where appropriate.

    • Each option’s:

      • effect in resolving the issues has been clearly identified, described on a basis of real evidence and justified

      • economic, social, legal and political effects have been identified and described clearly

      • resource costs, savings and associated trade-offs have been identified clearly, justified and described

      • impact on your agencies business operations costs has been assessed thoroughly

      • legal and legislative implications and costs have been identified clearly, justified and described

      • administrative and compliance implications and costs have been clearly identified, justified and described

      • implications for ethnic groups have been clearly identified, justified and described

      • implications by gender have been clearly identified, justified and described

     

    • International obligations have been taken into account.

    • If risks are associated with the favoured option, check:

      • all pertinent issues related to the risk are identified

      • issues and discussion are supported by research or statistical information

      • stakeholders have been canvassed for their views on the issues, and these have been recorded.

    step 5 access options
    step 6 formulate recommendations

    Step 6. Formulate Recommendations

     

    The aim of the sixth stage of the policy development process is to formulate your recommendation of the preferred option for resolving the problem and present your recommendation as a briefing paper to senior officials or Ministers.

     

    This guide covers the following topics:

    • Why recommendations are important

    • Key tasks when formulating recommendations

    • Things to avoid when formulating recommendations

    • Key deliverables and achievements

    • Quality standards

     

    1. Why recommendations are important

     

    Formulate clear recommendations, so the problem being considered can be resolved. Recommend a course of action (i.e. your preferred option) after assessing the options. Prepare that recommendation on the basis of the evidence and your analysis from previous stages. If you do not have enough information to make a recommendation, repeat previous stages until you do.

     
    2. Key tasks when formulating recommendations

     

    To formulate your recommendations undertake the following tasks:

    • Identify your preferred option. Describe precisely the decisions decision makers need to take. The option of a single activity is unlikely to address all aspects of a problem, so consider a mix of activities.

    • Generate justifiable recommendations.

    • Your recommendations must reflect the evidence, so do not make claims the evidence does not support. Link your recommendations to the evidence to strengthen the proposal’s credibility and the likelihood it will be accepted.

    • Acknowledge policy’s limitations. Take into account the credibility and validity of the information you are interpreting (including who has and has not been consulted and the pros and cons of the research strategy you chose), so you can acknowledge your policy’s limitations. Acknowledging the limitations, strengthens the conclusions you have reached and the likelihood your recommendations will be implemented.

    • State assumptions explicitly. State the assumptions you have made explicitly so they can be tested. For example, if you have made an assumption about a stakeholder's position, state that assumptions early so the stakeholder can confirm or correct it.

    • Identify links to the Government’s objectives. Link your recommendations to the Government's stated objectives. Read about the Government’s policy frameworks in its statements of general policy, party political documents, Acts of general and particular application, and Cabinet decisions. These documents may not always give you clear guidance about a particular policy issue, but they will give the Government’s strategic direction, so you can assess its political expectations.

    • Identify how outcomes will be measured. Include in your recommendations observable indicators that when measured will show when the policy has had an effect. This enables the policy to be monitored for effectiveness and efficiency, and adjusted in the light of experience.

    • Identify financial implications. If your recommendations have financial (fiscal) implications include information about:

    • costs (e.g. operating and capital costs, transitional and ongoing costs, administration costs and programme-related costs)

    • existing funding (or expenditure)

    • whether the expenditure can be absorbed and whether this is reasonable

    • savings

    • potential funding sources or new funding

    • the appropriation being sought

    • consultation with the Chief Financial Officer

    • Achieve agreement within and between government agencies. Where other agencies have a vested interest in the policy, the consultation process that must occur as part of development of the policy needs to result in agreement among the agencies about the recommended policy solution. Therefore, any disagreements need to be managed carefully.

    • Communicate your recommendations effectively. Your recommendations must stand alone, so their meaning is clear to a reader who has not read the whole paper. Ask, ‘Do the recommendations make sense to someone who knows nothing about the paper and its content?’

     

    3. Things to avoid when formulating recommendations

     

    When formulating recommendations and writing the paper, avoid the following pitfalls that can decrease the papers’ usefulness.

    • Failing to acknowledge limitations. If you do not acknowledge your information’s and processes’ limitations, your conclusions will be weakened and have less likelihood of being accepted.

    • Making recommendations that are not justified in the paper. Each recommendation must be supported by one or more statements in the paper. Do not introduce new material into the recommendations as this causes confusion and undermines the high-quality policy development process you followed in stages 1 to 6.

    • Writing unclear recommendations. If your recommendations do not stand alone (i.e. are not clear to a reader who has not read the whole paper) they are less likely to be approved.

     

    4. Key deliverables and achievements

     

    At the end of this stage you will have prepared a briefing or ministerial paper recommending a preferred option. Usually, a senior official will also give an oral briefing to the Minister.

     

    5. Quality standards

     

    Your recommendations must meet the following quality standards.

    • If risks are associated with the favoured option, check:

      • all pertinent issues related to the risk are identified

      • issues and discussion are supported by research or statistical information

      • stakeholders have been canvassed for their views on the issues, and these views have been recorded.

    • The recommendations stand alone (i.e. their meaning is clear to a reader who has not read the whole paper).

    • Appropriate links are made to other policies, the Government’s goals or agency outcomes.  

    Step 7. Implement

     

    The aim of the seventh stage of the policy development process is to develop a planned approach to implementation, so your policy can be implemented successfully.

     

    This guide covers the following topics:

    • Why planning implementation is important

    • Key implementation tasks

    • Things to avoid

    • Key deliverables and achievements

    • Quality standards

    ​

    1. Why planning implementation is important

     

    How and who implements the policy requires as much critical attention as you gave the policy’s development, because:

    • strategic choices in implementation affect the policy’s success and the use of resources

    • implementation is complex and may also involve other government departments

    • policy analysis and development is intricately related to implementation.

     

    Throughout the policy development process ensure the policy will be implemented effectively by consulting regularly and keeping the operational unit that will manage the implementation well informed.

     

    2. Key implementation tasks

     

    This section covers the following topics:

    • Implementing a policy successfully

    • Content of implementation plan

     

    Implementing a policy successfully

    To implement a policy successfully:

    • assess the implementation’s critical path to identify the key stages, so you can organise them efficiently, and identify and take corrective action to address potential problems promptly

    • develop an implementation plan

    • identify and manage risks

    • use reliable project management tools

    • develop a clear communications strategy.

     

    Content of implementation plan

    Include in your implementation plan:

    • a timetable for delivering the policy, including key deliverables, targets and milestones

    • the roles and responsibilities of the people and organisations involved in the policy’s delivery and maintenance, including the lead agency and governance structures

    • the resources allocated to implementation (e.g. money, skills and infrastructure)

    • how potential barriers and constraints (e.g. the capability of information technology systems and staff) will be managed, so resources can be allocated appropriately

    • contingency plans to ensure the policy will continue even when the unexpected happens

    • how performance is to be reported and monitored to ensure appropriate accountability

    • the risks and how they are to be managed

    • how project management tools are to be applied to the implementation

    • how the policy is to be communicated

    • reporting arrangements, including:

      • how regularly reports will be prepared (e.g. weekly, monthly or annually)

      • the level at which reporting will be directed (e.g. to the appropriate Minister)

      • the form reports will take (including progress against the milestones or targets and the success criteria).

     

    3. Things to avoid

    To implement your policy successfully avoid the following pitfall that can decrease the policy’s success.

    • Thinking implementation is solely an administrative task by assuming:

      • decisions will automatically translate into action.

      • the policies before implementation were right, so they will not need to be modified for implementation.

      • the controversies and choices in the policy’s development will disappear when the policy is implemented.

      • Other departments or agencies can and will implement policies they have not been involved in developing.

     

    4. Key deliverables and achievements

    At the end of this stage you will have an implementation plan that can be approved by senior management. If another department or agency will implement the policy, senior managers from both organisations must agree the plan.

     
    5. Quality standards

    To be a high quality implementation plan, your plan must include:

    • a timetable

    • key deliverables

    • milestones

    • the roles and responsibilities of those involved in the policy's delivery and maintenance

    • the resources allocated to implementation

    • potential barriers and constraints and how they are to be managed

    • risks and how they are to be managed

    • contingency plans

    • how performance is to be reported and monitored

    • how project management approaches are to be applied to the implementation

    • a communications plan

    • reporting arrangements.​

    step 7 implement

    Step 8. Monitor

     

    The aim of the eighth stage of the policy development process is to develop a monitoring programme that will gather information about a policy’s activities and results.

     

    This guide covers the following topics:

    • Why monitoring is important

    • Key tasks when designing a monitoring programme

    • Key deliverables and achievements

    • Quality standards

    ​

    1. Why monitoring is important

     

    Monitoring provides a record of activities and results. It may signal problems that need to be remedied as a policy is implemented. Information generated through monitoring also informs policy evaluations.

     

    2. Key tasks when designing a monitoring programme

     

    Follow these key steps when considering a monitoring programme:

     

    1. Determine whether the policy initiative needs a monitoring programme:

     

    The Government does not have the resources to monitor its policies. Therefore, it is important to prioritise policies for monitoring. Consider the following eight practical and political considerations when prioritising.

    • Costs: more expensive policies usually justify monitoring programmes.

    • Risks: policies that have greater risks usually justify monitoring programmes.

    • Dispute: policies may be controversial or new, so generate strong interest and receive media and public attention. They may receive bad publicity, be impacted by political activities, be sensationalised or be open to public scrutiny. Therefore, a monitoring programme may be justified to gain reliable information about the policy’s implementation and impact.

    • Impact or performance: policies with substantial opportunity for improvements to their impact or performance usually justify monitoring programmes.

    • Data availability: having insufficient data to undertake an analysis or having insufficient time to gather the data usually means a monitoring programme is unlikely to be effective, so is not justified.

    • Applicability: if policies can be measured or the effectiveness of current operations or future alternatives can be reasonably estimated, monitoring programmes may be justified.

    • Length of operation or time-span: monitoring takes time and the impact of some activities takes time to be felt, so if you expect an initiative to operate for more than a year or two it is worth considering a monitoring programme.

    • Expansion: if plans exist to expand the policy initiative (e.g. it is a pilot) a monitoring programme is usually justified.

     

    2. Determine the monitoring programme’s scope and scale

    The more focused you are about what you want to examine through monitoring, the more efficient you can be, the less time monitoring may take, and the less the monitoring will cost.

    The monitoring programme’s scale and scope depend on the sort of information needed to make major decisions and whether you are planning to evaluate the policy.

     

    3. Collect baseline information

    Before the policy is implemented document the characteristics of the people, conditions or events surrounding the policy initiative. This provides data that can be compared with the monitoring data collected during or at the end of a project to gauge its outcomes and effects.

     

    4. Identify outcome indicators

    Outcome indicators measure the outcomes you are trying to achieve with your policy, usually across social, economic, environmental and cultural dimensions. They are primarily descriptive and often used to assess the ‘big picture’ (i.e. the situation at a societal level).

    Because outcome indicators are based at the societal level, they are unlikely to be within any one organisation’s ability to control, so cooperation and partnerships are required.

     

    An outcome indicator:

    • forms a logical link between the available evidence (what is measurable) and the initiative’s outcomes (what you want to achieve)

    • is sufficiently sensitive to be able to measure changes within the initiative’s timeframe

    • is affordable, in terms of ‘return for effort’ and cost–benefit

    • needs to be stated clearly and concisely

    • needs to make sense to stakeholders.

     

    5. Identify performance indicators

     

    Performance (or output) indicators indicate organisational effectiveness, reflecting the achievement of agencies and specific programmes or initiatives. They are prescriptive, providing a way to evaluate actions. Performance indicators play an important role in how services are delivered and improved.

     

    Develop performance indicators in the context of planned goals and objectives.

    Performance indicators gauge progress toward the intended outcome of specific actions at an organisational, rather than a societal, level.

     

    Performance indicators are closely related to an organisation’s mission and purpose, so are usually within its ability to control. Examples of performance indicators are the proportion of customers satisfied with council libraries and the number of water conservation brochures distributed to households. 

     

    A performance indicator:

    • forms a link between the available evidence (what is measurable) and the initiative’s outcomes (what we want to achieve)

    • is sufficiently sensitive to be able to measure changes within the initiative’s timeframe

    • is affordable, in terms of ‘return for effort’ and cost–benefit

    • needs to be stated clearly and concisely

    • needs to make sense to stakeholders.

     

    6. Decide the data needed for each indicator

     

    For each indicator, identify the data you need collected. Decide how the required data can be gathered efficiently and realistically. Answer the following questions.

    • Is it practical to collect that data?

    • What will it cost to collect that data?

    • Who will collect that data?

    • When will that data be collected?

     

    How you go about collecting the data depends on your selected indicators. However, irrespective of the method you use, develop a standardised recording form (such as that shown in Table 1) and a standardised procedure for collecting the data.

     

    Table 1: Indicator collection guide

    step 8 monitor
    Indicator
    Source of data (records, clients, etc)
    Method (questionnaires, interviews etc)
    Indicator
    Source of data (records, clients, etc)

    7. Determine the data collection method

     

    Decide how the data will be collected by asking the following questions:

  • What data is to be collected?

  • Who will collect the data?

  • How will the data be collected?

  • When will the data be collected?

  • What will happen to collected data?

    •  

    Pre-test your data collection methods (e.g. if you are using a questionnaire have a few staff answer the questionnaire to check whether the questions are understandable and unambiguous).

     

    Use a standardised recording form (such as that shown in Table 2) when collecting the data.

     

    Table 2 – Data collection guide

    Title
    Indicator
    Source of data (records, clients, etc)
    Methods (questionnaire, interviews etc)
    Who collects data
    When to collect data
    Priority three Significant digital investments
    Priority three investments are significant and complex digital investments but are likely focused on only one or a few agencies and hence may not have the same level of cross-government impact.
    Priority one Flagship All-of-Government government digital investments
    Priority one investments are the governments most strategic, significant, and complex digital investments and contribute substantially to the governments digital strategy and cross-government efficiency and effectiveness.
    Priority two Strategically significant digital investments
    Priority two investments are strategic, significant, and complex digital investments but may not have the same level of cross-government impact as priority one investments.

    3. Key deliverables and achievements

    At the end of this stage you will have a monitoring plan that your manager or others can consider adopting.

     

    4. Quality standards

    Your monitoring plan must meet the agency quality standard.

    Step 9. Evaluate

     

    The aim of the ninth stage of the policy development process is to evaluate your policy to establish whether it is appropriate, effective, efficient and economical.

     

    This guide covers the following topics:

  • Why evaluation is important

  • Key tasks when designing an evaluation

  • Things to avoid

  • Key deliverables and achievements

  • Quality standards

    • ​​

    ​1. Why evaluation is important

    Evaluating a policy helps you to determine whether:

  • the policy is producing the desired effects

  • causal links exist between the policy and what is happening in the policy area

  • the policy instruments and settings are wrong or ineffective

  • changing circumstances are affecting the policy objectives or outcomes

  • changes in the policy environment (e.g. emerging research, changing economic circumstances or crises) are marginalising previously important policy assumptions

  • resources are being allocated effectively

  • policies are meeting current the Government’s priorities

  • resources from one policy initiative can be released (through cuts or the

    policy’s termination) to meet new policy demands.

    When resources are finite, a decision to implement one initiative means fewer resources are available for another initiative. Therefore, it is unethical not to evaluate an initiative’s benefits and costs.

     

    • 2. Key tasks when designing an evaluation

    All evaluations involve costs. Even the simplest evaluation takes time away from other activities, so think carefully when designing an evaluation. Create a work plan for conducting the evaluation by following these steps.

    ​

    1. Assess what is being evaluated

    The Government does not have the resources to evaluate all its policies to the same extent. Therefore, it is important to prioritise policies for evaluation. The practical and political considerations in selecting policies for evaluation are similar to those used for monitoring.

    ​

    2. Assess the scale of the evaluation

    Limited resources mean you must prioritise what you want to evaluate; you may not be able to evaluate every aspect of a policy. Decide whether the evaluation will be self-contained or integrated with other activities. The more focused you are about what you want to evaluate:

  • the more efficient the evaluation will be

  • the less time the evaluation will take

  • the less the evaluation will cost for your staff or a consultant’s time.

     

    • Consider the trade-off between the breadth and depth of information the evaluation could gather. For example, if you evaluate a wide range of a policy’s elements (breadth), you are unlikely to be able to investigate any element in great detail (depth) unless you have significant resources. On the other hand, if you examine an element in great detail (depth), you are unlikely to have enough resources to evaluate other aspects of the policy (breadth).

    ​

    3. Decide the type of evaluation

    Evaluations can be informal and tied to routine service or project operations. These evaluations are adequate for ongoing assessments to guide small changes in a policy’s activities, functions and objectives. They promote continual development by providing you with feedback about progress, encouraging you to reflect on outcomes, and providing a basis for you to consider future action strategies.

    ​

    Explicit, formal evaluations are important when the policy’s significance is great. The three types of formal evaluation (formative, process and outcome) are all part of the ongoing evaluation process.

  • Formative evaluations collect, assess and feed back information relevant to planning and operations to inform the policy’s design and implementation. They are used in a policy’s planning stages to ensure it is based on needs and its procedures, methods and materials are effective.

  • Process evaluations assess what a policy initiative consisted of in practice (e.g. what happened, why, where, when, with whom, in what context, at what cost, and how did it produced the results it did). This information aids a policy’s development or extension and ensures accountability for the resources it uses.

  • Outcome evaluations assess whether a policy has met its objectives. They identify the policy’s intended and unintended outcomes.

    •  

    ​

    4. Decide who will conduct the evaluation

     

    Evaluators can be internal staff or external(s).

    ​

    The advantages of using an internal evaluator are that he or she is likely to have:

  • good access to the policy being evaluated.

  • a good understanding of the policy's background, context and initiatives.

    •  

    ​​

    The advantages of using an external evaluator are that he or she may:

     

  • be more objective than an internal evaluator.

  • remain focused on the evaluation for as long as he or she is on contract.

    •  

    ​​

    5. Decide when the evaluation will be reported on

     

    Consider the likely requirement for regular in-house and ministerial briefing papers reporting on the evaluation.

     

    Things to avoid when designing an evaluation. 

    ​

    When designing or undertaking an evaluation, avoid the following pitfalls that can decrease the evaluation’s usefulness.

  • Imposing unnecessary cost burdens in terms of time and financial resources. Evaluations should not be so complex they detract from service delivery or are difficult to complete successfully. Evaluations should complement management processes by gathering the necessary information for improving and accounting for effectiveness. When possible, evaluation activities should draw on established management activities.

  • Assuming a causal pathway. Most evaluations show only relationships, not cause and effect. To show cause and effect you need:

    • a high degree of association between the causal factor and the effect

    • a logical time sequence where the programme precedes the effect

    • to eliminate other possible causes

    • an association that remains consistent when studied in different groups

      and at different times

    • agreement with known facts or theory

    • ​​

    • In some cases, a dose–response relationship (i.e. the more exposure (dose), the greater the effect (response). Most single studies alone do not show cause and effect beyond a doubt. By demonstrating the same results by different researchers over several studies, you can feel more confident in the findings.

  • Using individual cases as indicators of success. Accounts of individual performance are valuable in illuminating aspects of a policy success and can provide rich information for understanding how a programme operates. However, a policy’s success cannot be based on individual case studies. An evaluation at a service or project level needs to compare statistical and qualitative indicators. The overall assessment is usually expressed in terms of rates, percentages, correlations or other statistics.

  • Ignoring unanticipated outcomes. Be alert to unexpected outcomes and address these in the evaluation report as appropriate.

  • Adapting policy or implementation plan to fit evaluation design. Narrowing a project to fit the evaluation design, especially if the policy being evaluated takes a broad, multi-pronged approach, can lead to:

    • an evaluation showing the initiative has had limited effect (because the effects in the complex initiatives occur over a much longer period)

    • promising initiatives being undervalued or at risk of being terminated (because the critical interim outcomes, which are difficult to quantify, are overlooked)

  • Failing to collect base-line data. Document the characteristics of the people, conditions or events surrounding the policy before it is implemented, so you have data for comparison with the evaluation data.

    • ​

    Key deliverables and achievements

  • At the end of this stage you will have an evaluation plan that:

  • outlines what will be evaluated

  • outlines the data that will be collected in the evaluation

  • outlines how the information gathered in the evaluation will relate to each

    aspect of the evaluation questions

  • has been endorsed by the appropriate manager

  • your manager or others can consider adopting.

    ​

    • 5. Quality standards

    Your evaluation plan must meet the following quality standards.

    • The evaluation plan outlines:

  • what will be evaluated

  • what data will be collected

  • how the information gathered will relate to the evaluation questions.

    • step 9 evaluate
    bottom of page