6963736 2014 Evaluation of the SFF Main Report

7/25/2019 6963736 2014 Evaluation of the SFF Main Report

1/69

Evaluation of the SustainableFarming Fund

Main Report

31 January 2014

Prepared for: Ministry for Primary Industries

Prepared by: Judy Oakden, Judy Oakden Consultancy

Julian King, Julian King & Associates

Dr Will Allen, Learning for Sustainability

Published by: Kinnect Group

P.O. Box 2590 Wellington 6410

New Zealand

email:[email protected]

ISBN 978-0-473-27326-2
mailto:[email protected]:[email protected]


2/69

Citation:

Oakden, J., King, J., Allen, W. (2014) Evaluation of the Sustainable FarmingFund: Main Report.Kinnect Group: Wellington

Published by: Kinnect Group

P.O. Box 2590 Wellington 6410

New Zealand

email:[email protected]

ISBN 978-0-473-27326-2
mailto:[email protected]:[email protected]


3/69

3

Contents

Acknowledgements ..................................................................... 6

Evaluation team .................................................................. 6

1 Executive summary ........................................................... 7

2 Introduction ...................................................................... 9

Terms of reference for the evaluation ......................................... 11

Key evaluation questions ........................................................... 12

Methodology ............................................................................ 12

3 The unique value proposition of the SFF ...........................15

Summary ................................................................................ 15

Introduction ............................................................................ 16

Explicit role of SFF versus other funds......................................... 16

Funds of interest ............................................................... 16Strengths of SFF in relation to other funds .................................. 19

Perceived challenges for SFF in relation to other funds .................. 20

4 Has SFF been worth the investment to date?....................22

Summary ................................................................................ 22

Understanding MPI and partner investments ................................ 22

Areas where SFF contributes positively ....................................... 28

Increased capability ........................................................... 29

Development and adoption of new technology ...................... 31

Enhanced relationships ...................................................... 35

Behaviour change with positive environmental benefits .......... 36

Protecting or growing economic value .................................. 39

Unanticipated outcomes ............................................................ 41

Cumulative impacts of successive projects ........................... 41

Building capacity of SFF Project Managers ............................ 42

Mori self-determination supported ..................................... 44

Enablers and barriers to the success of the SFF ............................ 45

Enablers ........................................................................... 45

Barriers. ........................................................................ 47

5 Maximising value for money from SFF ..............................50Investment in MPI Advisers ................................................ 50

Administrative support for grantees ..................................... 50

Harnessing the existing capacity of SFF Project Managers ...... 51

Communications and extension strategy .............................. 51

Reframing eligibility criteria ................................................ 51

In summary...................................................................... 53

6 Appendix A: Context to the SFF ........................................55

7 Appendix B: Evaluative rubric ...........................................56

8 Appendix C: Evaluation methodology ...............................59


4/69

4

Phase One Scoping ................................................................ 59

Phase Two Data collection and analysis .................................... 59

Self-completion questionnaire for SFF Project Managers ......... 60

Key stakeholders ...................................................................... 61

Focus group with Fund Managers ........................................ 61

Case Studies ..................................................................... 61Other data sources included in the review ................................... 64

Limitations .............................................................................. 64

9 Appendix D: Challenges to evaluating and attributing

impacts ............................................................................ 66

10References ....................................................................... 67

Tables

Table 1: Key current policies that link to SFF ............................... 11

Table 2: Total mentions of SFF projects for each region ................ 25

Table 3: Summary of enablers and barriers to the success of the

SFF ......................................................................................... 45

Table 4: Changes in political and operating environment for SFF .... 55

Table 5: Evaluative criteria ........................................................ 56

Table 6: Evaluative rubric defining value for money from the SFF 57

Table 7: Examplesof possible SFF outcomes by Bennetts Hierarchy

level ....................................................................................... 58

Table 8: Summary of data collected for SFF evaluation ................. 60

Table 9: Cases for performance stories ....................................... 62Table 10: Outline of topic guides ................................................ 63

Table 11: Other data sources included in the evaluation ................ 64

Figures

Figure 1:Overview of the key people involved in SFF projects .......... 9

Figure 2: Outcomes Framework for the Sustainable Farming Fund .. 10

Figure 3: Key performance dimensions considered in assessing

VFM ........................................................................................ 13

Figure 4: Focus of SFF and other funding programmes with similar

aims ....................................................................................... 15

Figure 5: Overall landscape of funding from the key organisations

supporting environmental and agricultural innovation and

production in New Zealand ........................................................ 18

Figure 6: Total value of grant applications approved by fiscal year.. 23

Figure 7: Number of projects receiving SFF grants each year ......... 23

Figure 8: Average value of each SFF grant by fiscal years.............. 24

Figure 9: Distribution of SFF projects by duration of time to

completion .............................................................................. 25

Figure 10: SFF projects and funding by sector.............................. 26

Figure 11: Total (all-sources) project funding alongside SFF

funding by fiscal year ................................................................ 27

Figure 12: Social benefits arising from SFF .................................. 29


5/69

5

Figure 13: Extent of technology transfer ..................................... 32

Figure 14: Enhanced relationships from SFF projects .................... 35

Figure 15: Behaviour change with positive environmental benefits . 37

Figure 16: Assessment of the value of the SFF projects ................. 39

Figure 17: Extent to which Project Managers believe SFF

contributes to growing economic outcomes.................................. 40Figure 18: Diverse backgrounds of SFF Project Managers

responding to the online survey ................................................. 43

Fileref: 140131 SFF Main Evaluation Report vxxLast saved: 3-Feb-14

Disclaimer: The information in this report is presented in good faith using the information

available to us at the time of preparation. It is provided on the basis that the authors of the

report are not liable to any person or organisation for any damage or loss which may occurin relation to taking or not taking action in respect of any information or advice within this

report.


6/69

6

Acknowledgements

The evaluation team would like to express their gratitude to many people

who have shared their knowledge to support this study.

Firstly, we acknowledge the insight and support from the staff of the

Ministry for Primary Industries (MPI) who were closely involved in all

stages of the project.

We also acknowledge the wider MPI team who participated in the

development of the evaluative criteria and the sense-making sessions

We also express gratitude to the rural land owners and managers,

representatives from industry organisations and agribusiness, and the

researchers and consultants who took part in the study, either as

representatives of the case study projects or by participating in the online

survey.

Evaluation team

The Ministry for Primary Industries contracted Judy Oakden Consultancy,

a member of the Kinnect Group, to carry out the review. Judy Oakden led

the team and had overall responsibility for the project. The research team

comprised Judy Oakden, Julian King, Kate McKegg and Nan Wehipeihana

from the Kinnect Group in collaboration with Dr Will Allen from Learning

for Sustainability.

Companion Documents

This is the main evaluation report. Also available (from the MPI website)

are:

A summary report (approximately 17 pages)

A report documenting three case studies.


7/69

Evaluation of the Sustainable Farming Fund

7

1 Executive summary

1.

The Sustainable Farming Fund (SFF) invests in grass-roots projects with

the aim of delivering economic, environmental and social benefits. The

SFF was evaluated in 2013 to assess outcomes, value for money and

possible adaptations to ensure the Fund remains effective and fit forpurpose.

2.

MPI commissioned this evaluation of the SFF portfolio to provide

assurance of the outcomes and value for money to date (200013); and

to capture learnings and an evidence base for possible adaptations, to

ensure it is fit for purpose in the future. The evaluation uses an

evaluation specific methodology1based on an outcomes framework,

rubrics and mixed methods including economic methods. This

methodology differs from a cost benefit analysis in that a wider range of

criteria are considered in determining the extent to which the SFF was

value for money.

3.

The evaluation found that the SFF is good value for money and

makes a worthwhile and valuable contribution to primary

industries and rural communities.It supports the interests of science,

the environment, agribusiness and the community in ways not replicated

by other funding programmes.

4.

Available evidence indicates that SFF funds are being allocated and used

in accordance with the intended purpose and strategic priorities of the

SFF, and that the SFF has a track record of investing in successful

projects. Furthermore, the SFF has contributed to encouraging significantpartner co-investment in these projects.

5. Among the key benefits of the SFF are its contribution to:

increased capability for problem solving at individual, community and

sector levels

enhanced relationships and networks between farmers, rural

communities, scientists, local government and industry bodies

behaviour change that supports sustainable farming, including farmer

engagement and emergence of leaders to champion ongoing change

development and adoption of new technology and environmentally

sustainable practice

1For further information on what constitutes an evaluation specific methodology see

the following publications:

Davidson, E.J (2013) Evaluation-Specific Methodology: the methodologies that aredistinctive to evaluation. GenuineEvaluation. Retreived 20 December 2013

fromhttp://genuineevaluation.com/evaluation-specific-methodology-the-methodologies-that-are-distinctive-to-evaluation/

King, J., McKegg, K., Oakden, J. & Wehipeihana, N. (2013) Rubrics: A Method forSurfacing Values and Improving the Credibility of Evaluation.Journal of

Multidisciplinary Evaluation, 9:21,11-20.Scriven, M. (2008). A summative evaluation of RCT methodology: & an alternative

approach to causal research.Journal of Multidiscipl inary Evaluation, 5,11-24.
http://genuineevaluation.com/evaluation-specific-methodology-the-methodologies-that-are-distinctive-to-evaluation/http://genuineevaluation.com/evaluation-specific-methodology-the-methodologies-that-are-distinctive-to-evaluation/http://genuineevaluation.com/evaluation-specific-methodology-the-methodologies-that-are-distinctive-to-evaluation/http://genuineevaluation.com/evaluation-specific-methodology-the-methodologies-that-are-distinctive-to-evaluation/http://genuineevaluation.com/evaluation-specific-methodology-the-methodologies-that-are-distinctive-to-evaluation/


8/69


8

protecting and growing the economic value of primary industries,

including export opportunities.

6. In addition to achieving its aims, the SFF has contributed to unanticipated

benefits, including:

cumulative impacts from a number of related SFF projects over time,

with later projects building on learnings from earlier ones

development of skilled Project Managers who help broker

relationships, support individual and group change, and facilitate

project management

supporting Mori self-determination.

7.

Enablers and barriers to the success of the SFF have been identified to

guide future programme development. Opportunities to gain further value

for money from the SFF include: investment in MPI Advisers,

administrative support for grantees, harnessing the capacity of Project

Managers, a communications strategy and reframing eligibility criteria.


9/69


9

2 Introduction

8.

The Sustainable Farming Fund (SFF) invests in farmer2-led projects that

deliver economic, environmental and social benefits to New Zealands

primary industries and rural communities. The Fund was set up in 2000

and so has now been in existence for 13 years. Aquaculture was added in2011.3

9.

Its operation is based around annual funding rounds, which are run by

Ministry for Primary Industries (MPI) staff. Projects typically run for one to

three years, and the Fund invests up to $200,000 per annum per project

(see paras 52 - 68 and figures 6 - 11). The following diagram provides an

overview of the key people involved in the operation of SFF projects.

Figure 1:Overview of the key people involved in SFF projects

10. The SFF is a complex intervention operating in a range of primary sectors.

Projects have been funded with the broader intention of seeing them

contribute to: improving the economic performance of land-based sectors,

improving environmental outcomes and supporting rural communities.

The SFF initiatives and projects respond to the needs of multiple

stakeholders and relationships in their particular contexts, and

intermediate outcomes (both expected and unexpected) emerge over

time.

2The word farmers refers to farmers, growers, fishers, foresters and aquaculturalists,and this includes business owners and managers.3For further background on the Fund, please refer to Appendix A.


10/69


10

11.

Overall, the SFF aims to provide support to farmers to:

develop skills and capabilities to tackle shared problems and

opportunities

facilitate the development of broader relationships between them

and industry organisations, scientists and other consultants to

support the building of resilient communities

help position participants to leverage other funds

support technology transfer

champion projects that address the potential for market failure a

public good aspect of the intended outcomes from SFF.

12.

The following Outcomes Framework provides an overview of the theory of

how change occurs for communities of interest associated with a project

as a result of support from the SFF.

Figure 2: Outcomes Framework for the Sustainable Farming Fund

13.

Over the course of the Fund, the political and operating environments

have evolved, requiring the Fund to demonstrate how it contributes to,

and aligns with more recent Ministry and Government-level strategies and

policies. These clearly spell out desired long-term national and sector

outcomes in terms of economic growth, risk management and

environment.


11/69


11

Terms of reference for the evaluation

14.

MPI commissioned this evaluation of the SFF portfolio in 2013 to provide

assurance of the outcomes and value for money to date (200013). MPI

also sought an evidence base for possible adaptations to the SFF portfolio

design, to ensure it is fit for purpose in the current and future fundinglandscape. In addition, MPI wished to learn how it could improve SFF

implementation and the Ministrys collaborative partnering with related

funding programme managers. (Although the findings below are

expressed in the past tense, the SFF continues to operate.)

15.

The objectives of the evaluation were to:

tell the story of how well the SFF portfolio has performed, achieved

intended outcomes and provided value for money

identify SFFs key barriers, drivers and learnings

compare SFF against other like public funding programmes, and

evaluate the extent to which the SFF design remains fit-for-

purpose or needs to adapt

provide added-value elements to the MPI business, such as

collaborative partnering, demonstrating the innovative use of

evaluation, and/or building evaluative capability with MPI staff

(Ministry of Primary Industries, 2013, p. 1).

16.

This evaluation was designedto align with Government policies and

strategic priorities that intersect with the SFF project. The following table

shows the key policies that were taken into account in undertaking this

evaluation.

Table 1: Key current policies that link to SFFRelevantpolicies

Key points

NZ Govern-ments BusinessGrowth Agenda

(BGA)

Includes goal of lifting exports as a percentage of NZs GDP from 30%to 40% by 20254

Recognises that increased value from primary industries is critical toachieving this goal

Includes themes of Building Innovation (e.g. doubling businessexpenditure on R&D to more than 1% of GDP), building exports (with

targets for value and growth of exports), and regional economicdevelopment. http://www.mbie.govt.nz/pdf-library/what-we-

do/business-growth-agenda/rear/REAR%20Publication.pdfMPI Strategy

2030Includes two points of focus:

maximise export opportunities and improve sector productivity; and Increase sustainable resource use and protect from biological risk.

Strategies to achieve these include (among other things):

partnering with primary sectors to identify and seize opportunities for

improved productivity and market returns, while ensuring that growth isenvironmentally sustainable

encouraging and co-investing in industry innovation and adoption

partnering innovative approaches to environmental challenges.

engaging with Mori to improve economic returns for Mori and NZ as awhole.

MPI Mori Agri- This was a focus of SFF funding in 2012. See the announcement of the

4Achieving this means doubling the value of New Zealand's annual primary sector-based merchandise exports to $64 billion in real terms by 2025. This has becomeknown at MPI as the Ministrys Export Double goal.


12/69


12

business

Strategyspecial funding round:http://www.mpi.govt.nz/news-

resources/news/funding-to-promote-sustainable-resource-use-in-pri

Key evaluation questions

17.

The evaluation addresses the following three questions, which provide theoverarching structure for this report:

What is the current-day, unique value proposition of the SFF

relative to other funds?

To what extent and in what ways has the SFF been worth the

investment to date?

What are the opportunities to maximise the value derived from the

SFF?

Methodology

18. This uses an evaluation specific methodology and the evaluation draws

together evidence from a range of both quantitative and qualitative

sources including:

a review of the SFF project database of 906 projects

a survey of 136 SFF Project Managers (with experience of

approximately 400 projects)

case studies of three clusters of SFF projects (see box summaries

in this report below paras 90, 118 and 123; and the full case

studies in the companion document)

workshops with MPI staff and Fund Managers of MPI Funds and

other relevant funding programmes

a review of past evaluations5and other SFF documentation.

19.

Evaluation is the systematic determination of merit, worth or significance

(Scriven, 2012). An evaluation-specific approachwas applied for this

project, which is an approach that provides robust information about how

goodthe evaluation subject is, whether it is good enough, and how it can

be improved(Davidson, 2005).

20. Value for moneyin this evaluation refers to whether government and

SFF beneficiaries are using their combined resources well. Funding and

other resources (such as time, knowledge and skills) are limited. There is

an opportunity cost associated with their use. It is therefore desirable to

allocate resources to activities that return high-value outcomes.

21. Evaluation of value for moneycombines the above concepts. It

provides robust information about whether something is valuable enough

5BERL (2004), Barton (2002); MPI (2010) Ten Years of Grassroots Action.
http://www.mpi.govt.nz/news-resources/news/funding-to-promote-sustainable-resource-use-in-prihttp://www.mpi.govt.nz/news-resources/news/funding-to-promote-sustainable-resource-use-in-prihttp://www.mpi.govt.nz/news-resources/news/funding-to-promote-sustainable-resource-use-in-prihttp://www.mpi.govt.nz/news-resources/news/funding-to-promote-sustainable-resource-use-in-pri


13/69


13

to justify the government and SFF recipients resources used. This

provides the basis for addressing KEQ 2: To what extent has the SFF been

worth the investment to date?

22.

For this evaluation, Cost Benefit Analysis (CBA) was not considered

appropriate, as the evaluation scope was broader than economic valuealone and it was felt that a CBA might not adequately describe the full

value of the SFF.6The SFF is specifically designed to improve economic,

environmental and social performance, and is also cognisant of the

cultural environment of communities. Therefore this study incorporated

economic, social, environmental and cultural dimensions in addressing

VFM. Figure 3 conceptualises how the different aspect of performance

relate to one another.

Figure 3: Key performance dimensions considered in assessing VFM

23. Rubrics (a kind of performance framework) provided a process and

conceptual framework for integrating economic and other dimensions of

value within an evaluation that drew on the conceptual ideas in Bennetts

Hierarchy7. The evaluators worked collaboratively with MPI managers and

staff to develop evaluative performance criteria that specified what

excellent, very good, good and adequate value for money might

look like in terms of the intermediate outcome areas to be identified in the

outcomes framework, within the timeframe of interest. More informationon this process is included in Appendix B.

6Further, CBA would be difficult to apply credibly to this evaluation because it would

necessitate extrapolation of long-term economic outcomes from intermediate outcomesand attribution of some proportion of these to the SFF, and then require all outcomesto be valued in dollar terms. Significant assumptions would have to be made which,

though potentially useful for scenario analysis, would fall short of providing credibleevidence to evaluate the extent to which the SFF actually represents VFM.7Bennetts Hierarchy identifies the higher order outcomes that might occur in a well-

established project, as well as the earlier outcomes that might be expected, andseparates these from the inputs, activities and outputs that are undertaken as part ofthe project. Level 5 in Bennetts Hierarchy includes short-term and local outcomes,

Level 6 includes medium term outcomes at a wider level, and Level 7 includes longer-term outcomes at a national or industry level. A more complete description of theselevels can be seen in Appendix B, Table 11.


14/69


14

24. The evaluation methodology was submitted for approval to the Ministry

and its Quality Assurance (QA) group for the project (comprising the

internal Ministry project team, two internal experts on MPI funds, and

three evaluators from MPI and other government agencies) in

MarchApril 2013.

25.

More detail on the analysis of the SFF database, the online survey of SFF

project managers, the development of the three case studies and the

desk research undertaken as part of this project can be found in Appendix

C: Evaluative research methodologies. This section also covers limitations

of the study.

26. The evaluation was undertaken between March and October 2013.


15/69


15

3 The unique value proposition of the SFF

27.

This section of the report summarises detailed findings to address the first

Key Evaluation Question: What is the current-day, unique value

proposition of the SFF relative to other funds?

Summary

28.

The SFF is unique as it supports the interests of science, the environment,

business and the community in ways that are not replicated by other

funding programmes. The unique value proposition of the SFF relative to

other funding programmes is that it supports grass-roots, science-based

initiatives to protect and grow primary production businesses and rural

community resilience.

29.

The evaluation found the SFF remains fit for purpose and makes aworthwhile and valuable contribution to primary industries and rural

communities.8The following diagram illustrates the unique position of SFF

compared to other funding programmes with similar aims.

Figure 4: Focus of SFF and other funding programmes with similar aims

8Data which informs this aspect of the evaluation for comes from:

a focus group with six participants comprising Senior Managers and Fund Managersfrom a range of Funds, held at MPI in September 2013

documents related to the Natural Resources Sector Review including: NRS Non-Departmental Funds Duplication and Overlap: A report completed for Stage II of

the NRS Non-Departmental Funds Review (Ministry for the Environment andMinistry for Primary Industries, 2013),Natural Resources Sector: Stage 2 Funds

Analysis Review: Opportunities for efficiency and effectiveness and reprioritisationof NRS Non-Departmental Funds (Deloitte, 2013)

review of the websites for some of the key funds.


16/69


16

Introduction

30.

There are a range of funds that support the primary sector. The diagram

on page 18 outlines funds administered by Ministry for Primary Industries

(MPI), Ministry of Business, Innovation and Employment (MBIE), Ministry

for the Environment (MfE), Department of Conservation (DOC) and theTertiary Education Commission (TEC). It also notes additional assistance

available from New Zealand Trade and Enterprise (NZTE) and funds

available from the Callaghan Institute, the private sector and the

philanthropic sector.

Explicit role of SFF versus other funds

31. Overall, the funds that operate in a similar area to SFF tend to focus

around at least one of four key areas:

Science: the production of knowledge to solve problems, innovate,

and protect or improve productivity

The environment: focusing on understanding environmental issues,

environmental sustainability, renewal, etc

Business: improving sector productivity, being more innovative,

raising value of exports

The community: supporting communities to solve their own

problems, building community capacity and capability.

Funds of interest

32.

Funds that were identified as operating in a similar area to SFF are:

the Primary Growth Partnership and the Climate Change

Development Funds (MPI)

Biological Industries Research, Bioprotection Core, Environment

Fund, CRI Core Funding, Partnerships Fund, and the National

Sciences Challenge (MBIE)

Community Environment Fund, Fresh Start for Fresh Water (MfE)

Biodiversity Advice Fund, Biodiversity Condition Fund and

Weedbusters (DOC)

Performance-Based Research Fund (TEC)

Trade and Enterprise support for export initiatives

Agricultural and Marketing Research and Development Trust

(Agmardt)

other private-sector philanthropy


17/69


17

private-sector industry groups including Beef + Lamb New

Zealand, DairyNZ, the Foundation for Arable Research (FAR) and

Deer Industry New Zealand (DINZ).

33.

The following diagram shows the key funding organisations supporting

environmental and agricultural innovation and production in New Zealand,and the other funds they offer. Those that were identified by Fund

Managers, from the literature and from web searches as operating in a

similar area to SFF have a red border. (The three Department of

Conservation Funds with a brown border also appeared similar to the SFF,

based on searches of their websites.)


18/69


18

Figure 5: Overall landscape of funding from the key organisations supporting environmental and agricultural innovation and production inNew Zealand

Note: Funds that were identified by Fund Managers, and from the literature, and from web searches as operating in a similar area to SFF have a red border.


19/69


19

Strengths of SFF in relation to other funds

34. Feedback from the focus group of Fund Managers reinforces evidence and

perceptions by key stakeholders that the SFF is a bottom-up fund that

helps build community cohesion. It often provides training and enablespeople to work in groups to a common cause. SFF is seen to have good

output farmers are active and get things done. According to Fund

Managers the SFF appears to be used in three different ways:

to build incremental capability and capacity of both people and

ideas a mature example is illustrated in the Protecting the

sustainability of New Zealand vineyardscase study (see case

below para 118), and an early example is illustrated in the

Sustainable development and podocarp restoration on Tuawhenua

landscase study (see case below para 123)

to scale capability and capacity across regions as illustrated by

the Top of the South: Setting an example for sustainable water

quality case study (see case below para 90)

one-off projects of interest to a particular community.

Benefits of SFF

35.

Fund Managers and SFF Project Managers believe that the SFF offers a

number of benefits. Firstly, SFF is the only funding mechanism that

enables farming communities to address opportunities and solve problems

with the support of both science and business.

SFF projects are a critical vehicle for using quality science to address farmer issues

in a direct, focused and practical way. [Project Manager]

36.

Both Fund Managers and Project Managers observed that the SFF process

encourages groups to work together so farming communities really

grapple with science to solve practical environmental problems that they

face, and as a result significant development occurs within the

community. The evaluators also found SFF supports applied research at a

local level indeed all the case studies we have developed include clear

illustrations of this principle. A Project Manager summed up the benefits:

The Fund has supported important innovation in New Zealand farming systems. It

provides a way to address real and practical farming community issues. The SFF is

one of the few funds available to NZ farmers and growers that supports farm

extension and adoption. [Project Manager]

37. SFFs flexiblility in the way projects are administered, in order to respond

to the needs and pace of the community, was acknowledged by Fund

Mangers and Project Managers. For instance, Project Managers said that if

new learnings emerge, projects are able to alter their funding contract

milestones to accommodate these learnings. However, they also reported

there are sound administrative processes to ensure the funds are used as

planned and adequately accounted for.


20/69


20

SFF has always been highly supportive and endeavoured to make sure the

community of interest's needs are met. I think a key to SFF projects being

successful is that they are able to do this without putting unrealistic expectations

onto project teams if, for example, circumstances were to change unexpectedly or

if emphasis on a certain objective needed to shift due to project results. A structure

that is too rigid to accommodate fluctuations would limit the science and outputs

achieved whilst creating unneccesary stress on project teams. I am grateful thatmy experience with the SFF has always been supportive and enjoyable.[Project

Manager]

The application process is very good, with allowance for flexibility to change

milestones as the project develops which is vitally important. The process of

developing the project, once accepted, to the contract stage is excellent and

efficient. Changes have taken place to improve financial reporting, which is good.

Simple systems work best for all. [Project Manager]

38.

Fund Managers and MPI Project Advisers commented that the SFF

provides significant visible support at a regional level, which would be

noticed and missed if the SFF did not exist, and they believe this is

appreciated at a political level by Ministers. Fund Managers responsible for

a range of funds in the funding landscape (both within MPI and from other

organisations) believe that those seeking funding would seek MfE

Community funding instead if the SFF did not exist, reinforcing the view

that MfE Community Environment Fund is the nearest fund to SFF.

39.

The evaluators found SFF supports clusters of projects around common

problems and enables a wide range of industry sectors to build capability

incrementally over time for instance as has occurred in the wine

industry.

40.

Furthermore, the evaluators identified from a recent MPI evaluation of itsMori Agribusiness workthat the flexibility of SFF funding supports Mori

to harness the productive potential of Mori resources. The evaluators

found evidence that the SFF prepares Mori agribusiness to engage with

other funding by helping build relationships between iwi, scientists and

industry sectors.

Perceived challenges for SFF in relation to other funds

41.

Fund Managers acknowledged that while the SFF helps build community

capacity, this takes time. This is a challenge because the benefit of anyfunding is not immediately apparent.

42. Fund managers also acknowledged a negative perception that at times

the same projects appear to be simply repeated in a number of regions.

There is a range of complexity of projects. Some SFF projects have

developed technologies that provide an immediate cost-effective solution

to a largely private-good problem (e.g. an improved spray system for

horticultural enterprises) and can fairly easily be incorporated into the

existing production system. This contrasts with more complex situations,

such as those faced by catchment communities trying to improve water

quality. In these cases, the solutions are multi-faceted, may require there-design of farming systems and often require negotiations and collective


21/69


21

actions by a number of stakeholders (Allen et al. 2002). In these complex

situations communities need to learn things for themselves and develop

unique collective and individual responses (Douthwaite et al. 2001). This

shows why certain types of projects cant leap-frog early stages of project

development based on the learnings of other peoples projects, even

though something similar has been done elsewhere.

43.

Fund Managers did not see a clear current link between the SFF projects

and a contribution to exports. They believed that if the SFF did not exist

export earnings would not be directly affected. Fund Managers

questioned, therefore, whether the right activities are being supported

and whether there could be better leveraging between the SFF and other

funds to support better contribution to export earnings. However,

feedback from Project Managers indicates that the SFF is supporting

exports, albeit in many instances by its contribution to protecting

underlying sustainability and quality. This is covered in more detail

starting on page 36.

44.

Fund Managers believed there needs to be a more targeted and more

strategic approach to the Fund, while retaining its bottom up access. A

few Project Managers noted the re-focus on economic benefits, to align it

with aspects of the Governments wider strategic focus, and felt an

enviromental focus was also needed, which would also align with the

Governments strategies.

It seems SFF have gone away from sustainable and toward profit, so less

concerned about wider environmental performance; more concerned with economic

performance. This makes it more difficult to work on projects which have longer

term benefits and require seed funding in preference to those which have cleaer

and quicker economic benefits but less clear environmental benefits. [Project

Manager]

45.

Fund Managers thought it would be good if there was a form of

information about the different types of relevant funding, including SFF,

set out on a development continuum. There was a sense that currently

projects need somewhere to go after SFF though it was acknowledged a

next level up fund does not currently exist. The PGP, for example, was

seen to be too large a leap from SFF to be viewed as a next logical step

for these projects, as observed by this Project Manager who saw SFF as

an essential funding source:

I think the MPI SFF is an essential funding source for the primary industries. It

bridges a gap between the larger funds, PGP and MBIE, and accommodates

relatively short-term projects of an applied nature that will deliver benefits to the

industry. It is an increasingly important funding source for sectors and industry

groups. [Project Manager]


22/69


22

4 Has SFF been worth the investment to date?

46.

This section summarises detailed findings and addresses the second Key

Evaluation Question: To what extent and in what ways has the SFF been

worth the investment to date?

Summary

47.

Overall, the evaluators found SFF provides good value for money, taking

into account the range of social, environmental, cultural and economic

benefits to grantees and their communities.

48.

MPI spent $122.8 million on 906 SFF projects across 14 fiscal years

(200001 to 201314). Case studies, survey feedback from SFF Project

Managers and other available information including past evaluations9

indicate that the SFF has a track record of investing in worthwhile andsuccessful projects.

49. The SFF has contributed to encouraging significant co-investment from

partners in industry innovation and adoption. For example, partners

contributed $1.27 of financial resources for every $1 of SFF funding as

well as further contributions in kind.

50. SFF also contributed to encouraging farmer, grower, forester and

aquaculturalist-led projects to partner innovative approaches to

environmental challenges,as identified in the case studies.

51.

Among the key benefits of the SFF are its contribution to:

increased capability for problem solving and project management

enhanced relationships and networks that support innovation

behaviour change

development and adoption of new technology

protecting or growing economic value.

Understanding MPI and partner investments

52.

The SFF project database was analysed to better understand the nature of

investment in the Fund. Overall, MPI allocated $122.8 million on 906 SFF

projects across 14 fiscal years (200001 to 201314).

53.

The following graph shows that after the first year (when the total was

lower), grants of between $8m and $10m were made each year. Funding

allocations were fairly consistent from 20012008. Since then there has

9Including past evaluations of the SFF: BERL 2004, Barton 2002; and the 2010 MPI

publication Ten Years of Grassroots Action.


23/69


23

been more variation in funding, with the highest level of funding occurring

in 20082009 and the lowest in 20102011.

Figure 6: Total value of grant applications approved by fiscal year

54.

However, there was greater variation in the number of projectsreceiving

SFF grants each year, as the following chart shows. Apart from the first

year, the lowest number of grants was for the current fiscal year

20132014.

Figure 7: Number of projects receiving SFF grants each year

55.

The average size of individual grants has moved within a given range over

the years, from just under $100,000 to around $160,000 over the first 13

years. The evaluators note that in 20132014 the SFF moved toward


24/69


24

awarding fewer grants but allocating more money to each project

compared with previous years, with an average annual allocation of over

$200,000 for the first time.

Figure 8: Average value of each SFF grant by fiscal years

Funding duration

56. The expected timeframe for SFF projects is between one and three years.

As the following graph shows, the majority (79%) of SFF projects were

completed within three years. Indeed, the median actual duration of

funding was less than three years, at two years and two and a half

months.

57.

Of the nearly one in five projects (17%) that took more than the initially

planned three year maximum to complete, most (15%) were completed

within four years. Just 2% of SFF projects took more than four years to

complete. The longest time taken for a project was five years and eight

months. This indicates that the current strategy of allowing more time is

appropriate at times and that projects can almost always be completed

within a five-year time frame.


25/69


25

Figure 9: Distribution of SFF projects by duration of time to completion

Representation of regions

58.

To assess the extent to which the projects were spread across regions,

the evaluators analysed the SFF project database to determine how many

projects could be attributed to each region. Overall, projects could be

described as nationwide in total 36% of the time. The regions with the

most mentions were Canterbury (18%), Hawkes Bay (9%), Otago (9%),

Waikato (9%), Bay of Plenty (7%) and Northland (7%).

Table 2: Total mentions of SFF projects for each regionRegions % of Projects % of Funding

10

Nationwide 36% 35%

Canterbury 18% 20%

Hawkes Bay 9% 10%

Otago 9% 9%

Waikato 9% 10%

Bay of Plenty 7% 9%

Northland 7% 5%

Southland 5% 6%

Nelson 5% 4%

Marlborough 5% 5%

Auckland 3% 4%Manawatu 3% 3%

Gisborne 3% 3%

Taranaki 2% 3%

West Coast 2% 2%

Wellington 1% 0%

Chatham Islands 0% 0%

No response 2% 3%

Total 124% 131%

10The % of funding column includes all funds that are fully or partially attributable tothe region and therefore include some double-counting where a single project has beenattributed to more than one region (and hence the column totals greater than 100%).


26/69


26

59. The second column of the above table also shows that the regions with

the most projects also received the most funding overall.

Sectors

60.

To assess the extent to which the projects represented the primary

sectors, the evaluators analysed the total mentions for each sector in the

SFF projects database.

Figure 10: SFF projects and funding by sector

61.

It is clear that the pastoral and horticultural sectors undertook the

greatest number of projects and received the greatest amount of funding,

followed by the arable sector, cross-sectoral initiatives, water sector and

forestry sector.

62.

In further analysis the evaluators assessed the extent to which funded

projects worked for a single sector or more than one sector. Overall 73%

of the total SFF funding was attributed exclusively to one sector, while the

remaining 27% SFF funding was for projects that operated across sectors.

Attracting additional industry resources

63.

On average, for every $1 invested by the Ministry of Primary Industries

through the SFF, a further $1.27 in funding was contributed by others.

The following graph shows total project funding that is from all sources,

alongside SFF funding, by fiscal year.


27/69


27

Figure 11: Total (all-sources) project funding alongside SFF funding byfiscal year

64.

In addition to the above, partners have often contributed in-kind 11

resources. The database only provided the value of in-kind contributions

for 33 recent projects, and it is unclear to what extent these may be

representative of all SFF projects. Furthermore, it is unclear whether the

full value of time and support has been included in the values recorded.

These 33 projects had an aggregate value of $42.9 million in SFF funds

and received $0.8 million worth of in-kind contributions. Thus, for every

$1 of SFF funding, the recorded value of in-kind contributions for these 33

projects averaged $0.19, close to 20%.12Anecdotally, MPI Advisers

consider the in-kind contributions of SFF projects are likely to be much

higher than 20%.

65.

In-kind contributions can include farmers time, but at times also includes

things like stock.

Farmers were willing to contribute financially to the project as well as supply lambs

in good faith, to find out a little about relationships between pasture diversity and

meat quality.[This] showed there was keen interest in learning more about [therelationship between] diets and livestock performance I negotiated that the

farmers could get their lamb carcasses back, which was really appreciated at the

time. It may seem trivial, but there are attitudes in the industry that academics

take advantage of the generosity of farmers, and this was a classic case where the

scientists only needed a [sample] from an entire carcass for analysis. [Project

Manager]

11In-kind resources include, for instance, the SFF project teams time.12

Anecdotally, MPI Advisers consider the in-kind contributions of SFF projects are likely

to be much higher than 20%.


28/69


28

Intangible investments

66.

MPI has made a considerable investment in building robust internal

processes and efficiently running the Fund. In the early stages of the SFF,

MPI was particularly careful to establish a system of accountability to

ensure that funds were spent as intended. The system is seen as

straightforward and pragmatic.

The Fund has been well managed. The role of MPI staff through the process, from

prior to submission of the proposal to involvement in project team meetings,

means the expectations are well managed and the outcomes to help the industry

are readily delivered. In my view a clear strength has been the interest, knowledge

and engagement of the Project Managers in ensuring projects deliver the agreed

outcomes. [Project Manager]

A major advantage of SFF is the largely straightforward application and claim

process. Other grant organisations can often be complex and extraordinarily fiddly

to work with. We would be extremely concerned if these application and claim

processes were to change. [Project Manager]

67. SFF milestone reporting captures the funds provided by MPI and also

contributions from other funders and in-kind contributions. This means

there is detailed information, in narrative form, for many of the projects

regarding the leveraging they achieved both in attracting other funders,

and in the time and support they achieved from farmers and local

communities. (However, as noted above, this was only recorded in the

database for 33 of the projects at the time the database was made

available to the evaluators.)

68.

There has also been significant investment from MPI staff, particularly the

MPI Advisers, in SFF projects. Initially MPI Advisers were very involvedwith the projects and had considerable face-to-face contact with the

project leaders. It is evident that, despite a simple system of

accountability, a number of projects required additional support to get

started. As MPI Advisers provided support to set up projects and help

them successfully become established, they also built strong relationships

with key project team members.

This was a large, early project [that started in 2002] and the science provider and

the grower groups involved did not have structures set up to manage the finances

and reporting of the project. These are well-established now. [Project Manager]

Areas where SFF contributes positively

69.

The following sections outline the areas identified by the evaluators where

SFF is seen to have made a positive contribution, either socially,

economically, environmentally or culturally.


29/69


29

Increased capability

70. The survey of SFF Project Managers,13the case studies, and feedback

from MPI Advisers and a range of Fund Managers provides good evidence

of the SFF contributing to social and organisational capacity and

capability-building at individual, community and sector levels.

71.

The following graph shows some of the findings from the online survey

and illustrates the extent to which SFF Project Managers saw clear social

benefits of capability building arising from SFF projects.

Figure 12: Social benefits arising from SFF

Building knowledge and skills

72.

The majority of Project Managers (91%) said the SFF built the project

teams knowledge and skills, and the teams then used new skills to share

new knowledge and technologies with others.

The key outcome of the project was to deliver a concise, practical field guide to

growers. Although the project had enormous width, distilling the key messages and

testing these with focus groups enabled the concepts we were communicating to

be clear and useful. The project relied heavily on photographic recording and

reporting and allows quick visual comparison in the field and identification of

problems. [Project Manager]

73.

According to Project Managers, SFF Advisers and Fund Managers, SFF

grants help communities to build capability by supporting them to identify

their own challenges and opportunities and then to come up with ways

to address them, as the following example from a Project Manager

illustrates.

13SFF Project Managers surveyed as part of this project represented a diverse range of

stakeholders including farmers, or staff from industry bodies, community groups,Crown Research Institutes, rural consultancies, universities, local government or Mori

trusts or incorporations.


30/69


30

[The Top of the South case has showed that] what is important is that communities

define their own problems clearly, in terms they can understand. This thinking has

now been incorporated into other catchment initiatives around the country.

[Industry Representative]

74.

During SFF projects, farming communities have become aware of feasible

approaches to farming practice. These then harness resources for optimalproduction whilst protecting the environment, in collaboration with

scientists and other experts. They learned to accommodate the needs of

other stakeholders and regulators, and to safeguard their industries for

the longer term. The case Protecting the sustainability of New Zealand

vineyardsis an excellent example of social and organisational capacity

and capability-building occurring at individual, community and sector

levels. During this project, wine growers:

learned about ways to implement sustainable production

took part in developing a Sustainability Policy (SWNZ) for theindustry

took part in promising initiatives to eradicate leafroll virus

explored with scientists how mechanical thinning might be a viable

tool.

75.

The Fund Managers spoken to for this evaluation believe SFF encourages

farming communities to really grapple with science to solve practical

environmental problems that they face. This was also evident in the case

study Sustainable development and podocarp restoration on Tuawhenua

lands. This case provides clear examples, even at the early stage of thisproject, where SFF projects supported Tuawhenua Trust members and

hapu to develop the skills and capacities to:

obtain a greater understanding of the ecological challenges of their

forest, and how these challenges might be approached

gain a better understanding of the feasibility of different timber

extraction and milling options

identify areas for further planning and market development.

76.

MPI Advisers and Project Managers agreed that in many instances SFFprojects also assisted the project team members to develop transferable

skills in project management, proposal development, project

administration, and operational and financial management.

Accommodating the needs of Mori to build capacity

77.

Feedback from the survey and case studies indicated the SFF is flexible

enough to accommodate the needs of Mori but has been under-utilised

due to a range of barriers (see para 140 below). Moriare owners of

significant assets such as land that can be used for primary production

but do not always have access to the skills, services or technologies todevelop them. In 2012 MPI organised an out-of-cycle funding round


31/69


31

specifically for Mori agribusiness projects, including support to develop

and implement successful projects. This was successful in attracting

targeted applications. There is scope to further support Mori agribusiness

needs and aspirations.

Development and adoption of new technology

78.

There is evidence that SFF projects are contributing to increased

development and adoption of new technology.

79.

Another key social outcome is the SFF project support for the building of

knowledge and its transfer out to other communities and stakeholders.

[A highlight was] seeing a technique developed in our vegetable garden taken up

by farmers from Southland to Northland. [Project Manager]

80.

Three quarters (77%) of Project Managers surveyed said this increased

capacity went on to influence a wider audience at a national level.

[We were] able to demonstrate to DairyNZ's Strategy and Investment Leader the

differences in the greenhouse gas footprint for the different farming systems. This

information will be used by DairyNZ to help farmers make more informed decisions

on wintering-off systems. [Project Manager]

81.

At times the learnings gained are also about what did not work, i.e.,

failure is valuable in itself, identifying what science or directions to not

pursue or invest in further. Project teams appreciated that SFF is willing

to acknowledge the value of this.

This was a very complex project, but despite this it was well-structured. While the

project did not find a specifi c solution, it eliminated many assumptions andprovided a set of specific guidelines for future reference. [Project Manager]

82.

The evaluation also found that support from the SFF projects contributed

to the development of a number of community groups that have

continued beyond the life of the projects.

In our sector the SFF funding has been a critical part of the formation of sector

groups that have endured beyond the life of the funded projects. In an

economically tough, predominantly non-export production sector, SFF funding has

been almost the only form of assistance available to grower groups to get larger

projects off the ground. The simplicity of the application and reporting processes

and the willingness to assist funding of producer-led practical research and

extension have motivated grower groups in our sector to keep putting up projects.

[Project manager]

83. SFF Project Managers believed participants are now better informed about

successful farming or growing techniques (88%) and better able to apply

them (81%). SFF Project Managers also believed that the SFF project

assisted in implementation and use of effective on-farm management

practices and systems (80%). The following graph shows the extent SFF

Project Managers believe there has been successful technology transfer.


32/69


32

Figure 13: Extent of technology transfer

84.

Both MPI Advisers and SFF Project Managers provided many examples of

technology transfer amongst the farming community and maintained the

SFF grants supported communities to embrace change.

The website we created has been a successful hub for information sharing. The

field days received a lot of media attention and were well-attended, and farmers

reported that they would change some management practices due to the

information they received on the day. [Project Manager]

85.

Technology transfer is embedded in the Fund activities for the more

successful projects, not an afterthought. An important achievement from

SFF is that farmers have the opportunity to develop solutions alongside

scientists and industry, and through trial and reflection embed improved

practice in their communities. While at times the resulting change may

appear to be slow, these on-the-ground projects do actually achieve

voluntary change.

Key memorable aspects were: one the great collaboration between a number of

participants and their willingness to offer material for reporting purposes; two

uptake of some of the technologies by farmers, and their enthusiasm. [ProjectManager]

[A memorable aspect is] seeing the project take on life and continue even after

funding ended. [Project Manager]

86.

These comments highlight that achieving technology adoption is a socio-

cultural process. It is not just a process of communication, relying on field

days and newsletters to transfer information to farmers.

Adoption takes place in a social context with farmers discussing their ideas with

other farmers. Much adoption occurs when the idea or practice has become part of

the normative concept of good farm management (Journeaux, 2009).


33/69


33

We need to recognise that information is key to learning and subsequent behaviour

change, but learning will only happen if it is supported by a number of social

processes. These include shared understanding, bounded conflict and a supportive

environment. This implies a need to ensure that the different interest groups have

adequate capacity to participate in such processes (Allen et al., 2002).

87.

This is particularly true for projects with predominantly environmentaloutcomes that are not adequately recognised in the marketplace, and for

outcomes that might be contentious amongst local stakeholders.

88.

Understanding these social elements of technology transfer requires

particular skills, and some of the experienced Project Managers working

with the projects and the MPI Advisers are well-versed in this. At the

moment the MPI Advisers have the overview of the projects and each

oversees a range of projects. However, this national oversight is not being

recorded and fed back to the community in a cohesive manner; and there

is only information available on a project-by-project basis, which is much

harder to locate.

89.

However, technology transfer needs to occur at a number of different

levels, and while there is an imperative for the projects to disseminate

learnings as best they can, there is also an opportunity for MPI to provide

a central point to help the learnings to flow across projects and sectors.

Journeaux (2009) looked at the needs for a more efficient extension

system and identified the need for co-ordination at a higher level in the

system, rather than just relying on communication at a project level.

[The] model could be along the lines of the Sustainable Farming Fund (SFF), or an

entity incorporated into the current SFF, whereby government provides funding for

environmental extension, and industry, councils, private consultants andfarmer/community groups bid into this for funding (Journeaux, 2009)

90.

A good example of a project where technology transfer occurs at a

number of different levels is the Top of the South case study: Setting an

example for sustainable water quality.


34/69


34

Top of the South case study: Setting an example for sustainable water quality

The work on the Sherry was pioneering, it set a level of expectation for

our rivers and streams and demonstrated that improvements can be

made. [Getting] cows out of streams returned quick results [and]

improved effluent management can pay for itself over time. But when

it comes to dealing with run-off from land, it is much slower to make

improvements and effect meaningful results. [Project Manager]

It has become pressing for New Zealand to reduce contaminant losses from farms to

waterways, and begin to reverse the degradation of our waterways. Apart from theenvironmental reasons for doing so, it is necessary from an economic standpoint to

protect the international reputation of our agricultural farming industries in light ofconsumer demand ensuring ongoing market access and sustainability of our primaryexports.

In practice, addressing water quality issues is complex, technical and takes extendedperiods of time. Three SFF projects in the Top of the South (between 2006 - 2012)have together developed an effective model of collaborative action to improve freshwater quality in a catchment.

These projects left their farming communities with working plans for ongoing activitiesthat provide a pathway forward over the next few years. Farmers have beenencouraged as leaders of positive change. The projects also facilitated greaterconnectedness or social capital within the communities through partnerships between

land managers and a wide range of other stakeholders including scientists, central andlocal government, and community and industry facilitators.

These projects in the Sherry, Rai and Aorere catchments each featured significantparticipation by dairy farmers. Dairy farmers provide a window on the multi-level and

complex challenges communities face as they manage water quality issues. Theirstories also demonstrate the value of community-level approaches to catchmentmanagement, and the need to develop partnerships with Councils, neighbouring

communities and industry.

With annual exports in excess of $13 billion, the dairy industry is New Zealands largestexport earner. Dairy productivity has risen markedly over the past decade, withincreases in both cow numbers and milk yields per cow. At the same time, dairyfarming is being increasingly held to account for associated environmental impacts

including water quality.

Investment by the SFF and partners in the projects has demonstrated high levels ofcompliance with good management practice, such as that set out in the water accord(e.g. Effluent survey results 2013 see below para 102). Water quality cannot be

improved overnight, but significant steps have now been taken to arrest its decline.These steps involve the full range of land uses in the region. For example this has beenachieved for a project investment across two projects (the Aorere and the Rai) of$477,000

14 through the SFF and a further $2.2m from industry partners (including

past and anticipated investment in best management practices) less than 2% of

projected 15 year dairy income for these catchments.

91.

There are also other examples of the development and adoption of new

technology in a wide range of sectors:

Development and distribution to all industry of a comprehensive manual related to

the risk management of [Johnes Disease]15

on-farm [was] aimed at those who

already were experiencing issues with the disease and also to aid those free of the

issue to remain so. In addition, in association with the new action group, [we]

14$477,000 is the total investment for Aorere and Rai combined. We have not included

Sherry in this part of the story because we dont have their dairy turnover data or the

value of their industry contributions to the project.15

Johnes Disease is a wasting disease which occurs in cattle, deer, sheep, goats and

wildlife.


35/69


35

developed a nationwide network of specialist veterinarians to work one-on-one with

[deer] farmers. [Project manager]

[The] project aim was to improve persimmon taste. We were able to demonstrate

through Plant and Food Research that taste was related to astringency and soluble

solids sugars. We were then able to demonstrate to growers that astringency

was related to cool growing seasons and soluble solids to harvest timing. [The]industry adopted the findings, and market and grower returns improved. [Project

Manager]

One to-one on-farm visits and transfer of info, workshop that supplied info on

limiting factors that influence water quality and identifying minimum and maximum

nutrient limits required for different farm systems for the soils in that area. [Project

Manager]

Enhanced relationships

92.

SFF Project Managers, MPI Advisers and the Fund Managers all

commented that a particular strength of the Fund was its support of SFF

project teams to build relationships and networks between the farming

community, science and industry.

93.

The following Figure 13 shows the extent to which close working

relationships were developed with stakeholders and networks, and

evidence that relationships were enhanced through SFF projects.

Figure 14: Enhanced relationships from SFF projects

94.

Almost all (91%) Project Managers believed the SFF projects assisted

farmers to develop close working relationships with stakeholders, e.g.,

scientists, Council staff and representatives of industry-good

organisations.

Mutual trust between the farmers and scientists has built up over a period of time

before and during the SFF programme. [Project Manager]

The effort to engage with and involve interested parties, including the fact that

meetings were held in local communities across the country, was a vital part. The

project was the culmination of several years of consultation within the agricultural

aviation industry, with the consensus that the industry needed to be proactive in

managing environmental impacts, so the financial support from our co-funders


36/69


36

supported by the SFF Fund enabled real progress to be made. [Project

Manager]

95. The majority of Project Managers (89%) also reflected that project teams

had developed networks with other farmers and growers, scientists,

industry and Councils.

The SFF supports clusters of projects around common problems, and enables a

wide range of industry sectors to build capability incrementally over time for

instance in the wine industry. [Project Manager]

96.

The case studies also illustrated the depth of relationships built in

different sectors, and at the local, regional and national levels.

These research initiatives have contributed to wider efforts by New Zealand

Winegrowers to create an industry that actively seeks to link research with

practice. This leads to growers wanting to be involved in early field trials and a

wider industry looking to take up subsequent best-practice findings. [Industry

spokesperson]

Behaviour change with positive environmental benefits

97.

Behaviour change is notoriously difficult to achieve, particularly amongst

farmers, as they may have to prioritise spending of at times very

significant amounts to trial new ideas.16

Well, my work showed that the combination of a farm visit and attendance at a

Council field day did make a difference, on average, to farmers' intentions for

improved waterway management. There was evidence that stream health

improved where farmers did change their management. However, the degree of

change in both instances was pretty small. My work showed other things that

might help improve adoption rates, e.g. funding for fencing; also, importantly,

working with farmers during conversions when new fencing is going in... It also

seemed that non-fencing alternatives might be more readily adopted and still have

some benefit. [Project Manager]

98.

An important achievement from SFF grants is the level of active

engagement of farmers in making changes, with support from others. SFF

grants have enabled farmers to develop solutions alongside scientists and

industry.

The change in thinking and the development of the catchment plan by farmers

would not have happened without SFF funds. It takes time to build trust,

particularly when there has been strong conflict historical ly between parties. I mustsay the Regional Council has turned markedly from a strategy of top-down

directives to bottom-up listening, discussing and acknowledging the real concerns

farmers have. [Project manager]

I have personally noticed an attitude shift from denial (it must be some other

source) to anger (this will put us out of business) to quiet acceptance and

getting on with it. The famer-led process empowers people. [Council staff member]

99. The SFF Project Managers considered that SFF projects contribute to:

16Behaviour change is also difficult to evaluate. Mayne and Stern (2013) identify a

number of attributes that provide challenges for evaluating and attributing impacts of

projects, e.g. complex systems, market failures, multi-level impacts, long time-framesfor impacts to emerge and unpredictable outcomes that are highly context-dependent,and these are applicable to SFF investments.


37/69


37

improved environmentally sustainable practice amongst the

farming community (66%)

better environmental standards being incorporated in industry-

level systems (54%) and to a lesser extent to the Council policy

level (33%) or the Government policy level (21%)

projects are able to leverage off environmental credentials to

respond to market opportunities (43%).

100.

The following graph summarises the areas of environmental behaviour

change.

Figure 15: Behaviour change with positive environmental benefits

Greater use of environmentally sustainable practice amongst the farming

community

101.

The SFF has helped to achieve change on a number of levels. SFF projects

have provided farmers with the readiness to change; helped them develop

solutions that worked at a farm level and helped embed these solutionsinto their farm systems.

The project and its predecessor made a significant contribution to poplar and

willow use on-farm [to reduce erosion], and established a large database of users

and interested parties. [Project Manager]

As well as [being] Project Manager role for this project, I have been involved in

various other projects in a support or advisory role, and see real benefits. Within

the Clutha District there have been real changes in farming practice as a result of

SFF projects, especially in terms of environmental performance. The fact that

farmers are both behind and directly involved in the projects is key to this. [Project

Manager]


38/69


38

102. It is clear that the SFF made an important contribution to improving

environmental practice in the Aorere Catchment. However, a recent report

of the effluent survey results from the Aorere River Catchment Dairy Farm

Survey (see box below) shows that while SFF helped farmers make good

progress, this is still not enough to completely solve environmental

challenges (such as stopping faecal spikes occurring). This demonstrateshow challenging it can be to achieve the required changes and impacts on

the environment, and the importance of considering intermediate

outcomes (such as behaviour change) in assessing progress toward

achieving longer-term environmental objectives.

Effluent survey results

Under a system classifying risk to the environment, three out of 30farms were found

to have effluent systems of high risk, 12 were of moderate risk and 15 were of lowrisk. The council said the high-risk farms were in the process of designing and

constructing suitable storage and commissioning low-application irrigation systems,while others in the moderate-risk category were also undertaking improvements.

Of the 697 stock crossing pointslocated, 618 were found to be bridged orculverted. By the end of the survey they included 101 of 126 across permanentwaterways wider than a metre that were required to be covered by Tasman ResourceManagement Plan (TRMP) rules, with the other 25 allowed to operate uncovered as apermitted activity because of their less frequent use. The one non-compliant major

crossing on the back channel of the Aorere, which had since been bridged, wasdiscounted as a source of the May spikes. Farmers had voluntarily bridged orculverted 517 of 571 stock crossingsover small, temporary or artificial waterwaysnot subject to regulation or Clean Stream Accord definitions.

Stock were fully excluded either by fencing or by natural barriers from 89 per centofthe almost 100 kilometresof TRMP-defined waterways. This fell to 49 per cent forwaterways less than a metre wideand to 28 per cent for drains and swales.Further fencing has occurred since the survey was completed.

The bridging and fencing of significant waterways had reduced the likelihood of thembeing major contributors to faecal contamination, although sampling was needed to

see what impact the 25 less frequently used permitted crossings had on the widercatchment. One sample taken 24 hours after a crossinghad been used showedenough faecal contamination of the waterway to create a spike.

Source: Effluent Survey Results 201317

(Note: Bolding added by the evaluators)

Better environmental standards being incorporated in industry-level systems

103.

Project Managers and MPI Advisers were both able to provide examples ofwhere better environmental standards have been incorporated into

industry-level systems.

Workshops associated with [the project] have resulted in many growers actually

checking on flowering times in their orchards. The website material created for the

hazelnut variety identification guide has [also] been used by overseas browsers.

[Project Manager]

104.

Furthermore, change is likely to be sustained as leaders have emerged

from projects who are prepared to continue to champion ongoing change

within the sector and across regions.

17Source: Effluent survey results(2013, August 20) Fairfax NZ News. Retrieved from

http://www.stuff.co.nz/business/farming/agribusiness/9063955/Effluent-survey-results


39/69


39

This group in the Sherry was the first group I worked with. Other farmers from

different catchments listened to their experiences. [Then] Sherry Farmers visited

Golden Bay and told their story. [Project Manager]

105. The case study Protecting the sustainability of New Zealand vineyards

(see below para 118) provides a good example of how projects are able to

leverage off environmental credentials developed with SFF funding torespond to market opportunities, and other examples such as the

following were also provided by Project Managers.

What [SFF] has done to date is generate a lot more participation and interest in

readily sharing [information] from growers of the dropping and picking varieties.

The overall desire from growers for better quality macadamias and better

marketing through a grower co-operative has contributed to this success. [Project

Manager]

Protecting or growing economic value

106.

Case studies and results from the survey of SFF Project Managers providea consistent story of SFF grants protecting and growing economic value,

including some projects contributing to export opportunities. The on-line

survey data identified that overall, the surveyed Project Managers

believed the SFF projects generally achieved their intended results

(94%)18, and that the majority of projects (96%) were good value for

money, as shown in the following graph.

Figure 16: Assessment of the value of the SFF projects

107.

The following graph shows the extent to which Project Managers believed

the SFF projects contributed to growing economic outcomes.

18Percentages unless otherwise noted show those SFF Project Managers who gave acombined rating of moderate, considerable and high degree those shown in green inthe graphs.


40/69


40

Figure 17: Extent to which Project Managers believe SFF contributes togrowing economic outcomes

108.

One quarter of Project Managers surveyed considered their projects

contributed to actual export earnings (25%) while nearly three in five

considered there was potential for their projects to contribute to future

earnings (59%). Furthermore, over half those surveyed maintained that

their SFF projects helped protect, safeguard or maintainproductivity

(52%), and that the projects contribute to increasedprofitabilityat the

project level (49%).

109.

There were also clear signs that primary producers and agribusiness were

able to leverage off environmental credentials to respond to market

threats and opportunities, as shown by the case studies involving the

wine industry and water quality in catchment farming communities. The

viticulture case study also exemplifies how a relatively small, strategic

investment in a few key projects can have a significant impact on growing

industry capability and premium market positioning.

Potential contribution to economic value

110.

While it was beyond the scope of the study and available data sources todirectly measure economic impacts of the SFF, evidence from case

studies, survey feedback and past evaluations suggests there is a credible

prospect of the SFF breaking even that is, returning at least $122.8

million worth of economic benefits to a sector with current annual gross

revenues of over $26 billion (2013) given the aims and success rates of

the projects funded.

111.

Available evidence indicates that SFF funds are being allocated and used

in accordance with the intended purpose and strategic priorities of the

SFF. Case studies, survey feedback from SFF beneficiaries and other


41/69


41

available information19suggest that SFF funds have substantively been

invested in worthwhile and successful projects. Stakeholder feedback

suggests a significant proportion (a quarter to half) of these projects have

potential economic value.

112.

The funding disbursed through the SFF in 201314 represents aninvestment of $2.30 for every $10,000 of national income from primary

industries over the same period. Correspondingly, the SFF will break even

if its contribution to protecting, enhancing or growing the industry is

worth 0.023% of the annual industry contribution to GDP.

113.

In a 2009 SFF study, a cost benefit analysis20identified a net present

value of $10.9 million from a cluster of deer farming projects that sought

to improve the farm environmental and economic performance and the

long-term susta