www.cmhdaresearchnetwork.com.au
© 2015 The Mental Health Coordinating Council (MHCC) and the
Network of Alcohol and Other Drug Agencies (NADA)
Report author:
Aldo Spina - Evaluation Consultant
www.mhcc.org.au www.nada.org.au www.nswmentalhealthcommission.com.au
CONTENTS
EXECUTIVE SUMMARY 1
1. INTRODUCTION 4
2. EVALUATION METHODS 6
3. EVALUATION FINDINGS 7
3.1 Overview of survey respondents 7
3.2 Research in respondents’ workplaces 8
3.3 CMHDARN membership and awareness 9
3.4 Project Reference Group 11
3.5 Research forums 12
3.6. Reflective practice webinars 14
3.7. Community Research Mentoring Project 15
3.8. Research Seeding Grants Program 16
3.9. CMHDARN website 17
3.10 E-Newsletter 18
4. IMPACT ON KNOWLEDGE, SELF-EFFICACY AND RESEARCH CAPACITY 19
4.1 Purpose and relevance 19
4.2 Impact on knowledge and self-efficacy 22
4.3 Impact on research capacity 24
4.4 Building future research capacity 25
4.5 Challenges in building research capacity 26
4.6 Discussion of Research Capacity Building Framework 27
5. CONCLUSIONS 29
5.1 Overall findings 29
5.2 Usefulness of CMHDARN activities 30
5.3 Planning, implementation and communication 31
5.4 Measuring impact over time 31
5.5 Next steps for CMHDARN 32
6. RECOMMENDATIONS 34
APPENDIX 36
| 1
Community Mental Health Drug and Alcohol Research Network Evaluation
EXECUTIVE SUMMARY
The Community Mental Health Drug and Alcohol Research Network (CMHDARN) was established
by the Mental Health Coordinating Council (MHCC) and the Network of Alcohol and other Drugs
Agency (NADA) in 2010. Its aim is to a) broaden involvement of the community mental health
and drug and alcohol sector in practice-based research, and b) promote the value of research
and the use of research evidence in practice, and to improve service delivery to people
affected by drug and alcohol and mental health issues.
This evaluation has found CMHDARN is effective in working towards its goals. It has led to
increased self-efficacy among participants of CMHDARN activities. For example, 66 per cent of
survey respondents report greater confidence to use research findings. CMHDARN has also led to
increased knowledge of drug and alcohol, mental health and co-existing issues. For example, 59
per cent of survey respondents indicated that it has improved their understanding of co-existing
mental health and drug and alcohol issues.
There was very good feedback on the variety of activities that have been implemented by
CMHDARN. All these activities evaluated well. This activity (period December 2010 - November
2014) has included:
o 11 research forums with a total of 493 participants;
o 7 webinars with 257 participants;
o 7 newsletters distributed;
o 240 CMHDARN members; and
o 1,437 NSW visitors to the website with 11,063 website page views.
There remains an ongoing need for CMHDARN. Evaluation participants believe that CMHDARN’s
goals are very relevant to individuals and organisations working in mental health and drug
alcohol.
One area requiring further exploration is to how to better engage senior management in the
project. Apart from a focus on individual skill development, management need to be supported
to identify how they can better prioritise and nurture engagement with research in their
organisations.
An important issue that arose during the evaluation is the funding required to support more
practice-based research in community managed organisations. The Mental Health Commission
of NSW has recognised in its Strategic Plan for Mental Health in NSW the importance of translating
research into practice. This evaluation, as also noted in the evaluation of the Research Seeding
| 2
Community Mental Health Drug and Alcohol Research Network Evaluation
Grants Program and NGO Mental Health and Drug and Alcohol Research Grants Program,
acknowledges that research funding is required if community managed organisation are to
better support practice-based research.
Finally, the extent to which research capacity building is achieved does depend on the level of
participation and engagement from community managed organisations. Among those who
participated in CMHDARN activities, the feedback is overwhelming that it has had a positive
impact on building capacity at an individual and organisation level. Future evaluations of
CMHDARN should use some of the measures that have been used within this report to monitor
changes over time in individual and organisation research capacity.
Recommendations
1. CMHDARN should continue to focus on developing the research capacity of community
managed organisations, promoting the value of research and the use of research evidence
in practice.
2. CMHDARN should continue the variety of activities it is currently undertaking as they are
useful and are having an impact on skills and self-efficacy, this includes webinars, research
forums, and mentoring.
3. CMHDARN should continue with the current format of research forums (presentations which
are followed by guided small group discussions) as they are working well to promote learning
and sharing of experiences.
4. Given the usefulness of the Research Seeding Grants, the identification of the importance of
funding among evaluation participants, recommendations from the Research Seeding
Grants Program and recommendation from the Final Report on the NGO Mental Health &
Drug and Alcohol Research Grants Program, it is recommended that CMHDARN approach
funders to discuss research funding for community managed organisations to support
practice-based research.
5. CMHDARN should continue to maintain and update the CMHDARN website, further
enhancing its technical functions by enabling online registration for research forums and
more prominently featuring the option to subscribe to the email list.
| 3
Community Mental Health Drug and Alcohol Research Network Evaluation
6. CMHDARN should move to a subscription email list (rather than the current member model)
as it makes it easier for interested individuals to subscribe and unsubscribe from receiving
emails from CMHDARN.
7. In addition to the focus on knowledge and skills development, and given most workplaces
do not appear to have dedicated research and evaluation teams or positions, CMHDARN
needs to consider how it can support and engage management in community managed
organisations to better promote, value and support research and an evaluation culture
within their organisations.
8. CMHDARN should consider the implications of this report and amend its draft Evaluation
Framework to reflect the following as priority indicators for its implementation, with a view to
evaluating the program in two years’ time:
i. understanding of drug and alcohol and mental health issues
ii. confidence to use research findings in your work
iii. confidence in undertaking practice-based research
iv. impact on individual research capacity
v. impact on organisational research capacity.
9. A future evaluation of CHMDARN should run focus groups with senior managers of
community managed organisations to understand how they perceive the role of research in
their organisations and how CMHDARN can support this role.
10. CMHDARN has promoted the role of consumers/clients/carers in research and should
continue to emphasise the importance of this role.
| 4
Community Mental Health Drug and Alcohol Research Network Evaluation
1. INTRODUCTION
The Community Mental Health Drug and Alcohol Research Network (CMHDARN) was established
in 2010 by the Mental Health Coordinating Council (MHCC) and the Network of Alcohol and
other Drugs Agency (NADA). The MHCC and NADA are the NSW state-wide peak bodies which
represent community managed organisations in the mental health and alcohol and other drugs
sectors, respectively.
The CMHDARN was established in 2010 to:
broaden involvement of the community mental health and drug and alcohol sector in
practice-based research, and
promote the value of research and the use of research evidence in practice, and to
improve service delivery to people affected by drug and alcohol and mental health
issues
Following three years of funding for the CMHDARN from the NSW Ministry of Health, the NSW
Mental Health Commission provided funding for an initial twelve month period from December
2013 to November 2014, and has continued this partnership into 2015.
CMHDARN has implemented a range of education activities to build research capacity
including research forums, reflective practice webinars, mentoring, research seeding grants,
newsletters and a website. These are further described within the report.
The release of Living Well: A Strategic Plan for Mental Health in NSW 2014-2024 by the Mental
Health Commission of NSW identifies the importance of developing a Research Framework for
Mental Health in NSW (at the time of the evaluation it was subject to consultation). The
framework confirms that the overarching aims of NSW-funded mental health research are to:
improve the lived experience of those with a mental illness and keep people well in the
community;
accelerate the translation of research into practice; and
support and attract a robust research capacity and infrastructure for NSW.
| 5
Community Mental Health Drug and Alcohol Research Network Evaluation
CMHDARN can make an important contribution to
achieving these aims, particularly given its focus on
supporting practice-based research and the use of
research evidence. In fact, CMHDARN has already
been recognised for its contribution to research and
work across the sectors. In 2014, the CMHDARN was a
joint winner of the inaugural Tom Trauer Research and
Evaluation Award in recognition of its contribution to
research and evaluation. In 2013, CMHDARN received
a NSW Mental Health Matters commendation award in
2013 in the category of ‘Cross Sector Collaboration’.
Informing the work of CMHDARN is a framework for research
capacity building that outlines six principles:*
building skills and confidence;
ensuring the research is 'close to practice';
developing linkages and partnerships;
developing appropriate dissemination;
building elements of sustainability and continuity; and
investments in infrastructure.
Figure 1: Research Capacity Building Framework adapted to CMHDARN from Cooke, J
The evaluation has attempted to explore progress across all six principles and identify where
further activity is required.
Since 2010, a steering committee, consisting of representatives from MHCC and NADA, has
overseen the implementation of the project with a focus on ensuring it meets its deliverables and
reporting requirements. A Project Reference Group, consisting of researchers, consumers and
practitioners from the community sector and academic researchers, was also established to
advise and strategically guide the work of the research network.
* Cooke J. A framework to evaluate research capacity building in health care. BMC Family Practice 2005,
6:44
Tina Smith and Bradley Foxlewin (a member of
the Project Reference Group, accepting the
Tom Trauer award on behalf of CMHDARN.
| 6
Community Mental Health Drug and Alcohol Research Network Evaluation
2. EVALUATION METHODS
The evaluation occurred throughout November-December 2014. An independent evaluation
consultant was contracted to undertake the evaluation. There are four main evaluation
methods:
Online survey: An invitation to complete an online survey was emailed to CMHDARN
members (240 members) and attendees of network events (201 attendees). A link was
posted on the CMHDARN website and promoted through both the MHCC and NADA
websites and communications. A reminder email was sent to members approximately
one week after the initial invitation email was sent. (See Appendix A for survey)
Analysis of project activity: An analysis of all project activity was undertaken. This
included reviewing project activity reports and prior evaluation reports. An analysis was
also undertaken of website statistics.
Focus group: Following the ‘Navigating Research Ethics’ Forum on 2 December 2014, a
focus group was held with interested participants from the forum. Nine people
participated in the focus group. The focus group ran for approximately 50 minutes. The
session was facilitated by the consultant.
Key informant interviews: All members of the Project Reference Group were invited to
contact the evaluation consultant if they wished to participate in a short interview. They
were also reminded that they could input into the evaluation by completing the online
survey. Ten current or former members of the reference group completed the survey. This
was supplemented by three telephone interviews. The interviews ran for between 15-30
minutes.
Limitations of the evaluation
The online survey was forwarded to CMHDARN members or attendees of CMHDARN events. This
means those who participated in the evaluation generally already have some degree of
familiarity and involvement with CMHDARN. There may have been a small number of other
people who accessed the survey through the websites where the link was placed. The
evaluation is unable to assess how MHCC or NADA members who have not participated in any
events or are not CMHDARN members value its purpose and view its work. It is important to bear
in mind that there were 65 survey respondents, based on the number of email invitations that
were sent out; this represents only a 15 per cent survey uptake rate.
| 7
Community Mental Health Drug and Alcohol Research Network Evaluation
3. EVALUATION FINDINGS
3.1 Overview of survey respondents
Sixty-five individuals responded to the survey. Among the 65 survey respondents, the
overwhelming majority (83 per cent) work for a non-government organisation. Forty-two per cent
of all survey respondents work for an MHCC or NADA member organisation. Of survey
respondents from another non-government organisation, four are also CMHDARN members.
Three respondents are consumers, clients or carers; all three respondents also indicated that they
work for a non-government organisation. Twenty per cent of respondents were academic
researchers. Their answers have been excluded from parts of the analysis given some questions
are not relevant to their work.
Approximately a third of respondents are working for small size organisations (25 employees or
less), a third for medium size organisations (26-100 employees), and a third are working for large
size organisations (over 100 employees).
Table 1. What type of organisation do you work for? (Tick all that apply) (n=65)
Answer Options % No.
MHCC member organisation 13.8% 9
NADA member organisation 27.7% 18
Other non-government organisation 41.5% 27
I work as an academic researcher 20.0% 13
I'm a consumer or client or carer 4.6% 3
Other 4.6% 3
Table 2. What is the approximate size of your organisation? (n=65)
Answer Options % No.
10 or under employees 16.9% 11
11-25 employees 13.8% 9
26-100 employees 33.8% 22
Over 100 employees 33.8% 22
Not applicable 1.5% 1
| 8
Community Mental Health Drug and Alcohol Research Network Evaluation
3.2 Research in respondents’ workplaces
Survey respondents were asked a set of questions about research within their workplace to help
us better understand their research capacity. Respondents who are researchers are excluded
from the analysis below.
Using research
Eighty per cent of respondents indicated that they frequently or somewhat frequently use
research findings to inform their work. Only 12 per cent indicated that they somewhat
infrequently or infrequently use research findings to inform their work. Furthermore, when asked
how frequently they participate in the implementation of research, 67 per cent indicated
frequently or somewhat frequently. This time 17 per cent indicated that they somewhat
infrequently or infrequently participate in the implementation of research within their workplace.
It should be noted that those who are most likely to engage with CMHDARN are probably those
who are already interested in research or engaged with research in their workplace.
Table 3. How frequently or infrequently do you use research findings to inform your work?
(n=49)
Answer Options % No.
Frequently 46.9% 23
Somewhat frequently 32.7% 16
Neither frequently nor infrequently 8.2% 4
Somewhat infrequently 10.2% 5
Infrequently 2.0% 1
Not applicable 0.0% 0
Table 4. How frequently or infrequently do you participate in the implementation of research
within your workplace? (n=48)
Answer Options % No.
Frequently 22.9% 11
Somewhat frequently 43.8% 21
Neither frequently nor infrequently 12.5% 6
Somewhat infrequently 8.3% 4
Infrequently 8.3% 4
Not applicable 4.2% 2
Accessing research
The two most popular ways for respondents to access research is to either access the relevant
research themselves or to attend an event where research findings are presented, such as a
forum or conference. Among the 20 per cent who chose ‘other’, they generally commented
that they are studying or conducting their own research.
| 9
Community Mental Health Drug and Alcohol Research Network Evaluation
Table 5. How do you mostly access research related to your work? (Tick all that apply) (n=49)
Answer Options % No.
My organisation provides me with information about
relevant research 42.9% 21
I access relevant research myself (such as via journals,
websites, e-newsletters) 83.7% 41
I attend forums, seminars, workshops, and conferences 83.7% 41
Other (please specify): 20.4% 10
Internal research and evaluation support
Within their organisations, almost half of all respondents indicated that there is no dedicated
research or evaluation team or even a dedicated position. While 18 per cent indicated that they
have a research or evaluation team, another 18 per cent indicated that they have one position
that undertakes research. Among those who selected ‘other’, a few commented that “various
staff do this on an ad hoc basis” and another commented that “we have a lot of people
interested in it”.
Table 6. How would you best characterise the research and evaluation support in your
organisation? (n=49)
Answer Options % No.
We have a research or evaluation team 18.4% 9
We have one staff person who undertakes research or
evaluation 18.4% 9
We have no specific position which undertakes research
and evaluation work 44.9% 22
Not applicable 6.1% 3
Other (please specify) 12.2% 6
3.3 CMHDARN membership and awareness
Eighty-six per cent of survey respondents had heard of CMHDARN prior to the survey. This is not
surprising given that respondents were recruited to the survey because they are members of the
network, had attended an event, visited the website or related MHCC/ NADA websites.
Respondents who had heard of CMHDARN were asked how familiar they are with the activities
of the network. Twenty-eight per cent indicated that they are familiar, while 54 per cent
indicated that they are somewhat familiar. However, this left approximately 19 per cent of survey
respondents who are more neutral in their responses, somewhat unfamiliar or unfamiliar with the
work of the network.
| 10
Community Mental Health Drug and Alcohol Research Network Evaluation
Table 7. You indicated that you have heard of the CMHDARN. How familiar or unfamiliar are
you with the activities of CMHDARN? (n=54)
Answer Options % No.
Familiar 27.8% 15
Somewhat familiar 53.7% 29
Neither familiar nor unfamiliar 9.3% 5
Somewhat unfamiliar 7.4% 4
Unfamiliar 1.9% 1
CMHDARN Membership
Individuals can become members of CMHDARN by completing an online form or via a hard
copy membership form. There is no cost to become a member. By becoming a member you
receive notifications of CMHDARN activities and events. At the time of the evaluation, there
were 240 members. Approximately a third of survey respondents are members of CMHDARN,
although it may be higher as approximately another third were unsure if they are members.
Forty per cent of respondents indicated that they are not members.
Figure 2: Are you a CMHDARN member?
Two-thirds of CMHDARN members indicated that it was useful to be a member. While another 22
per cent indicated it was somewhat useful. Two respondents were more neutral in their
responses, one of these respondents indicated that they are new and have “not yet had a
chance to decide its usefulness”.
Table 8. You indicated that you are a member of the CMHDARN. Have you found it useful or
not useful to be a member of the network? (n=18)
Answer Options % No.
Useful 66.6% 12
Somewhat useful 22.2% 4
Neither useful nor not useful 11.1% 2
Somewhat not useful 0.0% 0
Not useful 0.0% 0
30.20%
39.70%
30.20%
CMHDARN Member?
Yes, I am a member No, I am not a member Unsure
| 11
Community Mental Health Drug and Alcohol Research Network Evaluation
3.4 Project Reference Group
Ten survey respondents are or have been members of the Project Reference Group. Five of these
respondents have found the Project Reference Group useful and another four have found it
somewhat useful.
Key informants commented that the Project Reference Group was working well and was making
a useful contribution to the work of the network. Two survey respondents commented on how it
has been useful.
“The events and activities that have grown out of the PRG have been excellent
and a great example of the collaborative nature of CMHDARN.”
“Collective planning and prioritising”
A survey respondent wondered whether these types of committees are generally useful, though
they did believe the network itself is doing “useful stuff”.
“Always wonder about how much committees achieve, but the network is doing
useful stuff so maybe the committee is working.”
Table 9. You indicated that you are (or were) on the Project Reference Group. Do you feel
that the Project Reference Group has made a useful or not useful contribution to the network?
(n10)
Answer Options % No.
Useful 50.0% 5
Somewhat useful 40.0% 4
Neither useful nor not useful 10.0% 1
Somewhat not useful 0.0% 0
Not useful 0.0% 0
| 12
Community Mental Health Drug and Alcohol Research Network Evaluation
3.5 Research forums
Research forums aim to facilitate research capacity building and build understanding about
research in the community mental health and drug and alcohol sectors. Eleven forums have
been held, with a total of approximately 493 participants. Following each forum, the agenda,
presentations, the evaluation and other relevant resources are made available on the website.
Table 10. Research Forums
Year Focus No.
attendees
2014 Navigating research ethics 16
Understanding best practice research when working with Aboriginal
and Torres Strait Islander organisations and people
46
Strategies for building research capacity in your organisation (Held in
Ballina)
27
Fundamentals of research 43
2013 Building your evidence-base – Lessons from the field 68+
Realising research in rural areas (Held in Orange) 55+
Sharing our learning – A research forum for the recipients of Research
Seeding Grants
18+
2012 Ethics in Research – they’re more important than you think! 39+
Nothing About us, Without Us - We are the Evidence: Consumer
representation and participant in research
47+
Seeding Grants Program Launch: From ideas to action – developing
your research proposal
59+
2011 Research and evaluation in community managed mental health and
drug and alcohol sectors
75+
+ Number of registrations
Forty-two per cent of survey respondents have participated in a research forum. On average
respondents attended two forums. When asked if these research forums have been useful, 76 per
cent of respondents indicated that the forums have been useful. While another 20 per cent
indicated that they were somewhat useful (see figure 3).
“forums have been excellent”
| 13
Community Mental Health Drug and Alcohol Research Network Evaluation
Figure 3: Usefulness of CMHDARN membership
Forum ratings
At the end of each forum participants are asked to complete an evaluation form. Participants
could rate the forum on a five point scale. Participants rated the forums very highly with over 90
per cent (using a sample of four forums – see Table 13) rating the forum that they attended as
excellent or very good.
Table 11. Research forum evaluation feedback
No.
evaluations
Excellent or
Very good
Understanding best practice research when working with
Aboriginal and Torres Strait Islander organisations and
people
46 96%
Strategies for building research capacity in your
organisation
27 100%
Fundamentals of research 43 95%
Building your evidence base – Lessons from the field 45 90%
Focus group participants stated that they found the forum that they had just attended
(Navigating Research Ethics) informative and helpful.
“Presentations were very good”
“Getting a lot of ideas of things I can take back to my place”
“New information”
76
20
4
0
Useful Somewhat useful Neither useful or not useful Not useful
| 14
Community Mental Health Drug and Alcohol Research Network Evaluation
There was also agreement among focus group participants that the format of the forum,
presentations followed by small group work exploring a case study, worked really well to support
learning.
“You never learn anything if everyone is just sitting their listening to people”
They also appreciated that the forums were free to participants as they thought this most likely
aided individuals being able to attend the forum.
3.6. Reflective practice webinars
Reflective practice webinars aim to promote knowledge
and understanding about recent research. The webinars
commenced in 2012. One of the intentions of the webinars is to
make the work of the network more accessible to interested
parties in rural and regional areas. Each webinar runs for 60
minutes. The forums are facilitated by Associate Professor
Katherine Mills from the National Drug and Alcohol Research
Centre. Seven webinars have been held by the CMHDARN
with approximately 257 participants.
Table 12. Reflective Practice Forums and attendance
Year Title No.
participants
2014 Integrated psychological treatment addressing co-existing alcohol
misuse and depression
34
Improving organisational capacity and demonstrating efficacy 20
Analysing journal articles - key questions to ask 11
2013 Designing evidence-based treatments for young people with complex
needs
63
Incorporating research findings into service delivery – an introduction to
implementation science
42
2012 Substance use among consumers of a community managed mental
health service
43
Stigma and discrimination towards people with drug and alcohol and
mental health issues
44
Number of participants is based on registrations only – in some workplaces more people have watched
the webinars than registered.
| 15
Community Mental Health Drug and Alcohol Research Network Evaluation
Sixteen survey respondents have participated (or later viewed or listened to) in a webinar. They
participated in an average of two webinars. Eighty per cent of respondents who attended a
webinar found them useful, 13 per cent found them somewhat useful. Only one respondent was
more neutral in their response.
Table 13. Overall, have the webinars been useful or not useful? (n=15)
Answer Options % No.
Useful 80.0% 12
Somewhat useful 13.3% 2
Neither useful nor not useful 6.7% 1
Somewhat not useful 0.0% 0
Not useful 0.0% 0
3.7. Community Research Mentoring Project
The Community Research Mentoring Project aims to support individuals and organisations who
have an interest in research with an academic mentor to support their knowledge and skills
development. Established initially in 2014, at the time of the evaluation, this program was still
underway.
The project is being implemented in collaboration with the Centre for Research in Excellence in
Mental Health and Substance Use (CREMS) at the National Drug and Alcohol Research Centre.
Ten mentees working in the community mental health and drug and alcohol sector have been
each matched up with one of the eight participating research mentors. Five survey respondents
are participating in the Mentoring Project as mentees. Another five participants are unsure if they
are participating. Four of the five who are participating in the program indicated it was useful.
One respondent indicated it was neither useful nor not useful (they provided no comment on
their rating).
Table 14. You indicated that you are participating in the Mentoring Project. Overall, is
participating in the program useful or not useful? (n=5)
Answer Options % No.
Useful 80.0% 4
Somewhat useful 0.0% 0
Neither useful nor not useful 20.0% 1
Somewhat not useful 0.0% 0
Not useful 0.0% 0
| 16
Community Mental Health Drug and Alcohol Research Network Evaluation
3.8. Research Seeding Grants Program
Research Seeding Grants Program provided 16 successful applicants with funds to conduct the
preliminary work needed to develop a research proposal, including partnering with an
academic researcher. In total, over $136,000 was disbursed.
This grant program was separately evaluated in 2014. The evaluation concluded that it had
achieved its key goal of increasing research capacity, building the confidence of participants
and improving relationships with academic research partners.*
Seven survey respondents indicated that their organisation had received a Researching Seeding
Grant. Eleven respondents were unsure if their organisation had received a grant. Five of the
seven respondents who said that their organisation had received a grant indicated that it was
useful, while the remaining two survey respondents indicated it was somewhat useful.
Table 15. You indicated that your organisation received a Research Seeding Grants. Overall,
was this grant useful or not useful? (n=7)
Answer Options % No.
Useful 71.4% 5
Somewhat useful 28.6% 2
Neither useful nor not useful 0.0% 0
Somewhat not useful 0.0% 0
Not useful 0.0% 0
Two respondents commented favourably on the grants.
“very very useful”
“We received a grant to run a pilot project. This enabled us to develop inclusive research
methodologies that facilitated the involvement of Consumers as co-researchers.”
A key informant commented that that “things wouldn’t have happened otherwise” if seeding
grants were not available. However, another two respondents outlined some of the challenges
that emerged from the implementation of the grants.
“it was brilliant, but again I was not supported by my organisation when trying to do the
research”
“I fell down a bit on the implementation, but we are still using the findings.”
* CMHDARN Research Seeding Grants Program: An Evaluation. Mental Health Coordinating Council and
Network of Alcohol and other Drugs Agencies, 2014.
| 17
Community Mental Health Drug and Alcohol Research Network Evaluation
3.9. CMHDARN website
The CMHDARN website was developed in 2012 and went live in late September 2012. It provides
an overview of the project, its activities and makes relevant resources available. For example,
the presentations and associated resources from each of the research forums and webinars are
on the site.
An analysis of the website statistics since September 2010
found:
1,437 NSW visitors
3,263 NSW sessions
11,063 page views.
Seventy-one per cent of survey respondents have visited the CMHDARN website. The majority of
survey respondents found the website informative, while 38 per cent found it somewhat
informative. Seven per cent of respondents were more neutral in their response.
Table 16. You indicated that you have visited the website. Overall, did you find the CMHDARN
website informative or uninformative? (n=42)
Answer Options % No.
Informative 54.8% 23
Somewhat informative 38.1% 16
Neither informative nor uninformative 7.1% 3
Somewhat uninformative 0.0% 0
Uninformative 0.0% 0
Providing information on the activities of the network to interested individuals working in the
mental health and drug and alcohol community sectors is the primary purpose of the site. Visitor
statistics suggests it achieves this goal. Overall NSW visitor engagements are:
3.39 average page views per session;
3.36 minutes average session duration; and
47 per cent bounce rate – these are visitors who arrive on the site but leave right away
without clicking through to any other page.
The bounce rate is at the higher end, though still not outside of normal expectations for websites.
Almost a third of all page views (3,943 page views) are of the Research Network Activities pages
which feature information on the research forums, webinars, etc. Returning NSW visitors have
much better engagement with the site. Staying almost twice as long (4.34 vs. 2.21 minutes) and
have slightly better average number of page views per session (3.66 vs. 3.04).
| 18
Community Mental Health Drug and Alcohol Research Network Evaluation
Currently, few visitors make it through to the Newsletter page or Membership page. The
Newsletter page has achieved 261 page views from NSW visitors since the website commenced
(less than 2 per cent of total page views). Similarly, the Become a CMHDARN Member page has
received 215 page views from NSW visitors (less than 2 per cent of all page views). These are
important communication and relationship management tools for the project and so should
feature prominently on the home page.
3.10 E-Newsletter
The CMHDARN e-newsletter, CMHDARN-Yarn, is published three or four times a year to keep
recipients informed about the work of the network and to provide them with information on
research activities and resources. It is also posted to the website. Each edition provides updates
of the networks activities and other events that may be of interest to recipients. It also provides a
brief summary of sector news and useful resources.
Seven newsletter have been produced:
Issue 7, July 2014 (pictured)
Issue 6, February 2014
Issue 5, December 2013
Issue 4, September 2013
Issue 3, July 2013
Issue 2, April 2013
Issue 1, August 2012.
Forty-one per cent of survey respondents have read the newsletter. Of those who had read the
newsletter, half said it was informative, while the remaining half said it was somewhat informative.
One respondent commented on the relevance of the newsletter.
“I understand it's difficult to source material that's exactly relevant to the diversity of
organisations that form part of this network”
Table 17. You indicated that you have read the e-newsletter. Overall, do you find the e-
newsletter informative or uninformative? (n=24)
Answer Options % No.
Informative 50.0% 12
Somewhat informative 50.0% 12
Neither informative nor uninformative 0.0% 0
Somewhat uninformative 0.0% 0
Uninformative 0.0% 0
| 19
Community Mental Health Drug and Alcohol Research Network Evaluation
4. IMPACT ON KNOWLEDGE, SELF-EFFICACY AND RESEARCH CAPACITY
4.1 Effectiveness and relevance
The aim of CMHDARN is to ‘broaden the involvement of drug and alcohol mental health
community organisations in practice-based research, to promote the value of research and to
increase the application of research evidence to every day practice’.
Effectiveness
Seventy-two per cent of survey respondents thought the CMHDARN was effective or somewhat
effective at achieving its aim. Nineteen per cent were more neutral and a further nine per cent
thought it was somewhat ineffective. One key informant commented that they thought the
CMHDARN had done “superbly well”.
Those respondents who were more neutral or critical are generally not CMHDARN members, are
less familiar with the work of CMHDARN and have had less involvement with the network than
other respondents. As one respondent notes below that when they selected the ‘neither
effective nor not effective’ option, they were indicating that they do not know.
“Sorry I am not familiar with the networks aims or goals so couldn't really comment.
Thanks (& I should inform myself!)”
“! would rather say -'I don’t know'
Table 18. How effective or ineffective do you think the CMHDARN has been at achieving its
aim? (n=58)
Answer Options % No.
Effective 22.4% 13
Somewhat effective 50.0% 29
Neither effective nor not effective 19.0% 11
Somewhat ineffective 8.6% 5
Ineffective 0.0% 0
| 20
Community Mental Health Drug and Alcohol Research Network Evaluation
Relevance
Overall, the goals of the CMHDARN are seen as relevant to respondent’s own work and the work
of their organisation. Sixty per cent agree the goals are relevant to their own work and 59 per
cent agree that the goals are relevant to their organisation’s work.
One of the reasons key informants and focus group participants praised CMHDARN is that they
believe its goal of encouraging community managed organisations* to participate in practice-
based research will produce better results for consumers/clients†.
“what we most want: priorities for service users, not priorities for academic researchers”
“in your workplace you’re closer to the changes and things you want to research and the
changes you want to make”
More generally, survey respondents praised the work of CMHDARN.
“Effective health research must seek to answer the questions of consumers and service
providers I believe CMHDARN has made this its focus which fits well with my role in my
current position.”
* ‘Community managed organisations’ is the terminology used within the report to describe not-for-profit
community organisations and non-government organisations. † ‘Consumers/clients’ is used to reflect the different language used in the mental health and drug and
alcohol sectors.
0
20
40
60
80
100
To your ownwork
To yourorganisation
60 58.6
26.7 25.9
13.3 13.8
1.7
Relevance of goals
Relevant Somewhat relevant Neither relevant/ irrelevant N/A
| 21
Community Mental Health Drug and Alcohol Research Network Evaluation
“The emphasis on evidence based practice is high in the sector right now (as it should
be). Supporting CMOs to further their skills in and opportunities for the development of a
sound body of evidence on which to base their service delivery is vital. Capitalising on
the evidence and practice knowledge from the field is a great way to add to the body
of evidence on mental health and AOD.”
Some respondents commented on what they have found most useful from participation in
CMHDARN.
“CMHDARN is an excellent initiative. The opportunity for CMOs and research professionals
to come together, share information and develop new learning is fantastic.”
“CMHDARN research forums have been very informative and useful for contacts with
other NFP's/ NGO's:
They also commented on how valuable it has been for them and their organisation.
“My organisation's (& my own personal) involvement with CMHDARN has been extremely
profitable. I think it is a great network and a wonderful resource.”
“I have found CMHDARN an extremely useful forum and network - hope funding
continues to be delivered for further development of its operations”
While a few respondents commented on their inability to participate more fully in CMHDARN
activities.
“I only go through the emails I receive. I never got an opportunity to attend any
sessions/symposium of CMHDARN”
“"however I am not able to attend as many activities as I would like to .We are short
staffed in my area of work. Even though I enjoy my work despite the shortage"
| 22
Community Mental Health Drug and Alcohol Research Network Evaluation
4.2 Impact on knowledge and self-efficacy
Survey respondents (excluding researchers) have indicated that participating in CMHDARN
activities has had a positive impact on their understanding of the issues:
61% strongly agree or agree that it has improved their understanding of drug and alcohol
issues;
54% strongly agree or agree that it has improved their understanding of mental health issues;
and
59% strongly agree or agree that it has improved their understanding of co-existing mental
health and drug and alcohol issues.
Those who indicated that they are familiar with the activities of CMHDARN, rate its impact more
positively on all the three indicators above. For example, among those respondents who are
familiar with CMHDARN activities, 85 per cent strongly agree or agree that it has improved their
understanding of co-existing issues (vs. 59 per cent of all respondents).
Table 19. Has participating in CMHDARN activities improved your understanding of drug and
alcohol and mental health issues? (n=41)
Answer Options Strongly
agree Agree
Neither
agree nor
disagree
Disagree Strongly
disagree
Drug and alcohol issues 24% 37% 37% 2% 0%
Mental health issues 18% 36% 44% 3% 0%
Co-existing MH/DA 15% 44% 41% 0% 0%
Those who tend to neither agree nor disagree are not likely to be members and have had more
limited participation in CMHDARN network.
“I cannot comment properly as I have no recollection of any engagement”
“I haven’t really participated in the activities”
Interestingly, two participants commented that while they didn’t think their knowledge has
shifted it has still been valuable experience.
“I don't think my knowledge has changed, but it has been valuable to work with
community based clinicians - I would be interested in developing these relationships”
“Working in a non-clinical Mental Health support service means that my understanding of
MH issues is relatively sound; likewise re co-morbidity but networking with specialist AOD
services has been useful”
| 23
Community Mental Health Drug and Alcohol Research Network Evaluation
Confidence
Two-thirds of respondents indicated that participating in CMHDARN activities has made them
more confident or somewhat more confident to use research findings in their work. The remaining
third of respondents indicated that they were neither more confident nor less confident.
Excluded from the analysis are those who indicated that this question was not applicable (21 per
cent of survey respondents) and survey respondents who were researchers. Again, of those who
are more familiar with the CMHDARN activities, all have indicated that it has made them more
confident or somewhat more confident.
Table 20. Overall, has participating in CMHDARN activities increased or decreased your
confidence to use research findings in your work? (n=35)
Answer Options % No.
More confident 25.7% 9
Somewhat more confident 40.0% 14
Neither more confident nor less confident 34.3% 12
Somewhat less confident 0.0% 0
Less confident 0.0% 0
Again, approximately two-thirds of respondents rated themselves as more confident or
somewhat more confident to undertake practice-based research as a result of participating in
CMHDARN activity. The remaining 37 per cent of respondents were neither more confident nor
less confident. (Excluded from the analysis are those who indicated that this question was not
applicable, 21 per cent of survey respondents, and survey respondents who were researchers).
Again, of those who are familiar with CMHDARN activities, all have indicated that it has made
them more confident or somewhat more confident.
Table 21. Overall, do you feel more or less confident in undertaking practice-based research
as a result of participating in a CMHDARN activity? (n=35)
Answer Options % No.
More confident 20.0% 7
Somewhat more confident 42.9% 15
Neither more confident nor less confident 37.1% 13
Somewhat less confident 0.0% 0
Less confident 0.0% 0
| 24
Community Mental Health Drug and Alcohol Research Network Evaluation
4.3 Impact on research capacity
Respondents were asked to rate the impact, from high impact to no impact on a five point
scale, of the CMHDARN on their own research capacity as well as on their organisation’s
capacity.
Impact on individual
When assessing the impact on their own research capacity, respondents gave the highest
impact rating to ‘use of research evidence’ (3.24 rating average) and the lowest impact rating
was ‘undertaking practice-based research’ (2.86 rating average).
Table 22. How would you rate the impact, if at all, of the CMHDARN on your own research
capacity? (n=42)
Answer Options
1
No
impact
2 3 4 5
High
impact
n/a Rating
Average
Use of research evidence 5 3 6 10 5 12 3.24
Collaboration with
researchers 4 6 7 5 6 11 3.11
Involvement of consumers
and carers in research 5 4 9 6 4 10 3.00
Undertaking practice-
based research 4 8 9 2 5 11 2.86
Impact on organisation
When assessing the impact on their organisation’s research capacity, the highest impact rating
was for ‘collaboration with research’ (3.17 rating average) and the lowest impact rating was for
‘undertaking practice-based research’ (2.97 rating average).
Table 23. How would you rate the impact, if at all, of the CMHDARN on your organisation's
research capacity? (n=42)
Answer Options
1
No
impact
2 3 4 5
High
impact
n/a Rating
Average
Collaboration with
researchers 5 5 6 6 7 11 3.17
Use of research evidence 5 7 4 9 6 10 3.13
Involvement of consumers
and carers in research 5 4 9 9 4 8 3.10
Undertaking practice-
based research 5 7 8 6 5 8 2.97
| 25
Community Mental Health Drug and Alcohol Research Network Evaluation
Those respondents who chose ‘not applicable’ when rating its impact on their own research
capacity or their organisations’ are often researchers, consumers/clients or individuals who have
had less involvement with CMHDARN activities. Similarly, those who select ‘no impact’ have
often had more minimal involvement in CMHDARN activities (e.g. no participation in the
Mentoring Project, Research Seeding Grants Program and webinars). By contrast, those that
gave a ‘high impact’ rating have had involvement in a broad variety of activities such as the
research forums and seeding grants. For example, participants of the Research Seeding Grants
Program rate the CMHDARN as having had a much higher impact on their own research
capacity and their organisations, across all four indicators.
4.4 Building future research capacity
Survey respondents (excluding researchers) were provided with 12 options for future activities
and were asked to select those that are most valuable to help ensure sustainable research
capacity in their organisation. These options were based on examples used in the Research
Capacity Building Framework mentioned at the start of the report and addressed in the next
section. The six most popular choices (in order of popularity) were:
‘partnerships with research’ and ‘best practice resources’
‘applying skills in practice’
‘support and mentoring’
‘skills training’ and ‘funding opportunities’.
Table 24. To help us plan future activity, from the list below, what activities do you think are the
most valuable to help ensure sustainable research capacity in your organisation? (Tick those
you most value) (n=42)
Answer Options % No.
Partnerships with researchers 57.1% 24
Best practice resources 57.1% 24
Applying skills in practice 54.8% 23
Support and mentoring 50.0% 21
Skills training 47.6% 20
Funding opportunities 47.6% 20
Dissemination of research findings 42.9% 18
Partnerships between MH and DA organisations 40.5% 17
Time allowance to undertake research 38.1% 16
Development of organisation research strategy or policy 28.6% 12
Management support for research 23.8% 10
Research included in job descriptions 11.9% 5
Other (please specify) 4.8% 2
| 26
Community Mental Health Drug and Alcohol Research Network Evaluation
Survey respondents and focus group participants commented on the need to identify funding
opportunities for research.
“how to get funding”
“Funding is the constant issue. It might be very useful to have regular info sessions on
where to go for funding, discussing new ways of raising funds such as "crowd funding"
and the pros & cons of such initiatives. Activities that foster a culture of collaboration
rather than competition would be good.”
“funding initiatives for projects”
4.5 Challenges in building research capacity
One of the key challenges identified by participants in the evaluation was that for CMHDARN to
achieve its goals it needs to address the priority accorded to research within community
managed organisations and the resources, both financial and human resources, allocated to
support research.
“CMHDARN can only be so effective without further resources (human and financial) in
the sector to support effective implementation and participation in research.”
Key informants and focus group participants commented that in their experience many
community managed organisations do not see research as core business.
“some see it as core business, research and evaluation, others it’s not part of their core
business”
“(my organisation) doesn’t have a culture of research and evaluation”
One participant noted that while they received a research seeding grant they did not feel
particularly supported by their organisation.
“it was brilliant (research seeding grant), but again I was not supported by my
organisation when trying to do the research”
| 27
Community Mental Health Drug and Alcohol Research Network Evaluation
More generally, focus group participants reported that the level of support for research within
their organisations varied. For some their organisations clearly support research and evaluation,
particularly if they work in a large organisation.
“(we have a) research and evaluation group in our organisation”
While some stated that their organisation “generally doesn’t” value research or have a “culture
of research and evaluation”, others noted that while it might be valued they “don’t budget for
research” which inhibits it occurring.
“I think CMHDARN do great work, I would like to see more support for funding research
and also trying to promote the benefits of it. I have been on my own in my organisation
trying to do research. When there is no funding no one cares”
“don’t budget for research”
Focus group participants discussed how their organisations generally sees themselves as working
at the “coalface” so the focus is on, as one participant described it, “doing the doing”.
In terms of where to next, key informants and focus group participants generally thought that the
network needed to keep building on the work it has commenced.
“continue with seeding grants – how can we build upon them?
“(on a) solid trajectory”
4.6 Discussion of Research Capacity Building Framework
As noted in the introduction, the CMHDARN has adopted a draft framework for research
capacity building.* To what extent is the project engaged in activities across the six principles
outlined in the framework?
Building skills and confidence: CMHDARN has had a significant focus on developing the skills and
confidence of the workforce to engage with research and conduct practice-based research.
Research forums, mentoring, research seeding grants and webinars are all designed to do this,
and the evaluation has found that they have been effective at achieving this.
* Cooke J. A framework to evaluate research capacity building in health care. BMC Family Practice 2005,
6:44
| 28
Community Mental Health Drug and Alcohol Research Network Evaluation
Ensuring the research is 'close to practice': The CMHDARN focus has been on practice research,
which inherently ensures research is close to practice. CMHDARN has encouraged community
managed organisations to conduct research relevant to their own practice, as well as facilitate
more collaboration with researchers who are interested in practice research. There has also
been an emphasis on the importance of consumer, client and carer involvement in research.
Developing linkages and partnerships: This is a strength of the CMHDARN. Firstly, the initiative is a
collaborative effort between MHCC and NADA. Secondly, there has been a significant focus on
nurturing collaboration and partnerships with researchers. Thirdly, the project has facilitated and
promoted the importance of consumers/clients/carers as partners in research.
Developing appropriate dissemination: Research findings dissemination has been supported by
the CMHDARN, particularly through the research forums and webinars. There were nine
presentations to CMHDARN Research Forum (November 2013) by research seeding grant
recipients. They have also disseminated their work through the publication of journal articles and
conference presentations. The Seeding Grants Program Evaluation identified that there were five
papers (published and submitted) and 12 external conference presentations up until January
2014.*
Building elements of sustainability and continuity: There has been a good attempt to ensure
acquired skills are sustained and extended through the Research Seeding Grant and Mentoring
Project. However, the extent to which others who participate in forums or webinars have applied
their skills to practice (or initiate collaborations or applied for research funding, etc.) in their
workplace is unknown. ‘Applying skills in practice’ is one of the top three priorities for further
CMHDARN activities identified by survey respondents.
Appropriate infrastructures: There have been some contributions to supporting infrastructure,
such as the Mentoring Project, however, CMHDARN had had limited influence on organisational
change (such as ensuring research is included in job descriptions or backfilling arrangements to
support research or organisational research dissemination strategies).
* CMHDARN. Research Seeing Grants Program: An Evaluation, 2014.
| 29
Community Mental Health Drug and Alcohol Research Network Evaluation
5. CONCLUSIONS
5.1 Overall findings
CMHDARN’s goals are relevant to individuals and organisations working in mental health and
drug alcohol sectors. CMHDARN is effective in working towards achieving its goals. There was
high level of awareness of the CMHDARN and familiarity with its work, although this is not
surprising given most survey respondents are either members of the network or have attended an
event.
CMHDARN has implemented activities (such as the Mentoring Project) cognisant of the often
limited research experience or skills of participants. This has been the right approach. According
to the survey, most organisations do not have a research team or even a research position to
undertake research (and even when they do their capacity may be limited). Individuals who
participate in CMHDARN activities report that it has had a positive impact on their knowledge,
skills and self-efficacy.
Key informants thought the CMHDARN has done a good job of trying to focus on both mental
health and drug and alcohol issues, although they observed that the mental health sector tends
to participate more in its activities. This was not the experience of the evaluation, where there
were twice as many survey respondents from NADA member organisations vs. MHCC member
organisations. Also a review of attendance at the Research Forums throughout 2014 indicated a
fairly even attendance between NADA and MHCC member organisations. CMHDARN is also
recognised as having worked to promote the importance of consumers/clients as partners in
research.
The evaluation has found that the impact the CMHDARN has on individuals and organisations
research capacity relates directly to the level of familiarity and involvement individuals have with
the CMHDARN, this suggest that an important component of future activity is to continue to
expand the reach and participation of community managed organisations in network activities.
| 30
Community Mental Health Drug and Alcohol Research Network Evaluation
Key impacts and achievements include:
CMHDARN has increased the self-efficacy of network participants:
– 66 per cent more confident or somewhat more confident to use research findings, and
– 63 per cent more confident or somewhat more confident in practice-based research.
CMHDARN has increased the knowledge of drug and alcohol, mental health and co-existing
issues among network participants:
– 61 per cent strongly agree or agree that it has improved their understanding of drug and
alcohol issues;
– 54 per cent strongly agree or agree that it has improved their understanding of mental
health issues; and
– 59 per cent strongly agree or agree that it has improved their understanding of co-
existing mental health and drug and alcohol issues.
Engagement with CMHDARN:
o 1,437 NSW visitors to the website;
o 11,063 website page views from NSW visitors;
o 11 research forums with a total of 493 participants;;
o 7 webinars with 257 participants;
o 7 newsletters distributed; and
o 240 CMHDARN members.
5.2 Usefulness of CMHDARN activities
There was very good feedback on the variety of activities (webinars, mentoring, research
seeding grants, research forums) that have or are being implemented by CMHDARN. For those
who have participated in these activities, they found them to be informative and useful.
Research forums and webinars are an important and useful activity, particularly given that these
types of events are one of the major ways survey respondents identify that they keep up to date
with research. The typical research forum format, presentations followed by workshops and/or
small group discussions, was well received. Focus group participants also commented that
research forums have been useful at facilitating collaboration with researchers, as well as
providing valuable opportunities to network with colleagues.
The CMHDARN website and e-newsletter are useful tools for providing information to interested
parties. A third of all page views related to finding more information on CMHDARN activities. The
| 31
Community Mental Health Drug and Alcohol Research Network Evaluation
usefulness of the website could be further enhanced by enabling participants to register for
forums online (via an embedded form).
CMHDARN has recognised that, due to the cost, obtaining journal articles is an issue for
individuals and organisations (this arose during the focus group and survey). CMHDARN offers a
service where it will source journal articles (two per request) for MHCC/NADA member
organisations and has promoted sites that offer ‘open access’ to research. The uptake of this
service to source journal articles has not occurred. Further consideration may need to be given
as to how the community sector can be supported to access journal articles. As one key
informant put it “library access would be good”.
5.3 Planning, implementation and communication
CMHDARN Project Reference Group is working well and makes a useful contribution to the work
of the network. There is also good ongoing monitoring of all project activities. All forums are
evaluated and reports posted online. This has made it easier to provide detailed reports to the
Project Reference Group, partners, funders and to inform this evaluation.
CMHDARN members are kept well informed of upcoming events. While a third of survey
respondents noted that they are members, there was some confusion, as nearly a third of
respondents were unsure if they are members. It would be easier and simpler to shift to an email
subscription service (managed online). The main advantage is that it makes it easier for people
to join the subscription (less information should be required, for example postal addresses would
not be required, and with one click you are subscribed).
In addition, it returns control to subscribers allowing them to easily withdraw from the subscription
when they chose which is more in keeping with current expectations for email recipients (as an
example, one respondent commented “I hope I will be removed from the address lists”).
5.4 Measuring impact over time
This evaluation has demonstrated the immediate impact CMHDARN has had on research
capacity. It will be valuable to assess the impact over the longer term. A number of impact
indicators used in this evaluation should be run again in approximately two years’ time to assess
how things may have changed. This includes:
understanding of drug and alcohol and mental health issues;
confidence to use research findings in your work;
confidence in undertaking practice-based research;
| 32
Community Mental Health Drug and Alcohol Research Network Evaluation
impact on your own research capacity; and
impact on organisational research capacity.
It is important to bear in mind that the level of involvement in CMHDARN activities will impact on
the performance as measured against the above indicators (i.e. those with greater involvement
n CMHDARN activities were more likely to rate its impact more highly). Furthermore, the overall
number of respondents to these questions was fairly small (approximately 40 respondents) so any
comparison between these findings and any future findings will be impacted by the number of
respondents recruited (i.e. a small increase in the number of respondents could significantly
change the outcomes).
This evaluation has focussed on interviewing those individuals who have participated in or
engaged with CMHDARN activities. A future evaluation should consider focus groups with senior
managers of community managed organisations to help us understand how they perceive their
organisations’ role in research and how CMHDARN can support this role.
5.5 Next steps for CMHDARN
There is strong support for CMHDARN to continue. Particularly survey respondents wanted
CMHDARN to:
develop the skills of the sector
support partnerships with researchers,
produce best practice resources,
support and mentoring
support the application of skills in practice.
It’s important that people not just learn but, as one respondent put it, “have a go at doing”
research. Within their workplaces, focus group participants and survey respondents noted that
they feel there is often a lack of support for research and evaluation within their workplace.
Furthermore, survey respondents generally indicated that there is no team or position that
focuses on research and evaluation (and even when there is they often have limited capacity).
This highlights that in addition to focussing on skills development, CMHDARN needs to give
consideration to how it can better support and engage management to build a workplace
culture that fosters research and evaluation. This would also align well with the research capacity
framework the project has adopted that emphasises the importance of sustainability and
continuity (principle five) and appropriate infrastructure (principle six).
| 33
Community Mental Health Drug and Alcohol Research Network Evaluation
However, to effectively do this discussions need to occur with funders about the funding required
to support practice-based research in community managed organisations. An evaluation of the
Research Seeding Grants Program Evaluation also made a similar recommendation.* Prior to the
formation of CMHDARN, the NGO Mental Health and Drug and Alcohol Research Grants
Program was administered by NADA and MHCC between 2007 and 2011. The program
evaluation† made a series of recommendations that are still relevant today (see Appendix B)
and are supported by the findings of this evaluation.
As noted in the introduction, the Mental Health Commission of NSW is developing a Research
Framework for Mental Health in NSW that identifies one of the overarching aims of NSW-funded
mental health services is to ‘accelerate the translation of research into practice’. This provides an
opportunity for CMHDARN, NADA and MHCC to advocate for greater funding support for the
research role of community managed organisations.
* CMHDARN. Research Seeing Grants Program: An Evaluation, 2014.
† EJD Consulting and Associates. NGO Mental Health & Drug and Alcohol Research Grants Program: Final
Report, 2012
| 34
Community Mental Health Drug and Alcohol Research Network Evaluation
6. RECOMMENDATIONS
1. CMHDARN should continue to focus on developing the research capacity of community
managed organisations, promoting the value of research and the use of research evidence
in practice.
2. CMHDARN should continue the variety of activities it is currently undertaking as they are
useful and are having an impact on skills and self-efficacy, this includes webinars, research
forums, and mentoring.
3. CMHDARN should continue with the current format of research forums (presentations which
are followed by guided small group discussions) as they are working well to promote learning
and sharing of experiences.
4. Given the usefulness of the Research Seeding Grants, the identification of the importance of
funding among evaluation participants, recommendations from the Research Seeding
Grants Program and recommendation from the Final Report on the NGO Mental Health &
Drug and Alcohol Research Grants Program, it is recommended that CMHDARN approach
funders to discuss research funding for community managed organisations to support
practice-based research.
5. CMHDARN should continue to maintain and update the CMHDARN website, further
enhancing its technical functions by enabling online registration for research forums and
more prominently featuring the option to subscribe to the email list.
6. CMHDARN should move to a subscription email list (rather than the current member model)
as it makes it easier for interested individuals to subscribe and unsubscribe from receiving
emails from CMHDARN.
7. In addition to the focus on knowledge and skills development, and given most workplaces
do not appear to have dedicated research and evaluation teams or positions, CMHDARN
needs to consider how it can support and engage management in community managed
organisations to better promote, value and support research and an evaluation culture
within their organisations.
| 35
Community Mental Health Drug and Alcohol Research Network Evaluation
8. CMHDARN should consider the implications of this report and amend its draft Evaluation
Framework to reflect the following as priority indicators for its implementation, with a view to
evaluating the program in two years’ time:
i. understanding of drug and alcohol and mental health issues
ii. confidence to use research findings in your work
iii. confidence in undertaking practice-based research
iv. impact on individual research capacity
v. impact on organisational research capacity.
9. A future evaluation of CHMDARN should run focus groups with senior managers of
community managed organisations to understand how they perceive the role of research in
their organisations and how CMHDARN can support this role.
10. CMHDARN has promoted the role of consumers/clients/carers in research and should
continue to emphasise the importance of this role.
| 36
Community Mental Health Drug and Alcohol Research Network Evaluation
APPENDIX
Appendix A: Survey Instrument
| 46
Community Mental Health Drug and Alcohol Research Network Evaluation
Appendix B: NGO Mental Health & Drug and Alcohol Research Grants Program: Final Report
The following recommendations are from the NGO Mental Health & Drug and Alcohol Research
Grants Program: Final Report (EJD Consulting and Associates, August 2011).
Research Culture within CMO
R5: Resource NADA and the MHCC as the peak bodies to support and promote the
development of evidence-based cultures within CMO, including with respect to:
a) the treatment and support of clients with co-existing mental health and alcohol and
drug issues, and
b) the inclusion of family and carers in treatment models.
R1: Provide AOD and mental health CMO, regardless of their size or location, with additional
opportunities and mechanisms to facilitate ongoing:
a) partnerships, collaborations and joint investigations with research organisations
b) capacity building, research training and skills transfer to enable CMO staff to lead and
manage research in their own right
c) promotion and dissemination of CMO focused research.
R4: Funding bodies allocate a proportion of their (core) grant funding to CMO to support
research, and enable them to:
a) collect and rigorously analyse client data and service delivery outcomes
b) undertake quality research using robust techniques
c) form partnerships with research organisations
d) disseminate their findings and share research outcomes with other CMO and research
organisations
e) in other ways implement a culture of continuous improvement and evidence-based
service delivery.
R13: NADA and the MHCC investigate further options to support CMO through:
a) commissioning joint research focused on co-existing mental health and drug and
alcohol issues relevant to CMO service delivery
b) promoting the findings of quality research relevant to CMO including via web-based
systems
c) (with appropriate approvals) promoting mechanisms to share client data that
supports service design, service delivery and/or research by other providers
d) expanding networking opportunities and information exchanges between CMO
interested in research
e) building partnerships between mental health and alcohol and drug sectors including
through supporting:
. i) joint research endeavours
. ii) other forms of collaboration and joint activity specifically related to clients with
co-existing mental health and drug and alcohol issues.
| 47
Community Mental Health Drug and Alcohol Research Network Evaluation
R14: Provide CMO with more training and information to specifically up-skill staff in:
a) relevant health and social sciences research techniques, methodologies and their
appropriate applications to CMO service settings
b) ethics approval processes and required ethics documentation
c) report writing and research dissemination.
R20:Decision makers at all levels better promote policy, funding and accountability systems that
are first and foremost client-driven, and facilitate effective, ‘joined-up’ responses to clients with
co-existing mental health and drug and alcohol issues.
Future Research Grant Administration
R2: Revise future CMO research grant funding processes to include a modest establishment grant
component to cover costs associated with ethics approval and staff recruitment.
R3: With legal input, draft a model partnership agreement for use by CMO and research
organisations covering clauses and options for the management of risks and other complex
issues including:
a) intellectual property
b) staff supervision and management
c) authorship and attributions related to research papers and publications
d) financial management and budget responsibilities
e) dispute and conflict resolution processes.
R8: Adopt similar reporting templates, and periodic feedback processes to that used for the
Research Grants Program in order to solicit formal reporting against key milestones and funding
accountabilities, together with qualitative feedback on issues and challenges arising.
R9: Amend future grant guidelines to include a provision that requires grant recipients to:
a) include the research grant as a separate line item within their organisation’s
annual financial statement and that this be:
i) reported against as part of their independently audited, end-of- year
financial statement
ii) submitted to the auspicing body as evidence of appropriate grant
expenditure
b) notify the auspicing body (on a quarterly basis for example) of any significant
over or under expenditure of grant funding based on the budget submitted,
together with either:
i) proposed measures for how to rectify the situation, and/or
ii) requests to modify or extend their research project, and allocate the
funding to complementary activities, particularly in cases where:
a) project savings have been made, or
b) grant under expenditure has occurred for other reasons.
| 48
Community Mental Health Drug and Alcohol Research Network Evaluation
R10: Include in future grant guidelines a provision for how the funding body will review and
respond to any request to vary grant funding expenditure. Criteria for approving grant
expenditure variations might include:
a) The proposal is consistent with the aims and objectives of the grants program
b) The proposal builds on or complements the initial research project’s purpose
c) The proposal can be delivered with the funding available, and within an
agreed timeframe
d) A final report (and where appropriate, also progress reports) on the revised
proposal outcome, using the standard reporting template, be submitted to the
funding body at the conclusion of the project
e) A short project plan is agreed to and signed that stipulates any requirements,
including obligations to report on project outcomes and expenditure.
R11: Future research guidelines include additional provisions for the management of unforeseen
delays in project delivery. This might involve the addition of a standardised ‘slippage’ clause in
the funding contract, that would allow, subject to appropriate notification, agreed project
milestones to be postponed for a specified number of months, without jeopardising the terms of
the grant.
R12: Future research guidelines (and related forms and processes) include a risk mitigation
section in relation to project plans. This might include risk management measures such as:
a) agreements and processes for managing the replacement or back-filling of
key personnel when extended sickness, leave or resignations occur mid-project.
b) communication protocols between partners for how significant problems or
changes in project plans or milestones will be managed.
One option might be to formally nominate ‘shadow’ or ‘back-up’ personnel within the
grant recipient organisation or research body who would be involved in key aspects of
the project and thus potentially available to assume responsibility should the need arise.
Research Partnerships
R6: Government and non-government agencies initiate new measures to extend collaboration
between the AOD and mental health sectors, at all levels, including through supporting
enhanced:
a) information sharing and referral pathways for clients
b) joint case management and service delivery options especially for clients with
co-existing mental health and drug and alcohol issues
c) partnerships agreements or memorandums of understanding defining joint
processes and procedures
d) cross-sectoral training and staff development opportunities
e) joint service delivery models, treatment and support options.
R7: Consider establishing a partnership development stream as part of core mental health and
AOD NGO funding to encourage and support formal collaboration and partnerships between
mental health and AOD CMO, as well as between CMO and government service providers.
| 49
Community Mental Health Drug and Alcohol Research Network Evaluation
Ethics Processes
R15: Establish a simplified and streamlined research ethics approval process in NSW suited to
NGO contexts and specifically including processes for managing:
a) working with vulnerable clients and other common though complex health
related issues
b) research involving multiple locations and potentially involving more than one
Local Health District.
R16: NADA and the MHCC investigate establishing a new peak ethics approval process (in
partnership with an established research body) specifically tailored to AOD and mental health
CMO in NSW.
Research Agenda
R17: Make sufficient funding available to enable the MHCC-NADA Research Network to be an
ongoing vehicle for the:
a) support, encouragement and dissemination of research relevant to CMO in NSW
b) establishment of partnerships with research bodies and individuals with an interest and
capacity for research relevant to CMO in NSW
R18: NADA and the MHCC enhance the current activities of the Research Network in order to
more intensively pursue activities listed at Recommendation 6, as well as:
a) enhancing inter-sectoral activities including via:
i) more face-to-face meetings and networking opportunities
ii) electronic exchange of members’ contact details and research interests
b) enhancing the web-based information available on current:
i) research relevant to AOD and mental health CMO
ii) research skills and development training opportunities
iii) research grant and research partnership opportunities
iv) upcoming conferences, forums and workshops.