B.1. Methodology overview
The scope of the Five-Year Review was to:
- review the legislative and regulatory settings underpinning CISS
- review impacts (if any) of CISS on diverse and disadvantaged communities, children and young persons
- review unintended consequences (if any) of CISS – both positive and negative
- review CISS’ role in achieving VCIS Reform outcomes to date.
This Review specifically investigates the extent to which CISS is meeting the intended outcomes articulated through the regulatory and legislative settings from:
- the Child Wellbeing and Safety Act 2005
- the Regulations
- the Child Wellbeing and Safety (Information Sharing) Amendment Regulations 2020 (the Amendment Regulations).
The following section details this Review’s methodology.
To guide this Review, an evaluation framework was developed to inform data collection and data analysis. This included the development of a program logic, key review questions, and performance indicator matrix.
Key review questions were developed for the evaluation framework which defined the scope and focus of this Review. A mixed-methods approach was adopted to obtain the relevant information and best respond to the key review questions. Collected data was then analysed thematically through a structured process of review, reflection and refinement.
B.2. Evaluation framework
The evaluation framework consists of a program logic, key review questions, and performance indicator matrix.
B.2.1. Program logic
A program logic was developed to define CISS’ expected chain of cause and effect by clearly articulating any assumptions and logical links which it is based on. It underpins this Review as it maps short-, medium- and long-term outcomes CISS is expected to deliver alongside the inputs, activities and outputs that support the realisation of these outcomes.
The Program Logic was based on the Outcome Measurement Framework developed by the Department in 2020, which was based on the CISS Program Logic Model at the time of this Review. Due to the comprehensive nature of the previous outcomes for CISS, no significant changes were made to the outcomes. Outcomes were instead re-classified with respect to timelines and thematic groupings. Where there was a significant conceptual overlap, long-term outcomes were merged.
Deloitte facilitated two Program Logic Workshops with the Department to discuss and co-design the Program Logic. Workshop 1 focused on CISS-related inputs, outputs, activities and outcomes. Following this Workshop, significant changes were made to the Program Logic, mainly in relation to the outcomes. These changes include:
- development of a vision statement, assumptions, and current contextual factors in Victoria to support the testing of the Program Logic during these reviews
- development of impacts to detail the intended broader, long-term impacts of CISS, past the point at which these systemic changes can be confidently attributed solely to these components of the VCIS Reform
- re-drafting of the outcomes to better reflect the current implementation stages (e.g., CISS Phase 1 and Phase 2) and levels of influence of CISS. At the time of these reviews, it is expected that CISS should have progressed past awareness of the VCIS Reform and so changes to the outcomes better highlight what CISS should be intending to achieve. The outcomes have also been grouped by the stakeholder group intended to benefit from CISS, which are children, service provider management, and service provider workforce.
Subsequently, Deloitte held Workshop 2 with the Department to discuss the changes to the Program Logic. The key review questions were developed off the Program Logic. The full Program Logic is shown in Appendix A.
B.2.2. Performance indicator framework
The performance indicator matrix is a tool to identify review questions and sub-questions and develop plans to collect the information needed to address them. Deloitte designed indicators and mapped corresponding data sources to enable assessment of the outcomes of CISS over this Review period and for findings to be made in relation to each key review question.
B.3. Data collection
A mixed-methods approach was adopted to obtain the relevant information and best respond to the key review questions outlined in Section 1.6.2. Primary and secondary data collection occurred between August to November 2023.
B.3.1. Primary data collection
Primary data for this Review was collected from semi-structured interviews with government, peak body, union, and workforce stakeholders, and a survey distributed to workforces who use CISS. The approach to primary data collection was informed by the following principles:
- being present and forming trusting relationships with stakeholders
- listening to and ensuring stakeholders have opportunities to outline how CISS meet or do not meet intended outcomes of the VCIS Reform
- working with the Department to ensure engagement is representative of the diversity of individuals who use CISS, including Aboriginal and Torres Strait Islander communities, culturally and racially marginalised communities, and lesbian, gay, bisexual, transgender, queer, and intersex (LGBTQI+) communities
- being flexible and adaptable in the approach to engaging with stakeholders to ensure our approach best meets their needs
- being independent and robust in the evaluation methodology
- sharing transparent and timely feedback with the Department and stakeholders
- recognising existing CISS strengths
- recognising that the outcomes of the VCIS Reform can take many years.
B.3.2. Ethics applications
Primary data collection occurred in an ethical manner and according to the Australasian Evaluation Society Guidelines. As this Review engaged with stakeholders from the Justice and Education sectors, applications to the Justice and Human Research Ethics Committee (JHREC), Research in Schools and Early Childhood Settings (RISEC), and Victoria Police’s Research Coordinating Committee were required.
An application to the JHREC was made on 4 August 2023. The purpose of this ethics application was to conduct ethical engagement with key workforces from the Department of Justice and Community Safety, and to ensure that this Review complied with the principles and guidelines set out in the National Statement for Ethical Conduct in Human Research. The application was given full approval on 14 September 2023 (Reference ID: CF/23/18898).
An application to Victoria Police’s Research Coordinating Committee was made on 28 July 2023. This application was eligible for a streamlined review process as a concurrent application was made to JHREC. The purpose of seeking this research approval was to engage with Victoria Police personnel in accordance with Victoria Police’s organisational strategic priorities. The application was given approval on 30 October 2023 (Reference ID: RCC 1074).
An application to the RISEC was made on 28 July 2023. The purpose of seeking this research approval was to conduct ethical engagement with government schools and early childhood settings in accordance with the Department’s duty of care. The application was given approval on 1 August (Reference ID: 23-07-072).
Data collection tools, participant information sheets, and consent forms were then developed based on advice from the JHREC, RISEC and Victoria Police ethics committees. These documents outlined key information including:
- what the purpose of the data collection and project is
- that participation is voluntary
- how information is stored and used
- that their information or survey responses will not be shared with other stakeholders
- all analysis (except for case studies with participant’s permission) will be conducted at a thematic level with no individuals identified
- who to contact with any questions.
B.3.2.1. Stakeholder consultation
36 semi-structured interviews and two focus groups occurred with 32 government, peak body, union, and workforce stakeholders between August and November 2023. These organisations are outlined in Table B.1 below.
Table B.1: Organisations who participated in semi-structured interviews
| Organisation | Number of people interviewed |
|---|---|
| Australian Education Union | 2 |
| Australian Nursing and Midwifery Federation | 2 |
| Australian Primary Health Care Nurses Association | 1 |
| Bendigo & District Aboriginal Co-operative | 1 |
| Bendigo Special Developmental School | 1 |
| Births Deaths and Marriages | 2 |
| Centre for Excellence in Child and Family Welfare | 2 |
| Centre for Mental Health Learning | 1 |
| Commissioner for Aboriginal Children and Young People | 1 |
| Commissioner for Children and Young People | 1 |
| Community Child Care Association Inc | 1 |
| Community Housing Federation of Victoria | 1 |
| Court Services Victoria | 2 |
| Department of Education | 5 |
| Department of Families, Fairness and Housing | 8 |
| Department of Justice and Community Safety | 2 |
| Department of Premier and Cabinet | 2 |
| Doveton College | 1 |
| Early Childhood Australia | 1 |
| Early Learning Association Australia | 1 |
| Melbourne Archdiocese Catholic Schools | 2 |
| Municipal Association of Victoria | 3 |
| Njernda | 1 |
| The Orange Door | 2 |
| Our Place | 2 |
| Principals Association for Specialist Schools | 1 |
| Rumbalara Aboriginal Co-operative Corporate | 1 |
| Sparkways | 1 |
| The City of Casey | 2 |
| The City of Greater Bendigo | 1 |
| The Department of Health | 2 |
| The Secretariat of National Aboriginal and Islander Child Care | 1 |
| United Workers Union | 2 |
| Victoria Police | 4 |
| Victorian Aboriginal Child Care Agency | 2 |
| Victorian Aboriginal Community Controlled Health Organisation Inc. | 2 |
| Victorian Aboriginal Education Association Inc. | 2 |
| Wathaurong Aboriginal Co-operative | 1 |
A total of 65 organisations were invited to participate in this Review. These organisations were recruited through emails which contained information around the scope of this Review, requirements to participate if they consented to, information on how their answers will be used, and mechanisms for additional questions or complaints. Follow up emails were sent for any non-responses within a week of the original email. If a response was received, Deloitte worked directly with the stakeholder to organise a time for an interview over Microsoft Teams. The following consultation principles were applied to conduct safe and considerate engagement with participating stakeholders:
- voluntary and consent-based engagement
- inclusive and respectful engagement
- listening and giving stakeholders the opportunity to share their views
- being flexible and adaptable to ensure consultation meets stakeholders’ needs
- remaining confidential and private.
B.3.2.2. Survey
An online survey was designed and developed in Qualtrics by Deloitte. It was disseminated in multiple tranches between 7 September and 27 October 2023, with a total of 340 completed responses received. The highest proportion of responses were from the education sector (40 per cent), followed by health (21 per cent) and early childhood education (16 per cent, n=340).
Further detail on the development and distribution of the CISS Workforce Survey, as well as the survey responses are provided in Appendix C.
B.3.3. Secondary data collection
Secondary data for this Review consisted of key documents provided by the Department to evaluate CISS’ implementation and effectiveness between 2020 (the end of the review period of the legislative Two-Year Review) and 2023. These included annual reports, previous reviews, training data, and other program documentation.
Supplementary information not shared by the Department was also reviewed against the key review questions. These included publicly available documents accessed through a desktop review, such as previous reviews of FVISS and MARAM and other similar information sharing schemes in other jurisdictions, and documents shared by stakeholders such as internal departmental manuals for information sharing under CISS.
B.4. Data analysis
Primary and secondary data were then thematically analysed to identify emerging themes, ensuring the privacy of evaluation participants is maintained. Qualitative data was then triangulated with findings from the secondary data and other data sets, to validate findings and gain more in-depth insights. Thematic analysis was undertaken through a structured process of review, reflection and refinement, outlined in detail below:
- Review – the information collected was read and topics/issues discussed or raised were coded to allow for consolidation into themes that can be drawn together across data sources
- Reflection – an initial thematic analysis was considered which reviewed the data collected through the reading and consultation
- Refinement – the themes were described as clearly and concisely as possible to minimise duplication between themes and consider the impact of the qualitative themes.
B.5. Limitations
B.5.1. Sample of participants
Primary data collection was conducted with a select mix of stakeholders who were familiar with CISS. This may impact the generalisability of the findings given CISS’ broad reach and use. Efforts to refine the implementation, effectiveness and legislative and regulatory settings require input from a more extensive range of stakeholders across different industries as well as the benefactors of CISS. This ensures that the findings and recommendations are developed to achieve desired information sharing for child wellbeing and safety. A subsequent review of CISS would benefit from further engagement with priority populations (e.g., culturally and racially marginalised populations, Aboriginal communities, individuals with disabilities) and Phase One workforces.
B.5.2. Establishing a baseline
The Survey developed and distributed for this Review contained some of the same questions as the Two-Year Review to allow for comparison. However due to the scope of this Review, the CISS Workforce Survey was not able to fulsomely establish a baseline. This resulted in limited ability to compare the findings of the Two-Year Review with the findings of this Review.
Updated