The minutes for each meeting of the Data Ethics Advisory Group between 2019 and 2022, grouped by date, are summarised here.
2023 - 2024 meeting summaries
Summaries of the group's guidance to specific agencies is also included here.
30 November 2022
Invited to attend
Advisory Group: Professor Colin Simpson (Chair), Kate O’Connor, Dr Will Koning.
Stats NZ Officials: Caleb Johnstone, General Manager – Data System Policy; Tennille Maxey, Manager – Data System Policy; Gillian Parry – Contractor, Data System Policy; Caleb Stone, Data Broker – Integrated Data.
Secretariat: Ellen Cox, Governance and Executive Secretariat
Item contributors
Item Two: Department of Internal Affairs (DIA). Justin Rowlands, GM Te Pou Manawa, Tim Waldron, Manager Business and Market Development, Grant Stark Principal Advisor – Identity and Biometrics
1. General welcome
The Chair welcomed members and officials to the meeting and a karakia was given.
2. Digital Identity – Te Tari Taiwhenua – Department of Internal Affairs (DIA)
No conflicts of interest were noted for this item. Though one member is working on a project to migrate identity data with DIA.
A presentation was given, and discussion was held with DIA representatives on their work with Identity Check and plans regarding Digital Verifiable Identity Credentials (VICs). The purpose of the Digital Identity project is to unlock Aotearoa’s digital services ecosystem and economy.
Managing, maintaining, and storing official identity information is a key service for DIA and they regard this information as a taonga. The intention is to give control to individuals of their information.
There is a recognition that there is a growing interest from the public on the social licence for use of biometric/facial recognition by the public and private sectors, and it is important to navigate this context with care.
The benefits of the project include time and money savings, removing the constraints to access services, and ensuring agencies are only collecting the information they need.
The Groups discussion identified issues on the following topics, Digital inclusion and equity, risks and benefits, engagement with key communities, trust and confidence in the government holding personal data, fraud protection features and accessibility, Digital Trust Legislation before the House, and similar digital identity issues over overseas and lessons learnt.
3. Committee administration
Approval of previous meeting minutes and forward agenda reviewed
4. Update on the Centre of Data Ethics and Innovation Proposal
Stats NZ Officials provided the group with a verbal update on the proposed Centre of Data Ethics and Innovation
5. Levers available to the DEAG
Discussion was held regarding the support available to DEAG under the terms of reference
6. Review of the Terms of Reference
An updated Terms of reference was presented to the group for feedback and discussion. Further updates were requested.
7. Meeting review and close
Next steps were discussed and meeting closed.
04 October 2022
Attendees
Advisory Group: Professor Colin Simpson (Chair), Kate O’Connor, Dr Will Koning.
Stats NZ Officials: Dr Craig Jones, Deputy Chief Executive – Data System Leadership; Caleb Johnstone, General Manager – Data System Policy; Tennille Maxey, Manager – Data System Policy; Jess Buck – Senior Advisor, Data System Policy; Caleb Stone, Data Broker – Integrated Data.
Secretariat: Ellen Cox, Governance and Executive Secretariat
1. General welcome
The meeting was opened with a welcome and karakia from the Chair. Introductions were made.
2. Context setting
An update on the current direction and priorities for the Data Ethics Advisory Group and the role of the GCDS in fostering trust and confidence and broader work to improve visibility and examine settings for the ethical use of data was held. It was noted that the group was seen as an important mechanism to support trust and confidence in addressing complex data use issues.
3. Operating model
This session focussed on the operating model for the group and refreshing the approach taken to providing advice to initiatives.
It was agreed that that agencies should be encouraged to engage with the group at early stages of an initiative when they are more formative, rather than be seen as a compliance exercise or too late to influence the direction. This way the Group would have meaningful impact and provide a space for agencies to test thinking as it develops in away that improves effectiveness and reduces risk
4. Membership
Current membership of the group was discussed.
The Secretariat and Stats NZ Officials were asked to prepare a draft appointments process for new members.
5. Forward agenda
A discussion was held focussing on the forward agenda for the group, and projects identified of being of interest to the group were noted.
6. Meeting review
Next steps were discussed, and a review of the Terms of Reference was requested of the Secretariat.
03 June 2020
A summary of the meeting agenda - 03 June 2020
Attendees
Advisory Group: Professor Juliet Gerrard (Chair), Dr Amohia Boulton, Dr Will Koning, Dr Nessa Lynch, Kirikowhai Mikaere, Professor Colin Simpson, Kate O'Connor, Dr Ang Jury
Stats NZ secretariat: Dale Elvy, Travis Ancelet, Nienke van Dijken
(Virtual meeting on Teams)
1. General welcome
Professor Gerrard opened the virtual meeting of the Group by welcoming all of its members.
2. Declaration of conflicts of interest regarding the Ministry of Education’s Equity Index agenda item
There were no conflicts of interest declared for this item.
3. Finalising guidance on the Ministry of Education’s Equity Index submission
The Group discussed changes made to their draft guidance based on additional information and detailed discussions with the Ministry of Education. Members agreed that some minor changes were needed before the guidance could be finalised. Members agreed that once these modifications were made, the guidance could be finalised.
4. Discussion about potential additional support for considering Stats NZ’s submission on the Ngā Tikanga Paihere framework at the Group’s next meeting
The Group briefly discussed the Stats NZ submission and then identified several potential experts who could provide valuable input related to the item.
The Group agreed to provide the secretariat with the names of some experts so that the secretariat could coordinate the experts’ attendance at the next meeting.
5. Discussion about papers commissioned by the Group on international data ethics frameworks
The Group discussed papers produced for them on international data ethics frameworks and a case study of a similar data ethics group in the United Kingdom.
The Group agreed that the papers provided useful context and would be beneficial for them moving forward. They also agreed to publish the papers on the Group’s website.
6. Priorities for the next meeting and any other business
Members reviewed the current forward agenda of items submitted to the Group and outlined the items they consider high priority for discussing at future meetings.
The meeting closed with members indicating that they would like to have a full meeting in-person in the near future.
19 May 2020
A summary of the meeting agenda - 19 May 2020
Attendees
Advisory Group: Professor Juliet Gerrard (Chair), Dr Amohia Boulton, Dr Will Koning, Dr Nessa Lynch, Kirikowhai Mikaere, Professor Colin Simpson
Stats NZ secretariat: Dale Elvy, Travis Ancelet, Nienke van Dijken
Apologies: Dr Ang Jury, Kate O’Connor
(Virtual meeting on Teams)
1. General welcome
Professor Gerrard opened the first virtual meeting of the Group by welcoming all of its members.
Apologies were extended on behalf of Dr Ang Jury and Kate O’Connor.
2. Declaration of conflicts of interest regarding the Data Ventures agenda item
Professor Simpson declared a potential conflict of interest regarding costings he received from Data Ventures for inclusion into two research grant proposals. It was noted that the costings and research proposals were undertaken after the Group had developed draft guidance for Data Ventures. Based on this, the Group unanimously agreed that there was no need for Professor Simpson to be excluded from the discussion on finalising the guidance.
3. Finalising guidance on the updated Data Ventures Population Density Project submission
The Group had a brief discussion about the draft guidance they had developed for Data Ventures around their Population Density Project. Members agreed to finalise the guidance after one minor change was made.
4. Update about new and outstanding items for the Group’s agenda
The secretariat provided the Group with a summary of the submissions received to date and an overview of current discussions with agencies interested in bringing their work to the Group.
A short prioritisation discussion was had to inform the items that would be considered in the Group’s next meeting.
5. Discussion about the Group’s membership
Ahead of the Group’s formal review, the Group discussed how they feel the year has gone, how they should operate in the future, and whether the current membership structure of the Group is appropriate.
The secretariat began the discussion by providing a summary of how the Group was established and its members were selected. The discussion then shifted to the upcoming review of the Group and the secretariat provided a brief overview about how the review would operate and what the Group could expect the review to consider.
Once the scene-setting was complete, each member of the Group provided their insights about the Group’s operation and what should be considered as part of the Group’s review. Common themes brought up by members included:
- whether the Group’s advice should be binding;
- the Group’s relationship with the Digital Council for Aotearoa New Zealand;
- whether the Group should provide its guidance to Ministers as well as officials; and
- whether the Group could also be commissioned by Ministers.
The Group also had some discussion around making sure that their role was never seen as creating too many obstacles or overly critical. It will be important to get the right balance.
6. Opportunity to discuss response to COVID-19
Group members had a free-form discussion about their experiences of the COVID-19 pandemic and reflected on the changes they had seen in their professional and personal environments.
Members also shared their perspectives on how they were seeing data used locally and internationally.
7. Other business and close
The meeting closed with members indicating their willingness to have another virtual meeting in around a fortnight.
20 November 2019
A summary of the meeting agenda - 20 November 2019
Attendees
Advisory Group: Professor Juliet Gerrard (Chair), Dr Amohia Boulton, Dr Ang Jury, Kirikowhai Mikaere, Dr Nessa Lynch, Kate O’Connor, Professor Colin Simpson.
Item contributors:
- Damian Edwards, Associate Deputy Secretary, Education System Policy, Ministry of Education
- Professor Stuart McNaughton, Chief Education Science Advisor
- Philip Stevens, Group Manager Analysis, Research and Evaluation, Ministry of Education
- Allan Vester, Former Principal, Edgewater College and Chair, Sector Reference Group for the Equity Index.
Guest experts:
- Dr Ana Koloto, Director Research and Evaluation, Ministry for Pacific Peoples
- Anna McDowell, Senior Manager - Integrated Data, Stats NZ
- Professor Richie Poulton, Professor of Psychology, University of Otago.
Stats NZ Secretariat: Travis Ancelet, Sabena Blackhall, Dale Elvy.
Apologies: Dr Will Koning.
1. Welcome and introductions from the Chair, Professor Juliet Gerrard
Professor Gerrard opened the day by welcoming our returning members and reaffirming the Group’s role as a friendly mechanism established to advise rather than chastise.
The Group also extended their welcomes and made introductions to the Group’s new member, Kirikowhai Mikaere, who will fill the seat reserved for a representative of the Māori Co-Design Group whilst it is being established.
Members formally co-opted Dr Ana Koloto, Director Research and Evaluation at the Ministry for Pacific Peoples, into the Group for this meeting. Dr Koloto was invited to attend this meeting at the request of the Group Chair. This was to ensure that the perspective of Pacific Peoples was adequately represented given the potential impacts of the Equity Index.
Apologies were extended on behalf of Will Koning who was unable to attend the meeting.
2. Review of Previous Meeting
The Group discussed whether they were happy with how the previous meeting had been run and where they thought improvements could be made. Through discussion members requested:
- more time be allocated for discussions with item contributors on future agendas
- that contributors be advised to keep presentation short to maximise discussion time
- a facility for a post-meeting follow-up teleconference call ahead of finalising feedback to agencies
- that future meetings were longer to allow for at least two items to be discussed by the Group
- that the cover sheets for completed submission forms be changed to improve clarity and ease of reference.
Members considered and approved the move to a 2-stage triage format for receiving submissions. This would involve submitting agencies completing a short form to allow the Group to make decisions on whether and when an item makes it onto a meeting agenda. Successful agencies would then complete a longer, more detailed form closer to the date of the relevant meeting to ensure up-to-date information for discussion.
Members discussed the work they were hoping the Group’s new intern would undertake. The intern is currently anticipated to undertake an international stocktake of similar groups and their work, and produce a paper for the Group’s use and reference. The scope of this work will be refined in discussion with members.
3. Discussion of updated materials and documents
Members approved the Group operating documents:
- Terms of Reference
- Terms of Membership
- Operating Model
- Submission Form(s).
Please note that, in variance to the agenda provided, the Group used the remainder of the time set aside for approving documents to discuss their questions on the Equity Index ahead of meeting with Ministry of Education representatives.
4. Declaration of conflicts of interest relating to upcoming agenda items
There were no conflicts of interest declared ahead of discussing the Equity Index.
5. Discussion of item: Equity Index
The Group was provided with a presentation on the Equity Index being developed by the Ministry of Education. This was led by Damian Edwards, who was supported in his presentation by Philip Stevens, Stuart McNaughton, and Allan Vester.
Members queried several aspects of the Index before deliberating what they had heard, in private without the item contributors present.
The Group identified some areas where they required further information from the Ministry of Education ahead of formulating their advice and this was requested by the Secretariat following the meeting.
6. Priorities for the next meeting and any other business
The meeting closed with members requesting that a teleconference call be set up between members ahead of delivering their final guidance so that they could discuss the additional information requested from the Ministry.
Members also reviewed the current forward agenda of items submitted to the Group and outlined the items they consider high-priority for discussing at future meetings.
11 September 2019
A summary of the meeting agenda - 11 September 2019
Attendees
Advisory Group: Professor Juliet Gerrard (Chair), Dr Amohia Boulton, Dr Ang Jury, Dr Nessa Lynch, Dr Will Koning, Kate O’Connor, Professor Colin Simpson
Guest expert: Liz MacPherson, Government Chief Data Steward
Item contributor: Robert Chiu, Data Lead, Data Ventures
Stats NZ secretariat: Sabena Blackhall, Dale Elvy
1. Welcome from the Government Chief Data Steward, Liz MacPherson
Liz MacPherson opened the day with her ambitions for the Group, grounding members in its origin, development and the role she sees it fulfilling in the data landscape.
Liz then provided an overview of the wider landscape in which the Group sits, highlighting existing ethics framework and initiatives, before highlighting the need to streamline and standardise these approaches – a role she sees the group playing a key part in.
2. Welcome and introductions from the Chair, Professor Juliet Gerrard
Professor Gerrard built on the foundation laid by Liz, outlining the importance of establishing the Group as a supportive mechanism to guide and improve the ethics of data use across government. She emphasised that, as an unknown, people may have reservations about the Group but that these should be viewed as opportunities.
3. Group welcome and introductions
The Group then delivered individual introductions of themselves outlining their skills, experience and where they felt there is need to add value to the ethics and safeguards of the current data system.
4. Discussion of draft group processes
The Group then focussed on discussing and refining their operations. This included:
- the Application Form – members were keen for more detail to be provided at this stage and requested that new questions be included on the form such as ‘what ethical considerations or frameworks have you used in developing this project’. They also indicated support to include ethical principles on the form to guide applicants.
- the Feedback Form – members requested this follow a structure where sections highlight:
i. what was good about the item
ii. suggestions for improvement
iii. where the Group had concerns
iv. where they were satisfied
v. when it would be appropriate to check back in with the Group.
- triaging items for inclusion on the agenda – members requested that applications be condensed into a table with a high-level summary followed by key time considerations to help guide decisions on what items to discuss. It was also requested that the deadline for applications close sooner so members would have the opportunity to identify and recruit external expertise where appropriate.
- conflicts of interest – it was agreed that conflicts of interest, both on-going and ad hoc, will be declared at the start of each meeting and the appropriate means of addressing these decided on a case-by-case basis by the Group. It is also requested that the secretariat hold a ‘register of interests’ to better capture potential conflicts.
5. Declaration of conflicts of interest relating to upcoming agenda items
A conflict of interest was declared by Will Koning with regards to his company’s proposed use of Data Ventures products.
- This was declared at the start and it was deemed appropriate for him to remain for initial discussion and then leave the room at points where the conflict could interfere with the independence of advice given. This was approved by all members and undertaken at the appropriate point in the meeting.
- Will was later asked to leave the room so the Group could discuss whether they felt the conflict had been appropriately handled. This was unanimously agreed.
6. Discussion of item: Population Density Project Pilot
The Group were provided with a presentation on the Population Density product being developed by Data Ventures. This was led by Robert Chiu, Data Lead, Data Ventures.
Members queried several aspects of the project before deliberating what they had heard in private, without the Data Ventures representative present.
The Group identified some areas where they required further information from Data Ventures ahead of formulating their advice and this was requested by the secretariat following the meeting.
Group guidance on Population Density Project Pilot
7. Priorities for the next meeting and any other business
The meeting concluded with the Group requesting that updated copies of operational documents be circulated ahead of the next meeting.
Electricity data - Ministry of Business, Employment, & Innovation
Discussed on 15 October 2020.
Data Ethics Advisory Group guidance - Electricity data [PDF 167 KB]
Feedback in summary
The Data Ethics Advisory Group is of the opinion that storing monthly Installation Control Point number (ICP) electricity data in the Integrated Data infrastructure (IDI) would be acceptable. The IDI is a safe space to store data and has the necessary safeguards in place. The public can be reassured that the data will be used for public benefit. We propose working within the normal guidelines of the IDI in adding and linking granular ICP monthly consumption data, but removing identifiable data - in this case ICP numbers and addresses.
If MBIE decides to store more granular detail on this electricity data in the IDI (for example, addresses and daily use), then the Group would like to reconsider its recommendations.
Positives
Members were impressed that:
- There are opportunities for New Zealand to lead the world in understanding more about energy hardship and develop policies to make a meaningful difference to people.
- The Data Ethics Advisory Group would like to commend MBIE on taking this item to the Group so early in the process, while MBIE is still in its formative thinking.
- MBIE are planning to do a full Privacy Impact Assessment as this work develops.
Concerns
Members were concerned that:
- The Group is concerned around the granularity of the data that would be added to the IDI and understands it will be monthly ICP data. ICP data is Personal Identifiable Information (PII) data (at the household level) as it directly ties to an address (one or more at each address). ICP identifiers should not be added to the IDI.
- The Group understand that electricity providers have concerns about how the data will be treated if it is added to the IDI. While the IDI has strict protocols for access, anonymisation of data for analysis, and disclosure control, once the data is in the IDI, it will be available to be used for a range of other research by IDI users.
Recommendations
Members recommend:
- MBIE’s preferred approach is to engage with retailers and seek permission to pass this data from the Electricity Authority (EA) to Stats NZ. The Group supports this approach and recommends for the data to be stored in the IDI, as long as the appropriate safeguards are set up. The Group suggests that the energy hardship working group should be established before adding the data to the IDI.
- For the data to be really useful in the IDI, it should be linked to the other data in the IDI - which has PII stripped out. Once linked, the ICP numbers should be stripped out. This means that granular linked data is in the IDI, but it is de-identified. Researchers can then analyse the IDI data by itself or in combination with other data, but without access to PII. We propose working within the normal guidelines of the IDI in adding and linking granular ICP monthly consumption data, but removing identifiable data - in this case ICP numbers and addresses. This will maximise the potential benefits and minimize (privacy) risks.
Noted
Members noted that:
- ICP data is effectively data on household level. ICPs and addresses are currently published in a searchable database (individually), ie via the registry hosted on the EAs website. This means that even though the address is effectively published, it is low level usage data.
- A privacy impact assessment will be done before adding the data to the IDI.
- MBIE also asked for guidance on how it should engage with stakeholders, but the Group did not feel they had enough information to be able to provide advice on this.
Future guidance
The Group would like to see you again once the discussion ranges beyond the proposed narrow use of electricity data and would move to wider usage.
Immigration NZ's risk analytics platform - Ministry of Business, Employment, & Innovation
Discussed on 05 August 2020
Data Ethics Advisory Group guidance - Immigration NZ's risk analytics platform [PDF, 176KB]
Feedback in summary
The Group commended the detailed thinking behind the governance arrangements supporting the development of Immigration NZ’s Risk Analytics Platform. They were concerned around the composition of the Data Science Review Board, particularly that a majority of members were drawn from MBIE or Immigration NZ or their current supplier. They recommend having increased representation from independent members, particularly with human rights and/or ethics experience. The Group also recommends openness and transparency around the advice from the Data Science Review Board.
Positives
Members were impressed that:
- There has been detailed thinking on the development of Immigration NZ’s Risk Analytics Platform and the governance arrangements to support this.
- Immigration NZ has identified the need to apply a data ethics lens to its processes.
- There will be external voices on the Data Science Review Board.
- MBIE is one of the founding signatories of the Algorithm Charter and that they intend to assess their algorithm decisions using the risk matrix from the Algorithm Charter.
Concerns
Members had concerns that:
- The composition of the Data Science Review Board is too internally focussed and the Board does not have human rights and ethics representatives.
- Training of the data models (machine learning) relies on past data and that Immigration NZ’s dependence on this could have implications for the operations of the Risk Analytics Platform. The Data Science Review Board should include independent voices with specific expertise to interrogate this risk.
- The Data Science Review Board will require clarity on how the risks of the data models and methods are being mitigated. The Data Science Review Board should include independent voices with specific expertise to interrogate this risk.
- While rule-based automation is easily reviewed, machine learning models and their underlying training data are not.
- There is not yet a risk and assessment framework for the Data Science Review Board to guide good governance.
- Greater clarity is desirable in how the Data Science Review Board will be empowered and what their role will be within the wider structure.
Recommendations
Members recommend:
- The appointment of independent members with a human rights and/or ethics background to the Data Science Review Board. It is important that these human rights and ethics principles will be embedded from the start and that the communities most affected be consulted. Given that the field is an emerging one, it is advisable to extend the call or search to more early-career experts and to consider inviting or co-opting guest experts for particular issues.
- That the Data Science Review Board has a focus on keeping models as simple as possible to minimise the risk of unintended bias and to facilitate effective review.
- That if MBIE does move beyond simple models, they ensure that these are tested using synthetic data where sets of tests are processed and the majority of elements are maintained while permutations of variables are added that should have no bearing on the outcome (i.e. partner of applicant is same-sex/opposite-sex, Pākehā/Māori) and report outcomes to the Data Science Review Board to help interrogate unintended bias.
- That the sampling period of training data is reported to the Data Science Review Board alongside a timeline of relevant sector events (i.e. regulatory change, visa scandal).
- The inclusion of an independent data scientist on the Data Science Review Board when reviewing more complex models.
- Openness and transparency around the advice from the Data Science Review Board.
- For a risk assessment framework for the Data Science Review Board to be developed before the Data Science Review Board commences.
Noted
Members noted that:
- It would be helpful for MBIE to consider further independent advice on the wider application of the Risk Analytics Platform past the low-risk use-cases mentioned.
- Human rights are fundamental to the operation as well as the policy and while existing governance structures are in place, the processes could benefit from a specific focus on human rights.
Future guidance
We note that this item related to low-risk category applications. Should the use of the Risk Analytics Platform begin to extend to other categories of visa, we would welcome the opportunity to provide further advice.
Please get in touch with the Group's secretariat should you have any questions about the guidance you have received. We are always delighted to hear from you.
We would also note that the Group reserves the right to extend an invitation to the Minister where they are concerned that their guidance has been misinterpreted or applied in bad faith.
Ngā Tikanga Paihere - Stats NZ
Discussed on 05 August 2020
Data Ethics Advisory Group Guidance - Ngā Tikanga Paihere [PDF, 186KB]
Feedback in summary
The Group commends the Ngā Tikanga Paihere framework as a great example of a collaborative way of working in partnership with Iwi/Māori to create better outcomes for all of Aotearoa New Zealand. The Group shared the concerns from the presenting group that the framework might be seen as a substitute for proper engagement with Iwi/Māori and commends future efforts to help users understand how the framework can be used.
Positives
Members were impressed that:
- In the process of developing the Ngā Tikanga Paihere framework, thoughtful engagement with Māori and iwi experts was apparent.
- This a great example of a collaborative way of working in partnership to create better outcomes for all of Aotearoa New Zealand.
- The framework, developed in partnership with the IDI, has innovated existing processes and has had a real and meaningful impact on culture within Stats NZ. The members were impressed with the lessons learned by the team.
Concerns
Members had concerns that:
- Around the practical implementation of the framework. We agree with the presenting team that there is a risk that the framework might be seen as a substitute for proper engagement with Iwi/Māori and/or might be used for unintended purposes.
- There are other interpretations of tikanga, and the framework should be clear about the concepts that underpin it. The team discussed this in depth and agreed that there is scope to document this approach to make the origins of the framework clear for later iteration.
Recommendations
Members recommend:
- Users of the framework are assisted to understand that the framework is not a substitute for engagement with Iwi/Māori and how the framework can be best used.
- The team maintains flexibility if researchers and users have different interpretations of their own tikanga.
Noted
Members noted that:
- The framework should line up with the Te Ara Tika Guidelines for Māori research ethics (Pūtaiora Writing Group) and complement other existing frameworks.
- Overall, Ngā Tikanga Paihere is a very positive framework and would like to commend the team on the work they have done.
- They thoroughly enjoyed this session and thank the presenters for their energy and commitment to inclusive data ethics.
Future guidance
The Group would be very happy to provide any further guidance if this framework evolves or changes.
Please get in touch with the Group's secretariat should you have any questions about the guidance you have received. We are always delighted to hear from you.
We would also note that the Group reserves the right to extend an invitation to the Minister where they are concerned that their guidance has been misinterpreted or applied in bad faith.
Adding PISA data to the IDI - Ministry of Education
Discussed on 05 August 2020
Data Ethics Advisory Group Guidance - Adding PISA data to the IDI [PDF, 168KB]
Feedback in summary
The Group was impressed by the Ministry of Education [MoE]'s thoughtfulness around this issue. The Group found it encouraging that MoE has already thought through the issues, looked at the risks, came up with possible solutions, weighed up these options and came to the Group with a clear plan for implementation. The Group was concerned that the wording of the letter seeking consent by Principals to participate in the PISA 2009 study did not reflect the intended use of that data for research. The Group recommends MoE contacts the principals, explains the issue with the PISA 2009 consent letter, explains how MoE intends to fix the problem by adding the data to the IDI and how it intends to do this.
Positives
Members were impressed that:
- MoE had already thought the issues through, assessed the risks, came up with possible solutions, weighed up these options and came to the Group with a clear plan for implementation.
Concerns
Members had concerns that:
- The PISA 2009 data would be used for purposes other than what it was collected for and other than what the Principals (and by implication the children and caregivers) provided consent for
Recommendations
Members recommend:
- That MoE contact the Principals, address the issues regarding consent for the PISA 2009 data to be used for research, explain that MoE intends to fix that problem by adding the data to the IDI and the steps it intends to take to achieve this, including how the data will be protected.
- After MoE has contacted the Principals, MoE can start working with Stats NZ on adding the PISA 2009 data to the IDI as it will be better safeguarded in the IDI.
- That separate space with additional checks and balances be created within the IDI to protect this data and consideration be given to whether it is automatically linked.
Noted
Members noted that:
- Education data from the PISA study fills an important gap in allowing researchers to understand how education achievement, and other social factors, can impact later life.
- The wording on the PISA participation letter to Principals was changed for subsequent PISA data (2012, 2015, 2018) and that this data has been added (or is scheduled to be added) to the IDI.
- That the 2009 PISA data had already been used by education researchers, in contravention of the assurance made in the 2008 letter to Principals.
- In 2009, there was also a parents survey. There are not enough benefits for adding this data to the IDI and would be contrary to the terms under which parents gave the information and their understanding of how it would be used.
Future guidance
We note that this is a one-off issue, but would welcome any other opportunity to provide guidance if this would be helpful.
Please get in touch with the Group’s secretariat should you have any questions about the guidance you have received. We are always delighted to hear from you.
We would also note that the Group reserves the right to extend an invitation to the Minister where they are concerned that their guidance has been misinterpreted or applied in bad faith.
Equity Index - Ministry of Education
Discussed on 20 November 2019
Data Ethics Advisory Group Guidance - Equity Index [PDF, 182KB]
Feedback in summary
The Equity Index is an improvement on the decile system. The focus on equity, and the aim of more up-to-date and granular data is to be commended. These positive aspects must be offset against wider concerns. The Equity Index will require greater stakeholder and public engagement and refinement as it develops beyond the current design.
Positives
Members were impressed that:
- Equity is the conceptual underpinning of the move from the decile system to the Equity Index
- The Equity Index should distribute resources more equitably than the existing system, using more regularly updated and granular data
- The Equity Index is being developed and introduced through a staged and considered process.• There has been stakeholder engagement with the education sector, and more comprehensive consultation planned.
Concerns
Members had concerns that:
Consultation
- There has been limited engagement with the public, although we understand more comprehensive consultation is planned
- Iwi, as the Treaty partner, have yet to be consulted on the Equity Index, which raises concerns about adherence of the Index to the principles of the Treaty of Waitangi
- Māori Data Sovereignty remains unresolved and issues of social and cultural license remain, as the Equity Index does not take into consideration the rights of the collective
- Although children have been consulted broadly on the Equity Index, they have not yet been consulted on the indicators used, despite the Equity Index having considerable implications for children’s daily lives.
Indicators and data quality
- There is currently a lack of data, or usable data, around key indicators of socio-economic outcomes such as health - though we understand the inclusion of such data will be considered once it is of an adequate quality and prevalence
- Groups who are not adequately represented in the Integrated Data Infrastructure will not have their socio-economic status as effectively represented by the Equity Index, such as:
- children who are not living with their biological parents, when these are a community of concern
- children of parents who were not born in New Zealand
- As a result, the algorithm was trained on incomplete or misleading data and could embed existing inequities – although we note that the Equity Index is not an individual-based model and therefore variance on each student’s estimation will be averaged out at a school level.
Equity
- Potential harms and benefits of moving away from deciles and to the Equity Index have not been effectively quantified or demonstrated
- There is no clear line of governance around the Equity Index or ensuring that the purpose of addressing inequity is achieved
- Two issues which remain (although we understand these are currently under consideration) are:
- It is not clear how the evidence base on whether the Equity Index is fit for purpose will be created or monitored, revealing an apparent absence of proper evaluation
- The introduction of a regional multiplier to effectively target regions that experience greater poverty.
Transparency
- There is an overall lack of transparency, with regards to both the development of the Equity Index and the algorithm that underpins it, which has the potential to:
- Confuse the public – between poor communication and a drive to avoid the ease of extrapolation inherent in the decile system, the Equity Index could be seen to obfuscate the system for school funding
- Conceal algorithmic imperfections – thereby magnifying their impact by making them difficult to identify and address
- Promote risk resistance and cause upset in the education sector
- Have enduring issues for schools’ ability to query their funding, risking resort to adversarial complaint processes
- Feed fears around the limits and anonymity of data use as questions around social license are unresolved
- If not designed and implemented effectively, the Equity Index could risk undermining trust in individualised data-based decision making and the wider data ecosystem of the country.
Recommendations
Members recommend:
- Iwi are consulted on the Equity Index as a matter of priority, both on the indicators used and more broadly
- Children be consulted on the indicators used
- Ethnicity data be included in the variables used by the Equity Index
- A clearer cost-benefit analysis of the Equity Index is carried out. This would include clarity on how the benefits of the Equity Index feed into wider ambitions around improving equity and exceed those of the decile system
- The Ministry of Education continue to consult the public widely as the Equity Index continues to develop and refine.
Noted
Members noted that:
- The Equity Index follows a risk model, based on the view that parents’ characteristics and circumstances predict the life-course of a child
- The omission of the sex of school pupils as an indicator is potentially inconsistent with the factors underpinning the choice of indicators used
- A data ethics assessment must necessarily consider whether the purpose of the use of the data is likely to be achieved; but questions such as the below remained unresolved:
- whether introducing the Equity Index risks significant disruption for little gain as its benefits are dependent on effective resourcing
- whether the Equity Index will influence cultural change or promote effective equity initiatives in schools
- how the funding allocated through the Equity Index would enable inequity to be addressed
- Potential external impacts of the Equity Index should not impact on the design of, or communications about the Index
- It will still be up to individual schools to make decisions on how to address equity with the allocation they are given through the Equity Index process.
Future guidance
We know that projects change and evolve. We would welcome further discussions with the Group about any substantive changes, including to the methodology being used or after consultation with iwi, children, and the wider public.
Population Density Product - Data Ventures
Discussed on 02 March 2020.
Data Ethics Advisory Group Guidance - Population Density Product - June 2020 [PDF, 156KB]
Feedback in summary
Overall, the Data Ethics Advisory Group felt this was an effective and proportional mechanism for assuring appropriate customer evaluation for the Population Density product, subject to the following considerations.
Positives
Members were impressed that:
- This appears to be a considered and effective tool for self-regulation
- Multiple team members within Data Ventures evaluate each item, safeguarding oversight and minimising the potential impact of bias in evaluation.
Concerns
Members had concerns that:
- The checklist approach to evaluation could introduce risks by not taking a holistic view of a project
- There is no clear metrci for arriving at scores - aside from aggregation of scores from different team members - which risks results appearing arbitrary and therefore potentially undermining trust for customers and the public.
Recommendations
Members recommend:
- That evaluation of customers includes multi-disciplinary perspectives to provide high-level oversight and ensure best practice
- That some guidance be produced to assist with scoring to improve standardisation.
Noted
Members noted that:
- Guidance has been predicated on the assumption that use of the Population Density product will be underpinned by a contract assuring that the data is only used for the purpose(s) specified in the customer’s application and will contain adequate checks and balances to ensure this is upheld
- We understand that the terms and conditions of the contract are currently being drafted.
Future guidance
We would be keen to reconsider this item should the granularity of the data available increase, especially in the context of the COVID-19 response. Please get in touch with the Group’s secretariat should you have any questions about the guidance you have received. We are always delighted to hear from you. We would also note that the Group reserves the right to extend an invitation to the Minister where they are concerned that their guidance has been misinterpreted or applied in bad faith.
Population Density Project Pilot
Presented by Robert Chiu from Data Ventures, and discussed on 11 September 2019.
Data Ethics Advisory Group Guidance - Data Ventures - Sep2019 [PDF, 153KB]
Feedback in summary
Overall, Group members were of the view that there would be public benefit of the Population Density product. The privacy risks were deemed to be very low since the data have been aggregated in high numbers and employ random rounding. Careful due diligence has been undertaken by Data Ventures to ensure compliance with ethical, privacy and governance standards. Members were reassured by the positive assessment from the Privacy Commissioner.
Positives
Members were impressed that:
- this project is adding value for New Zealanders and would contribute to positive social outcomes
- there are existing and comprehensive agreements made between Data Ventures and pilot users
- specific use cases were requested and the high standards of data processing specified in the contract were welcomed
- there was a good balance between granular data and safeguards around anonymity and privacy. They were of the view that it was clear that Data Ventures had been very careful in anonymising and aggregating the data and that this anonymity was guaranteed.
- the data trust model on which Data Ventures is positioned effectively ensures low risk by keeping data largely in situ
- the highly aggregated nature of data available to Data Ventures had a positive impact on lowering risk
- Data Ventures had clearly carried out appropriate due diligence, including assessment from the Office of the Privacy Commissioner
- there is no ability to reverse engineer the data provided to Data Ventures in order to identify individuals at this level of data aggregation.
Concerns
Members had concerns that:
- there is potential for this level of detail to be taken lower (to be more granular) in the future but welcomed the extent of protections in place at present
- some demographics (for instance children without mobile phones or those living in low signal areas) would not be present in the data and that this could consequently skew decisions made based on this information
- new uses of the data, for instance in disaggregating national and international SIM cards, would require aggregation to larger cell numbers to preserve privacy
- this project was vulnerable to culture change within Data Ventures, were future leadership to seek optimisation of the commercial aspects of this product.
Recommendations
Members recommend:
- that the limitations of the data be outlined to ensure that they are front of mind when used for planning, particularly with regard to the likely absence of many children from the dataset
- that it was important to front foot issues around lack of context for low signal areas
- that this product not move to meshblock level until privacy and accuracy are assured and that this goes to no further level of detail
- combining adjacent SA2 when counts drop below 100 and that consideration be given to this threshold being raised for commercial customers
- that greater safeguards be built into the licensing and contractual obligations around the data received from telcos as well as for users of the product to protect against unethical exploitation.
Noted
Members noted that:
- this use of data was already outlined in the telcos' terms of use and that it is possible to opt out of this use with one, but not all, providers
- Data Ventures had effectively filled a gap in the market that could otherwise have been filled by more financially-motivated private organisations and were of the view that this was to the public good
- it is important that Data Ventures tell the story of the project's benefits and how risks had been mitigated to nurture trust and confidence.
Future guidance
We know that projects change and evolve. We look forward to having Data Ventures back to the Group to hear how this item is doing and also to renew guidance should this be appropriate. The Group set the threshold for revisiting this item as:
- the moving of data areas from suburb to meshblock level, or
- if this product is made commercially available to the private sector, or
- if multiple new datasets are added to the product.
